Chapter 46: Automated Experiment Design

Overview

Every experiment costs something: reagents, compute cycles, telescope time, clinical trial enrollment. A naive experimenter selects measurements uniformly at random; a skilled experimenter chooses each measurement to maximally reduce uncertainty about the quantity of interest. This chapter teaches you to build the skilled experimenter as software. We formalize experiment selection as an optimization problem over acquisition functions that quantify the expected value of each possible measurement, then build closed-loop systems that alternate between measuring, updating beliefs, and selecting the next experiment.

The chapter connects three intellectual traditions. Active learning from machine learning asks which unlabeled data point, if labeled, would most improve a classifier. Sequential experimental design from statistics asks which treatment combination yields the most information about model parameters. Bandit algorithms from decision theory ask which arm to pull when each pull reveals a reward and updates beliefs. All three reduce to the same core loop: observe, update, decide.

We build on the optimization foundations of Chapter 45, the Bayesian methods of Chapter 32, and the search framework from Chapter 1. The closed-loop experiment planner we construct here feeds directly into the self-driving laboratories of Chapter 55 and the AI scientists of Chapter 53.

Prerequisites

This chapter assumes comfort with Bayesian inference (posterior distributions, Gaussian processes) from Chapter 32, optimization fundamentals from Chapter 45, and the information-theoretic concepts (entropy, mutual information) introduced in Chapter 1. Familiarity with scikit-learn, PyTorch, and basic NumPy is required.

Learning Outcomes

Implement query-by-committee, uncertainty sampling, and core-set active learning strategies from scratch and with modAL.
Derive information-theoretic acquisition functions (entropy reduction, BALD, expected information gain) and connect them to Bayesian optimal experimental design.
Apply Thompson Sampling, UCB, and contextual bandits to sequential experiment selection with unknown reward distributions.
Compute Bayesian Active Learning by Disagreement (BALD) acquisition scores using Monte Carlo dropout and GP posterior samples.
Quantify the Value of Information (VOI) for a candidate experiment and use it to decide when to stop experimenting.
Build a complete closed-loop experiment planner using BoTorch that runs BALD-based active learning on a simulated dose-response assay, demonstrating 3x sample efficiency over random selection.

Sections

46.1 Active Learning

Query strategies for selecting the most informative data points: uncertainty sampling, query by committee, core-set selection, and expected model change. Information-theoretic foundations and connections to pool-based, stream-based, and membership-query scenarios.

46.2 Sequential Experimental Design

Information-theoretic criteria for experiment selection: entropy, mutual information, BALD, and expected information gain. Bandit algorithms (Thompson Sampling, UCB, contextual bandits) as sequential design strategies. Value of Information and optimal stopping.

46.3 Building a Closed-Loop Experiment Planner

Recipe: build a BALD-based active learning system for a simulated dose-response assay. Ten acquisition rounds, Gaussian process surrogate, BoTorch acquisition optimization, comparison against random and uncertainty baselines. Discovery Workbench integration.

What's Next

Automated experiment design generates measurements, but those measurements need to be tracked, versioned, and made reproducible. Chapter 47: Experiment Registries and Scientific Provenance builds the infrastructure for recording every experiment the planner executes: what was measured, why it was selected, what the result was, and how it connects to prior experiments. Together, Chapters 46 and 47 form the decision-and-record backbone of the self-driving laboratories in Part VII.

Bibliography

Foundational Papers

Houlsby, N., Huszar, F., Ghahramani, Z., & Lengyel, M. (2011). Bayesian Active Learning for Classification and Preference Learning. arXiv preprint arXiv:1112.5745.

Introduces BALD (Bayesian Active Learning by Disagreement), the mutual-information acquisition function that is the centerpiece of Section 46.2 and the recipe in Section 46.3.

Settles, B. (2012). Active Learning. Synthesis Lectures on Artificial Intelligence and Machine Learning. Morgan & Claypool.

The standard monograph on active learning, covering pool-based, stream-based, and membership-query scenarios with a taxonomy of query strategies.

Thompson, W. R. (1933). On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika, 25(3/4), 285-294.

The original Thompson Sampling paper, foundational for the bandit-based sequential design strategies in Section 46.2.

Gal, Y., Islam, R., & Ghahramani, Z. (2017). Deep Bayesian Active Learning with Image Data. Proceedings of the 34th International Conference on Machine Learning (ICML).

Demonstrates that BALD with Monte Carlo dropout achieves state-of-the-art active learning performance on image classification, connecting Bayesian deep learning to experiment design.

Books & Surveys

Lattimore, T. & Szepesvári, C. (2020). Bandit Algorithms. Cambridge University Press.

The definitive reference on bandit algorithms, providing the theoretical foundations for Thompson Sampling and UCB used throughout Section 46.2.

Chaloner, K. & Verdinelli, I. (1995). Bayesian Experimental Design: A Review. Statistical Science, 10(3), 273-304.

A comprehensive review of Bayesian optimal experimental design, connecting information-theoretic criteria to classical design theory.

Ren, P., et al. (2022). A Survey of Deep Active Learning. ACM Computing Surveys, 54(9), 1-40.

Reviews modern deep active learning methods including BALD, BatchBALD, and learning loss, providing context for the neural network variants discussed in Section 46.1.

Tools & Libraries

BoTorch

A Bayesian optimization library built on PyTorch and GPyTorch, providing acquisition functions including BALD, knowledge gradient, and expected improvement. Central to the recipe in Section 46.3.

modAL

A modular active learning framework for Python built on scikit-learn, used in Section 46.1 for rapid prototyping of query strategies.

PyMC

A probabilistic programming library for Bayesian inference, used in Section 46.2 for posterior computation in sequential design problems.

scikit-learn

Provides Gaussian process implementations, ensemble methods for query by committee, and utility functions used throughout this chapter.

Scientific Applications

Merchant, A., et al. (2023). Scaling deep learning for materials discovery. Nature, 624, 80-85.

GNoME's materials discovery pipeline uses active learning to select which crystal structures to validate with DFT calculations, a real-world instance of the design loop built in Section 46.3.

Shields, B. J., et al. (2021). Bayesian reaction optimization as a tool for chemical synthesis. Nature, 590, 89-96.

Applies Bayesian optimization with active learning to optimize chemical reaction conditions, demonstrating the practical impact of the methods in this chapter.