Chapter 45: Optimization For Discovery

Overview

Scientific discovery rarely reduces to optimizing a single number. A drug candidate must balance efficacy, toxicity, and synthesizability. A catalyst must maximize yield while minimizing cost and environmental impact. An experimental protocol must trade off measurement precision against throughput. These problems share a common structure: the objective function is expensive to evaluate, possibly noisy, potentially multi-valued, and its analytical form is unknown. Classical gradient-based optimization is useless here. You cannot differentiate a wet-lab experiment.

This chapter builds your toolkit for black-box optimization in discovery settings. We start with Bayesian optimization, which constructs a probabilistic surrogate model of the objective and uses acquisition functions to decide where to evaluate next. We then move to evolutionary and multi-objective methods (CMA-ES, NSGA-III) that maintain populations of candidate solutions and navigate Pareto frontiers. We introduce reinforcement learning as a sequential optimization framework where the "objective" unfolds over time. Finally, we integrate these techniques into a working multi-objective optimization pipeline using BoTorch and pymoo.

The search framework from Chapter 1 is the conceptual ancestor of everything here: optimization is search with a quantitative objective. The Bayesian machinery from Chapter 32 provides the probabilistic foundations for Gaussian process surrogates. The differentiable programming techniques from Chapter 42 complement this chapter by covering the case where gradients are available. And the automated experiment design in Chapter 46 builds directly on the optimization methods we develop here.

Prerequisites

This chapter assumes familiarity with probability distributions and Bayes' theorem (Chapter 32), basic linear algebra (matrix inverses, Cholesky decomposition), and Python programming with NumPy. Prior exposure to Gaussian processes is helpful but not required; we derive the key results from scratch. For readers coming from Chapter 42, note that this chapter focuses on the derivative-free case.

Learning Outcomes

Derive Expected Improvement and other acquisition functions from Gaussian process posteriors.
Implement Bayesian optimization from scratch and compare against Optuna and BoTorch.
Explain CMA-ES as natural gradient descent on the space of search distributions.
Apply NSGA-III to multi-objective discovery problems and compute hypervolume indicators.
Frame sequential discovery decisions as reinforcement learning with policy gradients and actor-critic methods.
Build a complete multi-objective optimization pipeline that combines NSGA-III with BoTorch surrogates and evaluates solutions via Pareto hypervolume.

Sections

45.1 Bayesian Optimization

Gaussian process surrogates, kernel functions, posterior inference, and acquisition functions. Expected Improvement derivation. The BO loop for expensive black-box objectives. Practical implementation with BoTorch and Optuna.

45.2 Evolutionary and Multi-Objective Methods

CMA-ES as natural gradient descent on search distributions. NSGA-III for many-objective problems. Pareto dominance, crowding distance, and reference-point decomposition. Hypervolume indicator as the gold-standard metric.

45.3 Reinforcement Learning as Optimization

Sequential decision-making for discovery. Policy gradient theorem and REINFORCE. Actor-critic methods. Framing experiment campaigns as MDPs. When RL beats BO and when it does not.

45.4 Building a Multi-Objective Optimizer

Recipe: end-to-end multi-objective optimization combining NSGA-III with BoTorch GP surrogates. Hypervolume comparison across methods. Integration with the Discovery Workbench.

What's Next

With a full optimization toolkit in hand, Chapter 46: Automated Experiment Design applies these methods to the specific challenge of deciding which experiment to run next. Where this chapter treats the objective as a given black box, Chapter 46 addresses the meta-problem: how to design the experiments that produce the data the optimizer needs. The acquisition functions we derived here become the core of active learning and sequential experimental design.

Bibliography

Foundational Papers

Jones, D. R., Schonlau, M., & Welch, W. J. (1998). Efficient global optimization of expensive black-box functions. Journal of Global Optimization, 13(4), 455-492.

The paper that established Expected Improvement as the standard acquisition function for Bayesian optimization, introducing the EGO algorithm.

Hansen, N. & Ostermeier, A. (2001). Completely derandomized self-adaptation in evolution strategies. Evolutionary Computation, 9(2), 159-195.

The original CMA-ES paper, deriving the covariance matrix adaptation mechanism that makes the algorithm invariant to affine transformations of the search space.

Deb, K. & Jain, H. (2014). An evolutionary many-objective optimization algorithm using reference-point-based nondominated sorting approach, Part I. IEEE Transactions on Evolutionary Computation, 18(4), 577-601.

The NSGA-III paper, extending NSGA-II to handle many-objective problems through reference-point decomposition on the normalized hyperplane.

Williams, R. J. (1992). Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8(3), 229-256.

The REINFORCE algorithm paper that derived policy gradients via the log-derivative trick, foundational for RL-based optimization.

Books & Monographs

Rasmussen, C. E. & Williams, C. K. I. (2006). Gaussian Processes for Machine Learning. MIT Press.

The definitive reference on Gaussian processes, providing the mathematical foundations for the GP surrogates used in Bayesian optimization.

Sutton, R. S. & Barto, A. G. (2018). Reinforcement Learning: An Introduction, 2nd edition. MIT Press.

The standard RL textbook, covering policy gradients and actor-critic methods that Section 45.3 applies to discovery optimization.

Deb, K. (2001). Multi-Objective Optimization Using Evolutionary Algorithms. Wiley.

Kalyanmoy Deb's foundational text on multi-objective evolutionary algorithms, including the NSGA family and Pareto-based selection.

Tools & Libraries

BoTorch

A Bayesian optimization library built on PyTorch and GPyTorch, providing modular acquisition functions, multi-objective optimization, and GPU-accelerated GP inference.

Optuna

A hyperparameter optimization framework with pruning, multi-objective support, and a dashboard. Used throughout this chapter for practical BO workflows.

pymoo

A Python library for multi-objective optimization implementing NSGA-II, NSGA-III, and reference-direction methods with built-in performance indicators.

Ray Tune

A distributed hyperparameter tuning library that integrates with BO backends (Optuna, BoTorch) and scales across clusters, useful for parallelizing expensive evaluations.

DEAP

Distributed Evolutionary Algorithms in Python. A flexible framework for prototyping custom evolutionary optimization algorithms.

Surveys & Tutorials

Frazier, P. I. (2018). A tutorial on Bayesian optimization. arXiv:1807.02811.

An accessible tutorial covering GP surrogates, acquisition functions, and practical considerations for BO, excellent supplementary reading for Section 45.1.

Shahriari, B., Swersky, K., Wang, Z., Adams, R. P., & de Freitas, N. (2016). Taking the human out of the loop: A review of Bayesian optimization. Proceedings of the IEEE, 104(1), 148-175.

A comprehensive review of BO with emphasis on scientific applications, connecting the mathematical framework to real-world discovery pipelines.