Overview
Science advances by proposing hypotheses and testing them against evidence. The Bayesian framework makes this cycle explicit and quantitative: a prior encodes what we believe before seeing data, a likelihood connects the hypothesis to observable quantities, and the posterior tells us what we should believe after the evidence arrives. This chapter builds the complete Bayesian discovery toolkit, from specifying models that encode competing scientific hypotheses, through the computational machinery for computing posteriors (MCMC, variational inference, normalizing flows), to the decision-theoretic tools that tell us which experiment to run next.
What makes Bayesian methods uniquely valuable for discovery is their treatment of uncertainty as a first-class object. A point estimate tells you where the answer probably is; a posterior distribution tells you how much you should trust that answer, where the remaining ambiguity lies, and what observation would most efficiently resolve it. For scientific applications, this distinction is critical: reporting a drug effect size of 0.3 is very different from reporting a posterior that places 90% of its mass between 0.1 and 0.5 versus one that places 90% between 0.29 and 0.31.
The ideas here build on the probability foundations from Appendix A and the data modeling concepts from Chapter 5. They connect backward to the causal inference methods of Chapter 31 (Bayesian networks are a special case of causal graphs) and forward to the scientific machine learning of Chapter 33 (where Bayesian calibration meets physics-informed models). The automated experiment design of Chapter 46 relies heavily on the value-of-information calculations we develop here.
Prerequisites
Readers should be comfortable with probability distributions, conditional probability, and Bayes' theorem from Appendix A: Mathematical Foundations. Familiarity with gradient-based optimization from Chapter 26: Representation Learning will help with the variational inference material. The Chapter 5 treatment of statistical models provides useful context. No prior exposure to MCMC or PyMC is assumed.
Learning Outcomes
- Formulate scientific hypotheses as Bayesian models with explicit priors, likelihoods, and posterior quantities of interest.
- Implement and diagnose MCMC sampling (including NUTS) for posterior inference in PyMC and NumPyro.
- Apply variational inference and normalizing flows as scalable alternatives to MCMC, understanding the trade-offs.
- Compare competing models using WAIC, LOO-CV, and Bayes factors, and interpret the results for scientific decision making.
- Compute expected information gain to determine which experiment or observation would be most valuable.
- Use ArviZ for posterior diagnostics, calibration checking, and publication-quality visualization.
Sections
32.1 The Bayesian Workflow
Bayesian models as scientific hypotheses. Choosing priors: uninformative, weakly informative, and domain-informed. Likelihoods for common data types. Prior predictive checks. The iterative model-building cycle with PyMC and ArviZ.
32.2 Posterior Inference Methods
MCMC fundamentals: Metropolis-Hastings, Hamiltonian Monte Carlo, and the NUTS sampler. The geometry behind NUTS. Variational inference with ADVI. Normalizing flows for amortized inference. Diagnostics: R-hat, ESS, divergences.
32.3 Model Comparison and Decision Making
WAIC, LOO-CV, and Bayes factors for comparing competing scientific hypotheses. Calibration: does your model's uncertainty match reality? Value of information: deciding which experiment to run next.
32.4 Building a Bayesian Discovery Model
Recipe: two competing hypotheses about enzyme kinetics, full PyMC inference, LOO-CV comparison, expected information gain calculation, and integration with the Discovery Workbench.
Bibliography
Foundational Papers
The definitive tutorial on HMC and NUTS geometry, explaining why Hamiltonian dynamics make MCMC efficient in high dimensions. Essential reading for Section 32.2.
The NUTS paper that made HMC practical by eliminating manual tuning of path length, now the default sampler in PyMC and Stan.
Comprehensive review connecting variational inference to classical statistics, covering ADVI and its relationship to MCMC.
The foundational paper on PSIS-LOO-CV for Bayesian model comparison, the method implemented in ArviZ and used throughout Section 32.3.
Books
The standard graduate reference for Bayesian statistics, covering theory, computation, and applications. Chapter 7 on model checking is particularly relevant.
A practical, code-first introduction to Bayesian modeling with PyMC, ArviZ, and related tools. Excellent companion to this chapter.
Tools & Libraries
The primary probabilistic programming framework used in this chapter, built on PyTensor with NUTS sampling and ADVI.
JAX-based probabilistic programming with hardware acceleration. Used in Section 32.2 for GPU-accelerated MCMC and in Section 32.4 for the full recipe.
The visualization and diagnostics library for Bayesian inference, providing R-hat, ESS, LOO-CV, and publication-quality plots.
Combines normalizing flows with MCMC for efficient sampling of multimodal posteriors, particularly useful in astrophysics and gravitational-wave inference.
Tutorials & Surveys
The authoritative guide to iterative Bayesian model building: prior predictive checks, fake-data simulation, posterior predictive checks, and model comparison. The workflow that structures Section 32.1.
Comprehensive survey of normalizing flows, covering architectures, training, and applications to variational inference and density estimation.
Survey of simulation-based inference connecting Bayesian methods to scientific simulators, bridging this chapter to Chapter 43 on scientific simulation.