"I have spent centuries watching hypotheses come and go. The ones that survive are never the cleverest; they are the ones that had the decency to be testable."
A Hypothesis That Refused to Be Falsified
Overview
Before we can build machines that discover, we need to understand what discovery means in practice. This chapter traces the logic of scientific inquiry from observation through theory revision, formalizes belief updating with Bayesian probability, confronts the reproducibility crisis that threatens modern science, and closes with a hands-on recipe for comparing competing hypotheses computationally. By the end you will have both the conceptual vocabulary and the working code to evaluate scientific claims the way a careful researcher would.
Chapter 1 introduced discovery as search through a space of possible explanations. This chapter sharpens that idea: scientific discovery is search guided by evidence, constrained by falsifiability, and quantified by probability. The scientific method is the oldest and most successful algorithm for navigating explanation space, and Bayesian inference is its mathematical backbone. Understanding both is essential before we formalize knowledge representation in Chapter 3 or build reasoning engines in Chapter 4.
Prerequisites
This chapter assumes comfort with basic probability (conditional probability, Bayes' rule at the formula level) and Python fluency (functions, NumPy arrays, matplotlib). If you need a refresher on probability notation, Appendix A has you covered. You should have read Chapter 1 for the "discovery as search" framing that this chapter builds on.
What You Will Learn
- The five stages of the scientific method and why each matters for automated discovery
- Popper's falsificationism and its implications for hypothesis search
- How to update beliefs with Bayes' theorem, choose priors, and interpret posteriors
- Posterior predictive checking as a model validation strategy
- The reproducibility crisis: causes, consequences, and computational mitigations
- How to build a Bayesian hypothesis comparison notebook with PyMC and ArviZ
Sections
2.1 The Scientific Method Revisited
Observation, hypothesis, prediction, experiment, revision. Falsification and Popper. Why the scientific method is both humanity's greatest algorithm and frustratingly hard to automate.
2.2 Bayesian Science
Prior beliefs, evidence, posteriors. Posterior predictive checking. Model comparison via Bayes factors. Conjugate priors and the likelihood ratio test. Calibration and expected calibration error.
2.3 Reproducibility and Measurement
The replication crisis, measurement uncertainty, construct validity. Statistical power, p-hacking, and the garden of forking paths. Computational tools for reproducible science.
2.4 Building a Bayesian Hypothesis Tester
Recipe: a complete Bayesian hypothesis comparison notebook with PyMC, ArviZ, and Bayes factors. From data loading through model specification, sampling, diagnostics, and interpretation.
What's Next
With the scientific method and Bayesian reasoning in hand, Chapter 3: Knowledge Representation asks the next natural question: how do we encode the knowledge that scientific discovery produces? You will meet ontologies, knowledge graphs, embeddings, and the data structures that let machines reason about what science has found. The Bayesian updating machinery from Section 2.2 will reappear in Chapter 32 at full scale, and the reproducibility concerns from Section 2.3 will shape the experiment registries of Chapter 47.
Bibliography
Foundational Works
The classic statement of falsificationism. Popper argues that science progresses not by confirming theories but by attempting to refute them. Essential background for understanding why hypothesis testing is structured the way it is.
Introduced "paradigm shifts" and challenged the idea of purely cumulative scientific progress. Section 2.1 draws on Kuhn's observation that normal science and revolutionary science require different computational strategies.
The definitive argument for probability as an extension of logic. Chapters 1 through 4 provide the philosophical grounding for the Bayesian approach used throughout this book.
Bayesian Methods
A practical guide to Bayesian modeling with PyMC. The code in Section 2.4 follows conventions from this text. Excellent companion for readers who want more depth on ArviZ diagnostics.
The foundational paper for PyMC. Describes the NUTS sampler and the model specification API that Section 2.4 uses directly.
Reproducibility
The landmark study that attempted to replicate 100 psychology experiments and found that fewer than 40% replicated. Section 2.3 uses this as its central case study.
Extended the replication crisis to high-profile social science, finding roughly 60% replication success. Useful for calibrating how severe the problem is.
Tools and Libraries
The primary tool used in the hands-on recipe. Modern PyMC (v5+) uses PyTensor as its backend and provides state-of-the-art MCMC and variational inference.
The visualization and diagnostics library for Bayesian inference. ArviZ provides trace plots, posterior predictive checks, WAIC, LOO, and the comparison plots used in Section 2.4.
Practical guidance on prior selection. Argues that priors should be evaluated by their implied predictions, which is exactly what posterior predictive checking does.