Overview
Scientific progress depends on asking the right questions. A well-formed hypothesis focuses experimental effort, constrains the search space, and makes failure informative. Yet the generation of hypotheses has traditionally been treated as a purely human act of creativity, resistant to computational assistance. This chapter challenges that assumption. We build systems that identify gaps in existing knowledge, transfer insights across domains by analogy, generate candidate hypotheses using large language models, and score those candidates along three dimensions: plausibility (does it fit what we know?), novelty (does it tell us something new?), and testability (can we design an experiment to confirm or refute it?).
The approach is firmly grounded in the knowledge infrastructure we have built across Part IV. Chapter 36: Literature Mining taught us to extract structured claims from scientific text. Chapter 37: Retrieval Augmented Discovery showed how to retrieve relevant context from large corpora. Chapter 38: Knowledge Graph Discovery gave us graph-structured representations of entities and relations. Now we use all three as inputs to a hypothesis generation pipeline. The hypotheses produced here feed forward into Chapter 40: Research Agents, which can autonomously pursue them, and Chapter 41: Scientific Claim Validation, which evaluates their truth status.
We also connect backward to the foundations: hypothesis generation is a form of the search problems introduced in Chapter 1, and the scoring functions we develop here parallel the Bayesian reasoning of Chapter 32. The creative analogy mechanisms draw on the representation learning of Chapter 26, where embeddings capture semantic similarity that enables cross-domain transfer.
Prerequisites
Readers should be comfortable with Chapter 38: Knowledge Graph Discovery for graph traversal and link prediction concepts, and Chapter 37: Retrieval Augmented Discovery for embedding-based retrieval. Familiarity with LLM prompting from Chapter 10: Prompting to Programming is essential for Section 39.2. Basic probability and Bayesian concepts from Appendix A and Chapter 32 help with the scoring formulations but are not strictly required.
Learning Outcomes
- Identify knowledge gaps in a scientific domain by analyzing structural holes in knowledge graphs and sparse regions in embedding space.
- Implement analogical transfer between scientific domains using embedding alignment and relation mapping.
- Design prompting strategies that elicit novel, domain-grounded hypotheses from large language models.
- Score hypotheses along three orthogonal dimensions: plausibility, novelty, and testability.
- Quantify novelty as embedding distance from known claims and testability as estimated experiment cost.
- Build a complete gap-based hypothesis generation pipeline with multi-dimensional scoring and human-in-the-loop refinement.
- Use the Anthropic and OpenAI APIs, Qdrant, NetworkX, and PyMC to implement production hypothesis generation systems.
Sections
39.1 Gap Analysis and Analogical Transfer
Structural hole detection in knowledge graphs. Embedding-space gap analysis. Analogical reasoning across scientific domains. Relation mapping and cross-domain transfer. The computational geometry of what we do not yet know.
39.2 AI-Generated Hypotheses
LLM prompting strategies for hypothesis generation. Structured output with plausibility, novelty, and testability scoring. Novelty as embedding distance. Testability as experiment cost estimation. Human-AI refinement loops.
39.3 Building a Hypothesis Generator
Recipe: a complete gap-based hypothesis generation pipeline. Knowledge graph gap detection, LLM generation with retrieval augmentation, multi-dimensional scoring, Bayesian plausibility estimation, and integration with the Discovery Workbench.
Bibliography
Foundational Papers
The foundational work on computational scientific discovery, introducing the BACON, DALTON, and GLAUBER systems that automated hypothesis generation from numerical data.
Demonstrates that AI systems trained on the structure of scientific knowledge can suggest novel hypotheses that human scientists would not typically consider, accelerating discovery.
Showed that word embeddings trained on materials science abstracts implicitly encode knowledge sufficient to predict future discoveries, a landmark in embedding-based hypothesis generation.
Systematic evaluation of LLM-generated hypotheses across multiple scientific domains, establishing benchmarks for plausibility and novelty scoring.
Methods and Frameworks
The pioneering work on literature-based discovery, demonstrating that connecting disjoint literatures can reveal implicit hypotheses (the famous fish oil and Raynaud's disease connection).
The structural holes theory from social network analysis, adapted in this chapter for knowledge graph gap detection.
Survey of computational analogy models including SME (Structure-Mapping Engine) and their role in scientific creativity.
Tools and Libraries
The Anthropic API used in this chapter for structured hypothesis generation with tool use and constrained outputs.
The vector database used for embedding-based novelty scoring and nearest-neighbor retrieval over scientific claim embeddings.
The graph analysis library used for structural hole detection and knowledge graph traversal in the hypothesis generation pipeline.
The Bayesian modeling framework used for plausibility estimation via posterior probability computation over hypothesis spaces.
Surveys and Reviews
Comprehensive review of AI for scientific discovery, with a detailed section on hypothesis generation and evaluation methods.