Overview
In Chapter 40, we built research agents that handle individual phases of scientific work: literature search, coding, analysis, and review. This chapter takes the next step: systems that close the entire research loop autonomously. An AI scientist does not merely assist a human researcher; it proposes hypotheses, writes code to test them, runs experiments, evaluates results, critiques its own findings, and produces a written report. The human role shifts from executor to supervisor.
We trace the rapid evolution of these systems from Sakana AI's "The AI Scientist" ($15 per machine-learning paper, August 2024) through its v2 successor (agentic tree search, ICLR 2025 workshop acceptance) to systems that reach beyond ML: Coscientist for autonomous chemistry (Nature 2023), Google's AI Co-Scientist for biomedical hypothesis generation (2025), and DeepMind's FunSearch for mathematical discovery (Nature 2023). Each system reveals a different answer to the central design question: how much autonomy can you safely give an AI research agent, and what architectural guardrails keep it productive?
The chapter concludes with a hands-on recipe for building a supervised AI scientist, a system that runs the full research loop (hypothesis, code, experiment, review) under human oversight at every critical juncture. This recipe integrates the Claude Code SDK for agentic coding, MLflow for experiment tracking, and PaperQA2 for literature grounding, assembling components from across the book into a single coherent pipeline.
Prerequisites
You should have read Chapter 40: Research Agents for the four canonical agent roles and multi-agent coordination patterns, Chapter 39: Hypothesis Generation for gap analysis and analogical transfer, and Chapter 17: Multi-Agent Software Teams for debate loops and handoff protocols. Familiarity with experiment tracking from Chapter 47: Experiment Registries and automated experiment design from Chapter 46 will strengthen your understanding of Section 53.4.
Learning Outcomes
- Describe the automated research loop (propose, implement, run, evaluate, critique, report) and identify which phases current systems handle well and which remain fragile.
- Compare the architectures of AI Scientist v1, AI Scientist v2, Coscientist, Google AI Co-Scientist, and FunSearch, explaining the tradeoffs each makes between autonomy and safety.
- Explain how novelty filters use embedding distance to prevent an AI scientist from rediscovering known results.
- Implement a supervised AI scientist pipeline with human gates at hypothesis approval, experiment approval, and publication approval.
- Use the Claude Code SDK for agentic experiment coding, MLflow for run tracking, and PaperQA2 for literature grounding within the pipeline.
- Evaluate the current limitations of AI scientists (ML-domain bias, narrow novelty, reproducibility gaps) and design mitigations for each.
Sections
53.1 Automated Research Loops
The six-phase research loop: propose, implement, run, evaluate, critique, report. Formalizing what it means for a system to "do science." Novelty detection through embedding distance. The spectrum from human-driven to fully autonomous research. Why current AI scientists are confined to domains with cheap, deterministic experiments. Comparison with the hypothesis generation pipeline from Chapter 39 and the experiment design loop from Chapter 46.
53.2 AI Scientist v1 and v2
Sakana AI's "The AI Scientist" (August 2024): template-based ML research at $15 per paper. Architecture: idea generation, experiment coding, result analysis, LaTeX writing, automated peer review. AI Scientist v2 (2025): agentic tree search over the research space, open-ended experimentation beyond templates, ICLR 2025 workshop acceptance. FunSearch (DeepMind, Nature 2023): evolutionary search over programs guided by LLM mutation. Lessons from each system's failures and successes.
53.3 Coscientist and AI Co-Scientist
Coscientist (Nature 2023): LLM-driven autonomous chemical synthesis with hardware integration. Google AI Co-Scientist (2025): generation, reflection, ranking, evolution, and proximity agents for biomedical hypothesis generation. Debate as a quality filter. Meta-review for consistency checking. The spectrum from virtual-only to lab-integrated AI scientists. Safety architectures for systems that control physical equipment.
53.4 Building a Supervised AI Scientist
Recipe: a supervised AI scientist with human gates. Architecture: hypothesis generation agent, coding agent (Claude Code SDK), experiment runner (subprocess sandboxing), evaluation agent, review agent, and report generator. Novelty filter implementation. Integration with MLflow for experiment tracking and PaperQA2 for literature grounding. Testing the pipeline on a toy discovery task. Discovery Workbench integration.
Bibliography
AI Scientist Systems
Introduces the first end-to-end AI scientist system that generates research ideas, writes code, runs experiments, and produces complete papers with automated peer review, all for approximately $15 per paper. Demonstrates the template-based approach to automated ML research.
Extends AI Scientist v1 with agentic tree search over the research space, removing the template constraint and enabling open-ended experimentation. Achieves acceptance at an ICLR 2025 workshop, the first AI-generated paper to pass peer review at a top venue.
Introduces Coscientist, the first LLM system to autonomously plan and execute wet-lab chemical experiments. Integrates internet search, documentation lookup, code execution, and robotic hardware control into a unified agent loop.
A multi-agent system with generation, reflection, ranking, evolution, and proximity agents that produces novel biomedical hypotheses through iterative debate and tournament selection. Validated with wet-lab experiments across drug repurposing, antimicrobial resistance, and rare diseases.
Introduces FunSearch, which uses an LLM to evolve programs that solve open problems in mathematics. Discovers new constructions for the cap set problem and online bin packing that surpass previously known results.
Automated Research and Evaluation
PaperQA2 achieves superhuman performance on scientific literature comprehension tasks. Provides the literature grounding component used in Section 53.4's supervised AI scientist pipeline.
A controlled study comparing LLM-generated research ideas against expert-generated ones. LLM ideas are rated as more novel but less feasible, establishing key benchmarks for evaluating AI scientist output quality.
Proposes evaluation metrics and benchmarks for automated scientific discovery systems, covering idea quality, experiment validity, and report coherence. Used in Section 53.1 for the evaluation framework.
Tools and Frameworks
SDK for programmatic agentic coding with Claude. Enables the experiment coding agent in Section 53.4 to write, debug, and iterate on experiment scripts within a sandboxed environment.
Experiment tracking, model registry, and deployment platform. Provides the provenance layer for the supervised AI scientist pipeline, logging every hypothesis, experiment run, and evaluation result.
Graph-based framework for stateful agent workflows. Used in Section 53.4 to orchestrate the multi-phase research loop with conditional transitions and human-in-the-loop gates.
Lightweight multi-agent framework with typed handoffs, guardrails, and tracing. An alternative orchestration layer for the AI scientist pipeline discussed in Section 53.4.