53.2 AI Scientist v1 and v2

1. AI Scientist v1: Template-Based ML Research

What happens when you hand an LLM a working ML codebase, a GPU budget of \$15, and the instruction "find something publishable"? The original AI Scientist (Lu et al., 2024) answers with a fixed template structure. The user supplies a research template: a working codebase for a specific ML problem (diffusion models, language modeling, or learning dynamics), a description of the research area, and baseline results. The system generates variations, executes them, and writes up the findings.

A research template, in this context, is a self-contained, runnable ML project: training scripts, data loaders, evaluation code, baseline metrics, and a skeleton LaTeX paper. The agent modifies a single component (the noise schedule, the optimizer) while the rest of the pipeline keeps working. This constraint converts "do ML research" into "vary one piece of a working system and measure the effect," which is why the \$15 price tag works. Because the template already trains and evaluates correctly, the agent only proposes and implements a delta rather than architecting an entire experiment from scratch. Use a template-based approach to explore incremental variations reliably and cheaply; switch to an open-ended agent (like v2) when the research question itself is unclear or spans multiple codebases.

1.1 Architecture

AI Scientist v1 consists of five sequential stages, each implemented as a prompted large language model (LLM) call (Claude 3.5 Sonnet, GPT-4o, or Llama 3.1 405B in their experiments). Figure 53.1 shows this pipeline end to end.

1. Idea Generation. The LLM receives the template description and baseline results, then generates a list of research ideas. Each idea is a structured JSON object with fields for the hypothesis, proposed experiment modifications, and expected outcomes. A novelty check queries Semantic Scholar (an academic search engine indexing millions of papers) to filter out ideas that duplicate existing work.
2. Experiment Implementation. The LLM modifies the template code to implement the proposed idea. It uses Aider (an AI coding assistant) in a loop: write code, run tests, fix errors, repeat. The maximum number of coding iterations is capped at 5.
3. Experiment Execution. The modified code runs on a single graphics processing unit (GPU). Results (metrics, plots, logs) are captured automatically. If the run crashes, the system returns to the implementation phase for debugging.
4. Paper Writing. The LLM generates a LaTeX paper following the standard ICML/NeurIPS format. It fills in each section (introduction, related work, methods, experiments, conclusion) using the template's base paper as a scaffold, replacing the results and analysis with the new findings.
5. Automated Peer Review. A separate LLM call evaluates the generated paper on a 1-10 scale across multiple criteria: soundness, significance, novelty, clarity, and overall quality. This mimics the structure of a real conference review form.

1.2 Results and Failure Modes

In their evaluation, AI Scientist v1 generated over 100 papers across three ML domains. The automated peer review scores ranged from 3 to 8 on a 10-point scale, with a mean of approximately 4.5 (below typical conference acceptance thresholds of 6+). Key failure modes include:

• Comparison fabrication. The system sometimes generates comparison numbers that do not correspond to actual experimental runs. It may hallucinate baseline results or misreport its own numbers.
• Overclaiming. The paper-writing stage tends to describe modest improvements in grandiose language, claiming "significant advances" for 0.5% accuracy gains.
• Code-result mismatch. When the experiment code contains bugs that do not cause crashes (silent errors), the system cannot detect the problem and writes a paper about incorrect results.
• Scope creep. The system sometimes proposes modifications that are orthogonal to the template's research question, producing papers that are technically valid but scientifically unfocused.

The following code demonstrates the core idea generation step from AI Scientist v1, showing how hypotheses are generated and filtered. The example template uses a DDPM (Denoising Diffusion Probabilistic Model), a generative model that learns to reverse a gradual noising process to produce images from pure noise.

import json
from anthropic import Anthropic


def generate_research_ideas(
    template_description: str,
    baseline_results: dict,
    existing_ideas: list[str],
    n_ideas: int = 5,
) -> list[dict]:
    """Generate research ideas in the style of AI Scientist v1.

    Returns structured ideas with hypothesis, modifications,
    and expected outcomes.
    """
    client = Anthropic()

    prompt = f"""You are an AI research scientist. Given the following
research template and baseline results, generate {n_ideas} novel
research ideas.

TEMPLATE DESCRIPTION:
{template_description}

BASELINE RESULTS:
{json.dumps(baseline_results, indent=2)}

EXISTING IDEAS (do not duplicate these):
{json.dumps(existing_ideas, indent=2)}

For each idea, provide a JSON object with:
- "title": concise title (under 15 words)
- "hypothesis": testable scientific hypothesis
- "modifications": list of specific code changes needed
- "expected_outcome": what you expect to observe
- "risk_level": "low", "medium", or "high"
- "estimated_compute": approximate GPU hours needed

Return a JSON array of {n_ideas} ideas, ordered by expected
impact (highest first). Only propose ideas that can be tested
by modifying the existing template code."""

    response = client.messages.create(
        model="claude-sonnet-4-20250514",
        max_tokens=4096,
        messages=[{"role": "user", "content": prompt}],
    )

    # Parse the JSON response
    ideas = json.loads(response.content[0].text)

    # Filter by novelty (simplified; v1 uses Semantic Scholar)
    novel_ideas = [
        idea for idea in ideas
        if idea["title"].lower() not in
        [e.lower() for e in existing_ideas]
    ]

    return novel_ideas[:n_ideas]


# Example usage with a diffusion model template
ideas = generate_research_ideas(
    template_description=(
        "2D diffusion models on low-resolution image datasets. "
        "The template implements a DDPM with a U-Net backbone, "
        "trained on CIFAR-10 with cosine noise schedule."
    ),
    baseline_results={
        "fid_score": 12.4,
        "training_time_hours": 2.5,
        "parameters": "35M",
    },
    existing_ideas=[
        "Adaptive noise schedules for faster convergence",
        "Classifier-free guidance at low resolution",
    ],
)
for idea in ideas:
    print(f"  {idea['title']}: {idea['hypothesis']}")

2. AI Scientist v2: Agentic Tree Search

These failure modes all trace back to the same root cause: v1's rigid template locks the agent into a narrow corridor with no way to recover from a wrong turn, and no independent check on whether the results are real.

AI Scientist v2 (Yamada et al., 2025) addresses v1's key limitation: the rigid template structure. Instead of requiring a pre-built codebase, v2 gives the agent access to a coding environment and lets it explore the research space freely using agentic tree search.

2.1 Tree Search Over Research Space

Without structured exploration, an AI research agent that takes a wrong turn early (a flawed hypothesis, a buggy baseline) has no way to recover; it plows forward, wastes its entire compute budget, and produces a paper about artifacts. Tree search gives the agent the ability to backtrack, and that single capability is what separates a \$15 dead end from a workshop-accepted contribution.

The core insight of v2 treats research as a tree search problem. Each node represents a research state: an idea, a partial experiment, or a set of results. Each edge represents a research action: proposing a modification, writing code, running an experiment, or analyzing results. The agent explores this tree using a variant of Monte Carlo Tree Search (MCTS) adapted for scientific discovery.

Mental Model

Think of MCTS over the research space like a chef developing a new recipe. The chef starts with a broad concept (say, a chocolate dessert), then branches into options: cake, mousse, or tart. She tastes a quick prototype of the cake (that is the "simulation" step), finds it promising, and branches further: buttercream vs. ganache, adding espresso vs. orange zest. Each tasting updates her sense of which branch is worth exploring next. If the mousse prototype flopped, she does not keep refining it; she backtracks to the tart. The UCB formula is the chef's intuition formalized: it ensures she does not obsessively perfect one branch (exploitation) while ignoring branches she has barely tasted (exploration). Just as the chef allocates limited evening hours across tastings, the AI Scientist v2 agent allocates a fixed compute budget across research directions, always choosing the next experiment that maximizes the chance of finding something genuinely good.

The search tree has three levels: Figure 53.2.1 illustrates AI Scientist v2 agentic tree search over research space.

1. Idea-level nodes. Each represents a research direction. The agent can expand an idea node by proposing sub-ideas or modifications.
2. Experiment-level nodes. Each represents a specific experiment to test an idea. The agent can expand by modifying experimental parameters or adding controls.
3. Result-level nodes. Each represents an experimental outcome. The agent uses these to decide whether to continue exploring the current branch or backtrack to try a different idea.

The search uses an Upper Confidence Bound (UCB) variant to balance exploration (trying new ideas) against exploitation (refining promising ideas):

where \(\bar{Q}(n)\) is the average quality score of results under node \(n\) (using the composite quality metric from Section 53.1), \(N(n)\) is the visit count for node \(n\), \(N(p)\) is the visit count for the parent node, and \(c\) is the exploration constant.

2.2 From Templates to Open-Ended Research

The v2 agent has access to a full development environment: it can create files, install packages, write arbitrary code, and execute experiments. The environment is containerized for safety (see the sandboxing discussion in Section 53.4). This open-endedness comes with a risk: the agent might spend all its compute budget on infrastructure setup rather than research. V2 mitigates this with two mechanisms:

• Compute budgets per tree level. Idea-level exploration (brainstorming, literature search) is cheap. Experiment-level work (coding, debugging) is more expensive. Result-level work (training, evaluation) is most expensive. The system allocates budgets proportionally.
• Early termination heuristics. If an experiment's training loss is not decreasing after 20% of the allocated epochs, or if the code has crashed more than 3 times during implementation, the system prunes that branch and backtracks.

The following code shows a simplified version of v2's tree search over the research space.

from dataclasses import dataclass, field
import math
import random


@dataclass
class ResearchNode:
    """A node in the research exploration tree."""
    idea: str
    level: str  # "idea", "experiment", "result"
    quality_scores: list[float] = field(default_factory=list)
    children: list["ResearchNode"] = field(default_factory=list)
    visit_count: int = 0
    parent: "ResearchNode | None" = None

    @property
    def avg_quality(self) -> float:
        if not self.quality_scores:
            return 0.0
        return sum(self.quality_scores) / len(self.quality_scores)

    def ucb(self, exploration_c: float = 1.41) -> float:
        """Upper Confidence Bound for research exploration."""
        if self.visit_count == 0:
            return float("inf")  # unexplored nodes first
        parent_visits = (
            self.parent.visit_count if self.parent else 1
        )
        exploitation = self.avg_quality
        exploration = exploration_c * math.sqrt(
            math.log(parent_visits) / self.visit_count
        )
        return exploitation + exploration


class ResearchTreeSearch:
    """Simplified AI Scientist v2 tree search."""

    def __init__(
        self,
        root_topic: str,
        max_iterations: int = 50,
        max_depth: int = 3,
    ):
        self.root = ResearchNode(
            idea=root_topic, level="idea"
        )
        self.max_iterations = max_iterations
        self.max_depth = max_depth

    def select(self, node: ResearchNode) -> ResearchNode:
        """Select the most promising child by UCB."""
        if not node.children:
            return node
        best = max(node.children, key=lambda c: c.ucb())
        if best.children and random.random() > 0.3:
            return self.select(best)
        return best

    def expand(
        self, node: ResearchNode, generate_fn
    ) -> ResearchNode:
        """Expand a node by generating a child idea."""
        child_ideas = generate_fn(
            parent_idea=node.idea,
            existing_children=[c.idea for c in node.children],
        )
        if not child_ideas:
            return node
        next_level = {
            "idea": "experiment",
            "experiment": "result",
            "result": "result",
        }
        child = ResearchNode(
            idea=child_ideas[0],
            level=next_level[node.level],
            parent=node,
        )
        node.children.append(child)
        return child

    def backpropagate(
        self, node: ResearchNode, quality: float
    ) -> None:
        """Propagate quality score up the tree."""
        current = node
        while current is not None:
            current.quality_scores.append(quality)
            current.visit_count += 1
            current = current.parent

    def search(self, generate_fn, evaluate_fn) -> ResearchNode:
        """Run the tree search and return the best node."""
        for _ in range(self.max_iterations):
            # Selection
            selected = self.select(self.root)
            # Expansion
            if selected.level != "result":
                expanded = self.expand(selected, generate_fn)
            else:
                expanded = selected
            # Evaluation (simulation in MCTS terms)
            quality = evaluate_fn(expanded.idea)
            # Backpropagation
            self.backpropagate(expanded, quality)

        # Return the highest-quality leaf
        return self._best_leaf(self.root)

    def _best_leaf(self, node: ResearchNode) -> ResearchNode:
        if not node.children:
            return node
        best_child = max(
            node.children, key=lambda c: c.avg_quality
        )
        return self._best_leaf(best_child)

3. FunSearch: Evolving Programs for Mathematical Discovery

Both AI Scientist versions frame discovery as producing papers, but a paper is only a report of a finding, not the finding itself; what if the system's output were the discovery directly?

FunSearch (Romera-Paredes et al., 2024) takes a fundamentally different approach from the AI Scientist systems. Rather than generating papers, FunSearch generates programs. Rather than using tree search, FunSearch uses evolutionary computation (an optimization strategy that maintains a population of candidate solutions, iteratively selecting the fittest and generating new candidates through variation). The result is a system that discovered genuinely novel mathematical constructions, published in Nature.

3.1 Architecture

FunSearch combines an LLM (Codey, a code-specialized variant of PaLM 2) with an evolutionary algorithm (as of 2025, Google has superseded PaLM 2 with the Gemini model family; AlphaEvolve, discussed below, uses Gemini):

1. Program representation. The search target is a Python function that solves a specific problem (e.g., constructing a large cap set (a subset of vectors in which no three elements sum to zero) in \(\mathbb{F}_3^n\) (the vector space of dimension \(n\) over the three-element field)). The function signature and evaluation criteria are fixed; the function body is what the LLM generates.
2. Population. A pool of candidate programs, ranked by their scores on the evaluation function. The population starts with a simple seed program (often a trivial solution or a known baseline).
3. Mutation via LLM. The LLM receives 2-3 high-scoring programs from the population and generates a new program that combines or improves upon their strategies. This is the creative step: the LLM acts as a mutation operator that can make semantically meaningful changes, unlike random character-level mutation.
4. Evaluation. Each new program is executed and scored. Programs that crash, time out, or produce invalid output are discarded. Valid programs are inserted into the population if they score above a minimum threshold.

5. Island model (a parallel evolutionary strategy borrowed from population genetics, where isolated subpopulations evolve independently and occasionally exchange their best individuals). Multiple independent populations ("islands") evolve in parallel. Periodically, the best programs from each island are shared with other islands. This prevents premature convergence to local optima.

Figure 53.2 illustrates this architecture, showing how multiple island populations feed candidate programs to the LLM mutation operator and receive scored variants back from the evaluator.

3.2 Results and Significance

FunSearch achieved two significant mathematical discoveries:

• Cap set problem. For the cap set problem in \(\mathbb{F}_3^8\), FunSearch discovered constructions larger than any previously known according to the published results, surpassing results obtained by human mathematicians over decades. The discovered construction for \(n = 8\) has 512 elements, improving on the previous best of 496, a record that had stood reportedly for years despite sustained effort by professional mathematicians.
• Online bin packing. FunSearch found new heuristics for online bin packing (where items arrive one at a time and must be irrevocably assigned to a fixed-capacity bin) that outperform hand-designed heuristics on standard benchmarks. The discovered heuristics exploit structural patterns that human designers had not considered.

The critical difference between FunSearch and AI Scientist is the evaluation function. In FunSearch, correctness is verified automatically: a program either produces a valid cap set of the claimed size or it does not. There is no ambiguity, no statistical testing, no subjective judgment. This separation of generation from evaluation makes FunSearch's discoveries mathematically rigorous in a way that AI Scientist's ML papers are not. In short: the safest AI scientist is one whose creativity is checked by an evaluator it cannot fool.

4. Comparative Analysis

With all three architectures on the table, we can now ask what a system builder should actually choose: when does each design pay off, and where does each break down?

The three systems reveal a tradeoff space along three axes: scope, rigor, and autonomy.

5. Lessons for System Builders

From these three systems, we extract five architectural lessons that apply to any AI scientist system:

1. Separate generation from evaluation. FunSearch's success and AI Scientist v1's failure mode (comparison fabrication) both illustrate the same principle: the generator must never evaluate its own output. Use an independent evaluator, preferably one that does not rely on LLM judgment for correctness.
2. Structured exploration beats sequential generation. V2's tree search outperforms v1's sequential idea generation because it can backtrack from dead ends and deepen promising directions. When building your own system, invest in exploration infrastructure.
3. Templates trade novelty for reliability. V1's templates guarantee that the generated code will at least run (the base template works), at the cost of limiting the system to incremental modifications. V2's open-ended approach is more creative but more likely to produce code that crashes or experiments that do not make sense.
4. Budget allocation determines output quality. All three systems must decide how to allocate a fixed compute budget across idea generation, experimentation, and writing. Over-investing in idea generation (many ideas, few tested) produces speculative work. Over-investing in experimentation (few ideas, extensively tested) produces thorough but unambitious work. The right balance depends on the domain.
5. Automated peer review is unreliable. AI Scientist v1's automated reviewer gives scores that correlate weakly with human expert judgment (\(r \approx 0.3\) in their ablation). Never use automated review as the sole quality gate. Section 53.4 adds human gates at critical junctures for this reason.