40.2 AI Scientists and Coscientist

1. Coscientist: Autonomous Chemical Research

What distinguishes an "AI scientist" from the individual research agents in Section 40.1? The threshold is end-to-end autonomy: a system that can take a research question as input and, with minimal human intervention, execute the full cycle of literature review, hypothesis formation, experiment design, data collection, analysis, and iteration. The three systems in this section each cross that threshold in a different way and at a different scope.

In 2023, a GPT-4-powered system searched the literature for a synthesis procedure, wrote the control code for a robotic liquid handler, ran Suzuki and Sonogashira cross-coupling reactions (palladium-catalyzed methods for forming carbon-carbon bonds, widely used in pharmaceutical and materials synthesis), read the resulting spectra, and decided what to try next, all without a human touching a single flask. That system was Coscientist (Boiko et al., 2023), published in Nature as the first demonstration of a large language model (LLM)-driven agent that autonomously plans and executes real chemical experiments. It used GPT-4 as its reasoning engine, connected to five tool modules that gave it access to the physical world.

Without systems like these, a research team that wants to explore a novel synthesis must manually search the literature, hand-code instrument protocols, and wait for a human to interpret every spectrum before deciding the next step; a single unexpected byproduct can stall progress for days.

In a tool-augmented LLM agent, the language model serves as the central reasoning and planning engine. It delegates all concrete actions (computation, data retrieval, hardware control) to external tool modules through structured interfaces. This architecture matters because it separates what the model is good at (language understanding, planning, reasoning under uncertainty) from what it is bad at (exact arithmetic, factual recall, physical manipulation), letting each component handle the task it does best. The mechanism works as follows: the LLM receives a goal and emits a tool name with arguments as structured output. The system executes that tool, returns the result as text, and the LLM decides the next step. Use a tool-augmented single agent when your workflow is sequential and the tools are heterogeneous; switch to a multi-agent architecture (as in AI Co-Scientist) when you need parallel hypothesis generation or adversarial quality filtering.

Coscientist implements this pattern with a direct architecture. A central LLM planner receives a high-level goal ("Synthesize compound X") and decomposes it into a sequence of tool calls. The five tool modules are:

• Web search: Queries Google and chemical databases to find synthesis procedures, safety data, and reagent information.
• Documentation lookup: Reads technical manuals for laboratory hardware (specifically the Opentrons OT-2 liquid handler).
• Code execution: Runs Python code in a sandboxed environment for calculations, data processing, and experiment planning.
• Hardware control: Sends commands to the Opentrons OT-2 robotic system to physically execute synthesis steps (pipetting, mixing, heating).
• Spectroscopy analysis: Reads and interprets nuclear magnetic resonance (NMR) and ultraviolet-visible (UV-Vis) spectroscopy data to verify that the synthesized product matches expectations.

The critical insight from Coscientist is that the LLM never touches the chemicals. It operates entirely through tool interfaces, making decisions about what to do next based on textual descriptions of instrument readings and experimental outcomes. This separation between reasoning (LLM) and actuation (tools) is the same pattern we saw in the agent loop from Chapter 10, now applied to physical-world experiments. In short: The LLM reasons; the tools act; neither is sufficient alone, but together they close the loop from hypothesis to physical result.

"""
Simplified Coscientist architecture: a planner LLM with tool dispatch.
This captures the core loop without the chemistry-specific details.
"""
from dataclasses import dataclass, field
from typing import Any


@dataclass
class CoscientistState:
    """State maintained across the planning loop."""
    goal: str
    plan: list[dict] = field(default_factory=list)
    executed_steps: list[dict] = field(default_factory=list)
    observations: list[str] = field(default_factory=list)
    status: str = "planning"  # planning, executing, analyzing, complete


class CoscientistAgent:
    """Simplified Coscientist: LLM planner with five tool modules.

    Architecture follows Boiko et al. (2023), Nature 624, 570-578.
    """

    PLANNER_PROMPT = """You are a scientific experiment planner. Given a
    synthesis goal, decompose it into steps using these tools:
    - web_search(query): Search for procedures and safety data.
    - lookup_docs(instrument, topic): Read instrument documentation.
    - run_code(code): Execute Python for calculations.
    - hardware_command(instrument, action, params): Control lab hardware.
    - analyze_spectrum(data_path, spectrum_type): Interpret spectral data.

    Plan step by step. After each tool result, decide whether to proceed,
    revise the plan, or abort if safety concerns arise.
    """

    def __init__(self, llm_client, tools: dict[str, Any]):
        self.llm = llm_client
        self.tools = tools

    async def plan(self, goal: str) -> list[dict]:
        """Generate a multi-step experiment plan."""
        response = await self.llm.chat(
            system=self.PLANNER_PROMPT,
            user=(
                f"Goal: {goal}\n\n"
                "Generate a step-by-step plan as a JSON list. "
                "Each step: {\"tool\": \"name\", \"action\": \"...\", "
                "\"params\": {...}, \"safety_check\": \"...\"}."
            ),
        )
        import json
        return json.loads(response.content)

    async def execute_plan(self, goal: str) -> CoscientistState:
        """Execute the full Coscientist loop: plan, execute, observe, replan."""
        state = CoscientistState(goal=goal)

        # phase 1: initial planning
        state.plan = await self.plan(goal)
        state.status = "executing"

        for step in state.plan:
            # safety gate: check before executing
            if not await self._safety_check(step):
                state.status = "aborted_safety"
                state.observations.append(
                    f"Safety check failed for: {step['action']}"
                )
                break

            # execute the tool
            tool_fn = self.tools.get(step["tool"])
            if tool_fn is None:
                state.observations.append(
                    f"Unknown tool: {step['tool']}"
                )
                continue

            result = await tool_fn(**step["params"])
            state.executed_steps.append({
                **step,
                "result": str(result),
            })
            state.observations.append(str(result))

            # after each step: ask the LLM whether to continue or replan
            should_continue = await self._evaluate_progress(state)
            if not should_continue:
                # replan based on observations
                state.plan = await self._replan(state)

        state.status = "analyzing"

        # phase 2: analyze final results
        state.status = await self._final_analysis(state)
        return state

    async def _safety_check(self, step: dict) -> bool:
        """Verify that a step is safe to execute."""
        response = await self.llm.chat(
            system=(
                "You are a laboratory safety officer. Evaluate whether "
                "this step is safe. Consider: chemical hazards, equipment "
                "limits, and human safety. Return JSON: "
                '{"safe": true/false, "reason": "..."}'
            ),
            user=f"Step: {step}",
        )
        import json
        result = json.loads(response.content)
        return result["safe"]

    async def _evaluate_progress(self, state: CoscientistState) -> bool:
        """Decide whether the current plan is still valid."""
        response = await self.llm.chat(
            system="Evaluate experiment progress. Should we continue?",
            user=(
                f"Goal: {state.goal}\n"
                f"Completed: {len(state.executed_steps)}/{len(state.plan)}\n"
                f"Latest observation: {state.observations[-1]}\n"
                'Return JSON: {"continue": true/false, "reason": "..."}'
            ),
        )
        import json
        return json.loads(response.content)["continue"]

    async def _replan(self, state: CoscientistState) -> list[dict]:
        """Generate a revised plan based on observations."""
        return await self.plan(
            f"{state.goal}\n\nObservations so far:\n"
            + "\n".join(state.observations)
        )

    async def _final_analysis(self, state: CoscientistState) -> str:
        """Analyze whether the experiment achieved its goal."""
        response = await self.llm.chat(
            system="Analyze experiment results.",
            user=(
                f"Goal: {state.goal}\n"
                f"Steps executed: {len(state.executed_steps)}\n"
                f"Final observations: {state.observations[-3:]}\n"
                'Return: "complete" if goal achieved, "partial" if '
                'partially achieved, "failed" if not.'
            ),
        )
        return response.content.strip()

2. ChemCrow: Domain Expert Tool Augmentation

Where Coscientist uses general-purpose tools (web search, code execution) and relies on the LLM's latent chemistry knowledge, ChemCrow (Bran et al., 2024) takes the opposite approach: it augments the LLM with 18 expert chemistry tools that encode domain knowledge directly. The LLM acts as a router, deciding which tool to call, but the heavy lifting (retrosynthesis, where the system works backward from a target molecule to identify available starting materials and reaction steps, property prediction, safety assessment) is done by specialized chemistry software.

ChemCrow's 18 tools span four categories:

Molecular tools: Simplified Molecular-Input Line-Entry System (SMILES) validation, name-to-SMILES conversion, molecular weight calculation, similarity search against chemical databases. These replace the LLM's unreliable mental arithmetic with exact computation.

Reaction tools: Retrosynthesis prediction (using IBM RXN), forward reaction prediction, reaction feasibility scoring. These encode decades of organic chemistry knowledge in callable functions.

Property tools: Solubility prediction, toxicity screening (using Tox21 models), drug-likeness scoring (Lipinski's rules, a set of four molecular property thresholds that predict whether a compound is likely to be orally bioavailable), and ADMET (absorption, distribution, metabolism, excretion, toxicity) prediction. These are trained ML models exposed as tool interfaces.

Safety tools: Explosive potential screening, controlled substance checking, and general safety data sheet (SDS) lookup. These are critical guardrails that prevent the system from proposing dangerous syntheses.

"""
ChemCrow-style tool registry: domain tools exposed to an LLM agent.
Based on Bran et al. (2024), Nature Machine Intelligence 6, 525-535.
"""
from dataclasses import dataclass
from typing import Callable, Any


@dataclass
class ChemTool:
    """A chemistry tool with metadata for the LLM to select from."""
    name: str
    description: str
    category: str  # "molecular", "reaction", "property", "safety"
    function: Callable[..., Any]
    required_params: list[str]
    returns: str


class ChemCrowToolkit:
    """Registry of domain-expert chemistry tools."""

    def __init__(self):
        self.tools: dict[str, ChemTool] = {}

    def register(self, tool: ChemTool):
        """Register a chemistry tool."""
        self.tools[tool.name] = tool

    def get_tool_descriptions(self) -> str:
        """Format tool descriptions for the LLM system prompt."""
        lines = []
        for tool in self.tools.values():
            lines.append(
                f"- {tool.name}: {tool.description}\n"
                f"  Parameters: {tool.required_params}\n"
                f"  Returns: {tool.returns}"
            )
        return "\n".join(lines)

    async def call(self, tool_name: str, **kwargs) -> Any:
        """Execute a tool by name with safety validation."""
        tool = self.tools.get(tool_name)
        if tool is None:
            raise ValueError(f"Unknown tool: {tool_name}")

        # if the tool produces a molecule, run safety screening first
        result = tool.function(**kwargs)

        if tool.category == "reaction":
            safety_result = await self._safety_screen(result)
            if not safety_result["safe"]:
                return {
                    "result": result,
                    "safety_warning": safety_result["reason"],
                    "blocked": True,
                }

        return {"result": result, "blocked": False}

    async def _safety_screen(self, reaction_result) -> dict:
        """Screen reaction products for safety concerns."""
        # in production: call explosive screening, controlled substance
        # check, and toxicity prediction tools
        return {"safe": True, "reason": ""}


def build_chemcrow_toolkit() -> ChemCrowToolkit:
    """Build a ChemCrow-style toolkit with representative tools."""
    toolkit = ChemCrowToolkit()

    # molecular tools
    toolkit.register(ChemTool(
        name="smiles_to_name",
        description="Convert a SMILES string to an IUPAC chemical name",
        category="molecular",
        function=lambda smiles: _lookup_name(smiles),
        required_params=["smiles"],
        returns="IUPAC name as string",
    ))

    toolkit.register(ChemTool(
        name="molecular_weight",
        description="Calculate the molecular weight from a SMILES string",
        category="molecular",
        function=lambda smiles: _calc_mw(smiles),
        required_params=["smiles"],
        returns="Molecular weight in g/mol",
    ))

    # reaction tools
    toolkit.register(ChemTool(
        name="retrosynthesis",
        description=(
            "Predict synthesis routes for a target molecule using "
            "retrosynthetic analysis"
        ),
        category="reaction",
        function=lambda target_smiles: _retro(target_smiles),
        required_params=["target_smiles"],
        returns="List of synthesis routes with reagents and conditions",
    ))

    # property tools
    toolkit.register(ChemTool(
        name="toxicity_screen",
        description="Screen a molecule for toxicity using Tox21 models",
        category="property",
        function=lambda smiles: _tox_screen(smiles),
        required_params=["smiles"],
        returns="Toxicity predictions across 12 Tox21 endpoints",
    ))

    # safety tools
    toolkit.register(ChemTool(
        name="explosive_check",
        description="Check if a molecule has explosive potential",
        category="safety",
        function=lambda smiles: _explosive_check(smiles),
        required_params=["smiles"],
        returns="Risk assessment: low/medium/high with explanation",
    ))

    return toolkit


# placeholder implementations (real ones call RDKit, IBM RXN, etc.)
def _lookup_name(smiles: str) -> str:
    from rdkit import Chem
    mol = Chem.MolFromSmiles(smiles)
    return Chem.MolToSmiles(mol) if mol else "Invalid SMILES"

def _calc_mw(smiles: str) -> float:
    from rdkit.Chem import Descriptors, MolFromSmiles
    mol = MolFromSmiles(smiles)
    return Descriptors.MolWt(mol) if mol else 0.0

def _retro(target: str) -> list[dict]:
    return [{"route": "placeholder", "confidence": 0.0}]

def _tox_screen(smiles: str) -> dict:
    return {"endpoint_results": {}, "overall_risk": "unknown"}

def _explosive_check(smiles: str) -> dict:
    return {"risk": "low", "reason": "No energetic groups detected"}

The architectural lesson from ChemCrow is that domain tools reduce hallucination by replacing generation with computation. When you ask an LLM "What is the molecular weight of aspirin?", it might answer 180.16 g/mol (correct) or 181.42 g/mol (wrong, a common hallucination). When you give the LLM a molecular_weight tool that calls RDKit, it always gets the exact answer. Bran et al. report that tool-augmented performance on chemistry tasks exceeds bare GPT-4 by 20 to 40 percentage points, depending on the task category.

3. Google AI Co-Scientist: Debate and Tournament Ranking

Google's AI Co-Scientist (Gottweis et al., 2025) represents a fundamentally different architecture. Rather than a single agent with tools, it uses a multi-agent system where specialized agents generate, critique, and rank scientific hypotheses through an iterative debate process inspired by evolutionary computation.

The system comprises several specialized agent roles:

Generation agents produce candidate hypotheses based on a research goal and the existing literature. Multiple generators run in parallel, each with slightly different prompting or temperature settings, to ensure diversity.

Reflection agents take a generated hypothesis and improve it by identifying weaknesses, filling gaps, and strengthening the reasoning. This is an internal self-critique step before the hypothesis faces external review.

Ranking agents evaluate and compare hypotheses using a tournament-style bracket. Pairs of hypotheses are presented to a judge agent, which selects the stronger one based on novelty, feasibility, and scientific rigor. The tournament produces a ranked list where the top hypotheses advance to the next round.

Mental Model

Think of the tournament ranking like a competitive bake-off. Each baker (generation agent) submits a cake (hypothesis). A panel of food critics (ranking agents) never scores any cake in isolation on a 1-to-10 scale, because absolute scores are unreliable and inconsistent across judges. Instead, two cakes are placed side by side, and the critics say which one tastes better. After many such pairwise tastings, a leaderboard emerges (Elo ratings, where Elo is a numerical rating system originally developed for ranking chess players that updates each competitor's score based on the outcome of pairwise matches) that reliably orders all cakes from best to worst. The top bakers then share recipes and techniques (recombination) to produce the next round's entries. The key insight the analogy maps: comparative judgment between two concrete options tends to be more reliable than absolute scoring, both for human judges and for LLMs (Zheng et al., 2024).

Proximity agents check each hypothesis against the existing literature to assess whether it is genuinely novel or merely restates known findings. This prevents the system from confidently "discovering" something that was published five years ago.

Meta-review agents synthesize feedback across the tournament rounds and produce an overall assessment of the hypothesis landscape: which directions are most promising, which have been adequately explored, and where gaps remain.

The full loop, illustrated in Figure 40.2, operates as follows. Figure 40.2.1 illustrates the AI Co-Scientist multi-agent evolutionary pipeline.

where \(k\) is the generation population size, \(m < k\) is the number of survivors after tournament selection, and the "evolve" step recombines the best hypotheses (analogous to crossover in genetic algorithms) to produce the next generation's seed. This process repeats for a configurable number of generations.

"""
AI Co-Scientist tournament ranking mechanism.
Based on Gottweis et al. (2025), Google DeepMind.
"""
from dataclasses import dataclass, field
import random


@dataclass
class Hypothesis:
    """A candidate scientific hypothesis with metadata."""
    id: str
    content: str
    reasoning: str
    generation: int = 0
    tournament_wins: int = 0
    tournament_losses: int = 0
    novelty_score: float = 0.0
    feasibility_score: float = 0.0
    elo_rating: float = 1200.0  # starting Elo

    @property
    def win_rate(self) -> float:
        total = self.tournament_wins + self.tournament_losses
        return self.tournament_wins / total if total > 0 else 0.5


class TournamentRanker:
    """Rank hypotheses using pairwise tournament with Elo ratings.

    Inspired by the AI Co-Scientist's tournament selection mechanism,
    which uses LLM-as-judge to compare hypothesis pairs.
    """

    def __init__(self, judge_llm, k_factor: float = 32.0):
        self.judge = judge_llm
        self.k_factor = k_factor

    async def run_tournament(
        self,
        hypotheses: list[Hypothesis],
        rounds: int = 3,
    ) -> list[Hypothesis]:
        """Run a multi-round tournament and return ranked hypotheses."""
        for round_num in range(rounds):
            # shuffle and pair
            candidates = list(hypotheses)
            random.shuffle(candidates)

            pairs = []
            for i in range(0, len(candidates) - 1, 2):
                pairs.append((candidates[i], candidates[i + 1]))

            # judge each pair
            for h_a, h_b in pairs:
                winner = await self._judge_pair(h_a, h_b)
                self._update_elo(h_a, h_b, winner)

        # sort by Elo rating (highest first)
        hypotheses.sort(key=lambda h: h.elo_rating, reverse=True)
        return hypotheses

    async def _judge_pair(
        self, h_a: Hypothesis, h_b: Hypothesis
    ) -> str:
        """Ask an LLM judge to compare two hypotheses."""
        import json

        response = await self.judge.chat(
            system=(
                "You are a scientific hypothesis evaluator. Compare two "
                "hypotheses on: (1) novelty, (2) feasibility, (3) potential "
                "impact, (4) testability. Select the stronger one.\n"
                "Return JSON: {\"winner\": \"A\" or \"B\", "
                "\"reasoning\": \"...\", \"scores\": "
                "{\"A\": {\"novelty\": 0-10, ...}, "
                "\"B\": {\"novelty\": 0-10, ...}}}"
            ),
            user=(
                f"Hypothesis A [{h_a.id}]:\n{h_a.content}\n"
                f"Reasoning: {h_a.reasoning}\n\n"
                f"Hypothesis B [{h_b.id}]:\n{h_b.content}\n"
                f"Reasoning: {h_b.reasoning}"
            ),
        )

        result = json.loads(response.content)
        return result["winner"]

    def _update_elo(
        self, h_a: Hypothesis, h_b: Hypothesis, winner: str
    ):
        """Update Elo ratings based on the match outcome."""
        # expected scores
        e_a = 1.0 / (1.0 + 10 ** ((h_b.elo_rating - h_a.elo_rating) / 400))
        e_b = 1.0 - e_a

        # actual scores
        if winner == "A":
            s_a, s_b = 1.0, 0.0
            h_a.tournament_wins += 1
            h_b.tournament_losses += 1
        else:
            s_a, s_b = 0.0, 1.0
            h_b.tournament_wins += 1
            h_a.tournament_losses += 1

        # update ratings
        h_a.elo_rating += self.k_factor * (s_a - e_a)
        h_b.elo_rating += self.k_factor * (s_b - e_b)


class CoScientistPipeline:
    """Complete AI Co-Scientist generate-debate-rank pipeline."""

    def __init__(
        self,
        generator_llm,
        reflector_llm,
        judge_llm,
        proximity_checker,
        population_size: int = 10,
        num_generations: int = 3,
        survival_fraction: float = 0.5,
    ):
        self.generator = generator_llm
        self.reflector = reflector_llm
        self.ranker = TournamentRanker(judge_llm)
        self.proximity = proximity_checker
        self.pop_size = population_size
        self.generations = num_generations
        self.survival_fraction = survival_fraction

    async def run(self, research_goal: str) -> list[Hypothesis]:
        """Run the full generate-reflect-rank evolutionary loop."""
        hypotheses = []

        for gen in range(self.generations):
            # generate new hypotheses
            if gen == 0:
                new_hyps = await self._generate_initial(
                    research_goal, self.pop_size
                )
            else:
                # evolve from survivors
                survivors = hypotheses[
                    : int(len(hypotheses) * self.survival_fraction)
                ]
                new_hyps = await self._evolve(
                    research_goal, survivors, self.pop_size
                )

            # reflect on each hypothesis
            refined_hyps = []
            for h in new_hyps:
                refined = await self._reflect(h)
                refined.generation = gen
                refined_hyps.append(refined)

            # check novelty against literature
            for h in refined_hyps:
                h.novelty_score = await self.proximity.check_novelty(
                    h.content
                )

            # combine with surviving hypotheses from prior generations
            all_candidates = (
                hypotheses[: int(len(hypotheses) * self.survival_fraction)]
                + refined_hyps
            )

            # tournament ranking
            hypotheses = await self.ranker.run_tournament(
                all_candidates, rounds=3
            )

        return hypotheses

    async def _generate_initial(
        self, goal: str, n: int
    ) -> list[Hypothesis]:
        """Generate the initial population of hypotheses."""
        import json

        hypotheses = []
        for i in range(n):
            response = await self.generator.chat(
                system=(
                    "Generate a novel scientific hypothesis for the "
                    "given research goal. Be creative and specific. "
                    "Return JSON: {\"hypothesis\": \"...\", "
                    "\"reasoning\": \"...\", \"testable_prediction\": \"...\"}"
                ),
                user=f"Research goal: {goal}\n(Candidate {i + 1} of {n})",
                temperature=0.9 + (i * 0.02),  # diversity via temperature
            )
            data = json.loads(response.content)
            hypotheses.append(Hypothesis(
                id=f"gen0_h{i}",
                content=data["hypothesis"],
                reasoning=data["reasoning"],
            ))
        return hypotheses

    async def _reflect(self, hypothesis: Hypothesis) -> Hypothesis:
        """Improve a hypothesis through self-critique."""
        response = await self.reflector.chat(
            system=(
                "Critique and improve this hypothesis. Identify "
                "weaknesses, fill gaps, strengthen the reasoning. "
                "Return the improved version."
            ),
            user=(
                f"Hypothesis: {hypothesis.content}\n"
                f"Reasoning: {hypothesis.reasoning}"
            ),
        )
        return Hypothesis(
            id=hypothesis.id + "_reflected",
            content=response.content,
            reasoning=hypothesis.reasoning,
        )

    async def _evolve(
        self, goal: str, survivors: list[Hypothesis], n: int
    ) -> list[Hypothesis]:
        """Generate new hypotheses by recombining survivors."""
        import json

        hypotheses = []
        for i in range(n):
            # select two parents
            parent_a = random.choice(survivors)
            parent_b = random.choice(survivors)

            response = await self.generator.chat(
                system=(
                    "Combine ideas from two parent hypotheses to create "
                    "a novel child hypothesis. Take the strongest elements "
                    "of each and resolve contradictions creatively."
                ),
                user=(
                    f"Research goal: {goal}\n"
                    f"Parent A: {parent_a.content}\n"
                    f"Parent B: {parent_b.content}\n"
                    "Return JSON: {\"hypothesis\": \"...\", "
                    "\"reasoning\": \"...\"}"
                ),
                temperature=0.8,
            )
            data = json.loads(response.content)
            hypotheses.append(Hypothesis(
                id=f"gen_evolved_h{i}",
                content=data["hypothesis"],
                reasoning=data["reasoning"],
            ))
        return hypotheses

4. Architectural Comparison

The three systems represent three points on a design spectrum. Table 40.1 summarizes where each sits, helping you choose the right architecture for your own research agent system.

The key trade-off is between flexibility and reliability. Coscientist can search the web for any procedure but is most susceptible to LLM errors propagating into physical experiments. ChemCrow gives exact answers through domain tools but cannot operate outside its 18-tool repertoire. AI Co-Scientist explores the widest hypothesis space but at the highest cost and with no physical validation.

5. Safety and Human Oversight

The reliability concerns in Table 40.1 are not merely academic inconveniences; when a research agent controls physical hardware or informs clinical decisions, an unchecked error can cause real harm. Research agents that can plan experiments, execute code, and propose chemical syntheses carry risks that go beyond incorrect outputs. The safety considerations fall into three categories:

Dual-use risk. A system that can plan synthesis routes for beneficial drugs can also plan routes for harmful compounds. Coscientist addresses this with a safety check module that screens each step before execution. ChemCrow includes an explosive screening tool and controlled substance checker. Both rely on the same principle: the safety tools must be harder to bypass than to use. In practice, this means the safety check should be a hard-coded gate in the execution pipeline, not a removable plugin.

Cascading errors. A literature agent that misreads a paper feeds incorrect information to the coding agent. The coding agent then writes an experiment based on wrong assumptions, and the analysis agent dutifully executes it. The reviewer agent (Section 40.1) is the primary defense, but it can only catch errors it can detect. Plausible-looking but wrong results pose the greatest danger. Defense in depth requires multiple independent checks: the reviewer agent, human spot-checks at critical junctures, and automated consistency tests (does the claimed result match the raw data?).

Automation bias (the tendency for humans to over-trust outputs from automated systems, especially when those outputs appear confident and detailed). When a sophisticated AI system reports a finding with confidence, human researchers tend to accept it without sufficient scrutiny. This is especially dangerous when the AI system produces results that are technically impressive but scientifically flawed. The mitigation is structural: research agent outputs should always be labeled as provisional, requiring explicit human sign-off before being incorporated into publications or decision-making. The human gates from Chapter 17 apply here with even greater force.

Exercises

1. Conceptual: Compare Coscientist's replanning loop with AI Co-Scientist's tournament ranking. Both are mechanisms for improving quality through iteration. Under what conditions would you prefer replanning (adjusting a single plan based on observations) over tournament ranking (generating many candidates and selecting the best)? Consider cost, latency, and the nature of the research task.
2. Coding: Extend the TournamentRanker to detect and mitigate position bias (the tendency of LLM judges to prefer whichever hypothesis is presented first). Implement a double-blind comparison where each pair is judged twice (A vs B, then B vs A), and the winner is determined by the consistent verdict. If the two judgments disagree, mark the pair as a draw. Test with 10 hypothesis pairs and report the rate of position-bias disagreements.
3. Analysis: The AI Co-Scientist pipeline with population size 10, 3 generations, and 3 tournament rounds makes approximately $10 \times 3 + 10 \times 3 + \binom{10}{2} \times 3 \times 3 = 30 + 30 + 405 = 465$ LLM calls (generation + reflection + tournament judging). At \$0.01 per call (GPT-4o-mini) or \$0.10 per call (GPT-4o), compute the total cost. At what point does it become cheaper to hire a postdoc for a week? What does this imply about the optimal model choice for each agent role?