54.2 Debate and Peer Review Protocols

1. Adversarial Critique as Quality Filter

Consider a language model that proposes a drug will inhibit a target protein by binding its active site, backs the claim with three published crystal structures, and gets the underlying causal logic exactly backwards: the hypothesis reads so well that a human reviewer might nod along, yet a second large language model (LLM) agent, primed to attack the reasoning, catches the flaw before a six-month wet-lab campaign begins.

Adversarial critique pairs a dedicated reviewer agent against a proposer agent. The reviewer systematically attacks the logic, evidence, and methodology of the proposer's hypothesis. This separation matters because a single LLM that generates and self-checks its own output falls prey to confirmation bias: the same reasoning pathways that produced the error are unlikely to catch it on a second pass. The protocol assigns incompatible objectives to the two agents. The proposer maximizes the persuasiveness of the hypothesis while the critic maximizes the probability of finding a flaw. This adversarial tension surfaces errors that neither agent would find alone. Use adversarial critique instead of self-consistency checking (where one agent re-evaluates its own output) whenever a false positive costs more than an extra LLM call. Prefer it over voting when the review must produce actionable feedback that improves the hypothesis, not just a binary accept/reject signal.

Empirical Evidence for Debate

Du et al. (2023) demonstrated this empirically: multi-agent debate improved accuracy on mathematical reasoning tasks by 15 to 20 percentage points over single-agent generation. The improvement peaks on tasks where the initial answer is plausible but wrong, precisely the failure mode that matters most in scientific discovery. The accuracy gain comes not from the debate format itself but from forcing the proposer to defend its answer against specific objections. Defending activates different reasoning pathways than the initial generation.

The cost of adversarial critique is measured in additional LLM calls. A single review round roughly doubles the token cost per hypothesis. Two review rounds triple it. The question is whether the quality improvement (measured as the fraction of accepted hypotheses that are genuinely correct) exceeds the quality of hypotheses generated without review, adjusted for the cost ratio. We formalize this tradeoff as a return on investment (ROI):

where \(Q\) is the quality metric (e.g., fraction of hypotheses that replicate) and \(C\) is the total cost per accepted hypothesis. A positive ROI means review is worth the cost; a negative ROI means the resources would be better spent generating and testing more hypotheses without review. In short: the cheapest experiment you will ever run is the one a good reviewer kills before it starts.

Given that adversarial review is almost always cheaper than the experiments it protects, the practical question becomes which debate structure to use and when. The three protocols differ in their communication topology, as Figure 54.2a illustrates.

2. Three Debate Protocols

The three structured debate protocols shown in Figure 54.2a suit different discovery scenarios. They differ in communication topology, number of rounds, and how disagreements are resolved.

2.1 Proposer-Critic (Two-Agent Debate)

The simplest protocol pairs a proposer with a single critic in an iterative dialogue. The proposer presents a hypothesis; the critic raises objections; the proposer responds to each objection (either revising the hypothesis or defending the original claim); the critic evaluates the responses. This continues for a fixed number of rounds or until the critic is satisfied.

from dataclasses import dataclass, field
from typing import Literal
import json


@dataclass
class DebateMessage:
    """A single message in a structured debate."""
    round_num: int
    speaker: Literal["proposer", "critic"]
    content: str
    message_type: Literal[
        "proposal",     # Initial hypothesis presentation
        "objection",    # Critic raises a specific concern
        "defense",      # Proposer responds to an objection
        "revision",     # Proposer revises the hypothesis
        "concession",   # Proposer concedes a point
        "verdict",      # Critic's final assessment
    ]


@dataclass
class DebateOutcome:
    """Result of a structured debate."""
    hypothesis_id: str
    rounds_completed: int
    final_verdict: Literal["accepted", "revised", "rejected"]
    transcript: list[DebateMessage]
    revisions_made: list[str]
    unresolved_objections: list[str]
    proposer_concessions: list[str]


async def run_proposer_critic_debate(
    hypothesis: "Hypothesis",
    proposer_agent: "Agent",
    critic_agent: "Agent",
    max_rounds: int = 3,
    early_stop_on_accept: bool = True,
) -> DebateOutcome:
    """Execute a proposer-critic debate over a hypothesis.

    Protocol:
      Round 1: Proposer presents -> Critic objects
      Round 2: Proposer defends/revises -> Critic re-evaluates
      ...
      Final:   Critic issues verdict

    Args:
        hypothesis: The hypothesis to debate.
        proposer_agent: Agent configured with proposer role.
        critic_agent: Agent configured with critic role.
        max_rounds: Maximum debate rounds before forced verdict.
        early_stop_on_accept: Stop early if critic accepts.

    Returns:
        DebateOutcome with full transcript and final verdict.
    """
    transcript: list[DebateMessage] = []
    revisions: list[str] = []
    current_hypothesis = hypothesis

    for round_num in range(1, max_rounds + 1):
        # Step 1: Proposer presents or defends
        if round_num == 1:
            proposer_msg = await proposer_agent.run(
                f"Present this hypothesis for review. Explain the "
                f"mechanism, evidence, and why it is worth testing.\n\n"
                f"Hypothesis: {current_hypothesis.statement}\n"
                f"Mechanism: {current_hypothesis.mechanism}"
            )
            transcript.append(DebateMessage(
                round_num=round_num,
                speaker="proposer",
                content=proposer_msg.text,
                message_type="proposal",
            ))
        else:
            # Respond to previous objections
            last_objection = transcript[-1].content
            defense_msg = await proposer_agent.run(
                f"The reviewer raised these objections:\n\n"
                f"{last_objection}\n\n"
                f"Respond to each objection. You may defend your "
                f"original claim with additional evidence, revise "
                f"the hypothesis, or concede the point. Be specific."
            )

            # Classify the response type
            msg_type = "defense"
            if "revise" in defense_msg.text.lower()[:100]:
                msg_type = "revision"
                revisions.append(defense_msg.text)
            elif "concede" in defense_msg.text.lower()[:100]:
                msg_type = "concession"

            transcript.append(DebateMessage(
                round_num=round_num,
                speaker="proposer",
                content=defense_msg.text,
                message_type=msg_type,
            ))

        # Step 2: Critic evaluates
        debate_history = "\n\n".join(
            f"[{m.speaker.upper()} - Round {m.round_num}]: {m.content}"
            for m in transcript
        )

        is_final_round = (round_num == max_rounds)
        critic_instruction = (
            f"Review the following scientific debate:\n\n"
            f"{debate_history}\n\n"
        )
        if is_final_round:
            critic_instruction += (
                "This is the final round. Issue your verdict: "
                "'accepted' (hypothesis is sound and worth testing), "
                "'revised' (hypothesis improved but needs more work), "
                "or 'rejected' (fundamental flaws remain). "
                "List any unresolved objections."
            )
        else:
            critic_instruction += (
                "Raise specific, actionable objections. Focus on: "
                "(1) methodological flaws, (2) unsupported causal claims, "
                "(3) missing controls or confounders, (4) statistical issues. "
                "If all major concerns are addressed, you may accept early."
            )

        critic_msg = await critic_agent.run(critic_instruction)

        if is_final_round:
            transcript.append(DebateMessage(
                round_num=round_num,
                speaker="critic",
                content=critic_msg.text,
                message_type="verdict",
            ))
        else:
            transcript.append(DebateMessage(
                round_num=round_num,
                speaker="critic",
                content=critic_msg.text,
                message_type="objection",
            ))

        # Check for early acceptance
        if early_stop_on_accept and "accept" in critic_msg.text.lower()[:50]:
            if not is_final_round:
                transcript.append(DebateMessage(
                    round_num=round_num,
                    speaker="critic",
                    content="Early acceptance: all major concerns addressed.",
                    message_type="verdict",
                ))
            break

    # Parse the final verdict from the last critic message
    final_msg = [m for m in transcript if m.message_type == "verdict"][-1]
    if "rejected" in final_msg.content.lower()[:100]:
        verdict = "rejected"
    elif "accepted" in final_msg.content.lower()[:100]:
        verdict = "accepted"
    else:
        verdict = "revised"

    return DebateOutcome(
        hypothesis_id=hypothesis.statement[:50],
        rounds_completed=round_num,
        final_verdict=verdict,
        transcript=transcript,
        revisions_made=revisions,
        unresolved_objections=[
            m.content for m in transcript
            if m.message_type == "objection" and m.round_num == round_num
        ],
        proposer_concessions=[
            m.content for m in transcript
            if m.message_type == "concession"
        ],
    )

The proposer-critic protocol is simple and cheap (2 LLM calls per round), but it has a known weakness: with only one critic, the review quality depends entirely on that critic's perspective. A single critic may have blind spots that systematically pass flawed hypotheses. Round-robin review addresses this limitation.

2.2 Round-Robin Review

In round-robin review, the hypothesis is sent to \(k\) independent reviewers simultaneously. Each reviewer evaluates the hypothesis without seeing other reviews, then the judge aggregates the verdicts. This protocol maximizes independence (a key requirement of the Condorcet jury theorem, which holds that majority voting improves accuracy only when voters judge independently) at the cost of higher total token usage.

import asyncio
from dataclasses import dataclass


@dataclass
class RoundRobinResult:
    """Result of round-robin review by multiple independent reviewers."""
    hypothesis_id: str
    individual_reviews: list["ReviewVerdict"]
    agreement_kappa: float
    aggregated_decision: str
    aggregated_confidence: float
    score_summary: dict  # {dimension: {mean, std, min, max}}


async def run_round_robin_review(
    hypothesis: "Hypothesis",
    experiment_result: "ExperimentResult",
    reviewer_agents: list["Agent"],
    reviewer_configs: list["ReviewerConfig"],
) -> RoundRobinResult:
    """Send hypothesis and results to k independent reviewers.

    All reviewers evaluate in parallel, without seeing each
    other's reviews. The judge aggregates afterward.

    Args:
        hypothesis: The hypothesis under review.
        experiment_result: Experimental results to evaluate.
        reviewer_agents: List of k reviewer agent instances.
        reviewer_configs: Corresponding reviewer configurations.

    Returns:
        RoundRobinResult with individual and aggregated reviews.
    """
    review_prompt = (
        f"Review the following scientific hypothesis and "
        f"experimental results.\n\n"
        f"HYPOTHESIS:\n{hypothesis.statement}\n"
        f"Mechanism: {hypothesis.mechanism}\n"
        f"Predictions: {hypothesis.predictions}\n\n"
        f"EXPERIMENTAL RESULTS:\n"
        f"Status: {experiment_result.status}\n"
        f"Statistical summary: "
        f"{json.dumps(experiment_result.statistical_summary, indent=2)}\n"
        f"Success criteria met: "
        f"{json.dumps(experiment_result.success_criteria_met)}\n"
        f"Anomalies: {experiment_result.anomalies}\n\n"
        f"Provide your review as a structured assessment with "
        f"scores for methodology (1-10), statistical rigor (1-10), "
        f"novelty (1-10), and reproducibility (1-10). Give an "
        f"overall score and decision: accept, revise, or reject."
    )

    # Run all reviews in parallel for maximum independence
    review_tasks = []
    for agent, config in zip(reviewer_agents, reviewer_configs):
        # Customize prompt for reviewer's specialization
        specialized_prompt = (
            f"Your review focus is: {config.focus}. "
            f"Pay special attention to issues in this area.\n\n"
            f"{review_prompt}"
        )
        review_tasks.append(agent.run(specialized_prompt))

    raw_reviews = await asyncio.gather(*review_tasks)

    # Parse structured reviews from agent outputs
    verdicts = []
    for raw, config in zip(raw_reviews, reviewer_configs):
        verdict = parse_review_verdict(raw.text, hypothesis)
        verdicts.append(verdict)

    # Compute inter-reviewer agreement
    kappa = compute_agreement(verdicts)

    # Aggregate using confidence-weighted voting: each reviewer's
    # vote is scaled by its self-reported confidence, so a
    # high-confidence "reject" outweighs a low-confidence "accept."
    decision, confidence = confidence_weighted_vote(verdicts)

    # Compute score statistics across dimensions
    dimensions = [
        "methodology_score", "statistical_rigor_score",
        "novelty_score", "reproducibility_score", "overall_score",
    ]
    import numpy as np
    score_summary = {}
    for dim in dimensions:
        scores = [getattr(v, dim) for v in verdicts]
        score_summary[dim] = {
            "mean": float(np.mean(scores)),
            "std": float(np.std(scores)),
            "min": int(min(scores)),
            "max": int(max(scores)),
        }

    return RoundRobinResult(
        hypothesis_id=hypothesis.statement[:50],
        individual_reviews=verdicts,
        agreement_kappa=kappa,
        aggregated_decision=decision,
        aggregated_confidence=confidence,
        score_summary=score_summary,
    )


def parse_review_verdict(
    raw_text: str,
    hypothesis: "Hypothesis",
) -> "ReviewVerdict":
    """Parse a structured ReviewVerdict from raw agent output.

    In production, use a structured output parser (e.g., Pydantic
    with OpenAI function calling or Anthropic tool use). This
    simplified version extracts key fields with heuristics.
    """
    # Production implementation would use structured output parsing
    # (instructor, LangChain output parsers, or native tool use).
    # Simplified for illustration.
    text_lower = raw_text.lower()

    if "reject" in text_lower[:200]:
        decision = "reject"
    elif "accept" in text_lower[:200]:
        decision = "accept"
    else:
        decision = "revise"

    return ReviewVerdict(
        hypothesis_id=hypothesis.statement[:50],
        decision=decision,
        confidence=0.7,  # Would be extracted from structured output
        strengths=["Parsed from structured output in production"],
        weaknesses=["Parsed from structured output in production"],
        questions=[],
        methodology_score=6,
        statistical_rigor_score=6,
        novelty_score=6,
        reproducibility_score=6,
        overall_score=6,
        suggested_experiments=[],
    )

The confidence_weighted_vote function aggregates the independent reviews by weighting each reviewer's decision (accept, revise, or reject) by its self-reported confidence score. A reviewer that reports 90% confidence in "reject" contributes more to the final decision than one that reports 55% confidence in "accept." This approach avoids the failure mode of simple majority voting, where a lukewarm majority can override a single reviewer who is highly confident it spotted a fatal flaw.

2.3 Tournament Selection

When the team generates multiple competing hypotheses, tournament selection pits them against each other in head-to-head comparisons. A reviewer agent sees two hypotheses side by side and selects the stronger one, providing a comparative judgment rather than an absolute score. This protocol is inspired by the Elo rating system (a method for calculating relative skill levels in zero-sum games, originally developed for chess rankings) and the pairwise comparison methods used in reinforcement learning from human feedback (RLHF).

import random
from dataclasses import dataclass


@dataclass
class TournamentMatch:
    """Result of a head-to-head hypothesis comparison."""
    hypothesis_a_id: str
    hypothesis_b_id: str
    winner_id: str
    reasoning: str
    margin: Literal["clear", "slight", "toss-up"]


@dataclass
class TournamentResult:
    """Final rankings after a tournament."""
    rankings: list[tuple[str, float]]  # (hypothesis_id, elo_score)
    matches: list[TournamentMatch]
    total_comparisons: int


async def run_tournament(
    hypotheses: list["Hypothesis"],
    reviewer_agent: "Agent",
    rounds: int = 2,
    initial_elo: float = 1500.0,
    k_factor: float = 32.0,
) -> TournamentResult:
    """Run a round-robin tournament to rank hypotheses.

    Each pair of hypotheses is compared by the reviewer agent.
    Elo ratings are updated after each match. Multiple rounds
    smooth out noise from individual comparisons.

    Args:
        hypotheses: List of candidate hypotheses to rank.
        reviewer_agent: Agent that makes pairwise comparisons.
        rounds: Number of complete round-robin passes.
        initial_elo: Starting Elo rating for each hypothesis.
        k_factor: Elo update magnitude (higher = more volatile).

    Returns:
        TournamentResult with Elo rankings and match history.
    """
    # Initialize Elo ratings
    elo = {h.statement[:50]: initial_elo for h in hypotheses}
    matches: list[TournamentMatch] = []

    for round_num in range(rounds):
        # Generate all pairs, shuffle for fairness
        pairs = [
            (hypotheses[i], hypotheses[j])
            for i in range(len(hypotheses))
            for j in range(i + 1, len(hypotheses))
        ]
        random.shuffle(pairs)

        for h_a, h_b in pairs:
            # Present both hypotheses to the reviewer
            comparison_prompt = (
                f"Compare these two scientific hypotheses. Which is "
                f"more promising for advancing scientific understanding? "
                f"Consider novelty, testability, and plausibility.\n\n"
                f"HYPOTHESIS A:\n"
                f"Statement: {h_a.statement}\n"
                f"Mechanism: {h_a.mechanism}\n"
                f"Predictions: {h_a.predictions}\n\n"
                f"HYPOTHESIS B:\n"
                f"Statement: {h_b.statement}\n"
                f"Mechanism: {h_b.mechanism}\n"
                f"Predictions: {h_b.predictions}\n\n"
                f"Choose the winner (A or B) and explain your reasoning. "
                f"Rate the margin: 'clear' (obvious winner), 'slight' "
                f"(close call), or 'toss-up' (nearly equal)."
            )

            result = await reviewer_agent.run(comparison_prompt)
            text = result.text.lower()

            # Parse winner
            id_a = h_a.statement[:50]
            id_b = h_b.statement[:50]

            if "hypothesis a" in text[:100] or "winner: a" in text[:100]:
                winner_id = id_a
            else:
                winner_id = id_b

            # Determine margin
            if "clear" in text:
                margin = "clear"
            elif "toss-up" in text:
                margin = "toss-up"
            else:
                margin = "slight"

            matches.append(TournamentMatch(
                hypothesis_a_id=id_a,
                hypothesis_b_id=id_b,
                winner_id=winner_id,
                reasoning=result.text[:500],
                margin=margin,
            ))

            # Update Elo ratings
            loser_id = id_b if winner_id == id_a else id_a
            expected_winner = 1 / (
                1 + 10 ** ((elo[loser_id] - elo[winner_id]) / 400)
            )
            elo[winner_id] += k_factor * (1 - expected_winner)
            elo[loser_id] -= k_factor * (1 - expected_winner)

    # Sort by final Elo
    rankings = sorted(elo.items(), key=lambda x: x[1], reverse=True)

    return TournamentResult(
        rankings=rankings,
        matches=matches,
        total_comparisons=len(matches),
    )

Tournament selection has two advantages over independent scoring. First, comparative judgments tend to be easier for LLMs than absolute scores: asking "which hypothesis is better?" typically produces more reliable answers than asking "rate this hypothesis from 1 to 10," because pairwise comparison requires only ordinal ranking rather than maintaining a consistent numeric scale. Second, the Elo system automatically handles transitivity: if hypothesis A beats B and B beats C, then A should rank above C even if A and C are never directly compared. The Google DeepMind AI Co-Scientist (Yamada et al., 2025) uses this tournament approach to rank competing hypotheses before committing experimental resources.

All three protocols assume that the reviewer agents produce reliable judgments, but that assumption must be tested; an inaccurate reviewer can waste resources just as surely as no reviewer at all.

3. Calibrating Reviewers

A reviewer agent is only useful if its judgments correlate with ground truth. An uncalibrated reviewer that accepts everything is worthless; one that rejects everything is worse than worthless (it blocks good work). Calibration is the process of measuring and adjusting a reviewer's accuracy on known examples.

3.1 The Calibration Protocol

We calibrate reviewers using a held-out set of hypotheses with known outcomes: some are correct (supported by subsequent experiments), some are flawed (contain known errors), and some are borderline. The reviewer evaluates each hypothesis without knowing the ground truth, and we measure its accuracy across four categories.

from dataclasses import dataclass
import numpy as np


@dataclass
class CalibrationResult:
    """Calibration metrics for a single reviewer agent."""
    reviewer_id: str
    true_positive_rate: float   # Correctly accepted good hypotheses
    true_negative_rate: float   # Correctly rejected bad hypotheses
    false_positive_rate: float  # Incorrectly accepted bad hypotheses
    false_negative_rate: float  # Incorrectly rejected good hypotheses
    calibration_score: float    # Overall accuracy (0-1)
    bias: str                   # "permissive", "balanced", or "strict"


async def calibrate_reviewer(
    reviewer_agent: "Agent",
    calibration_set: list[dict],  # [{hypothesis, ground_truth, evidence}]
) -> CalibrationResult:
    """Measure reviewer accuracy on a labeled calibration set.

    The calibration set should contain:
      - 40% correct hypotheses (ground_truth = "accept")
      - 40% flawed hypotheses (ground_truth = "reject")
      - 20% borderline cases (ground_truth = "revise")

    Args:
        reviewer_agent: The reviewer to calibrate.
        calibration_set: Labeled examples with known outcomes.

    Returns:
        CalibrationResult with accuracy metrics.
    """
    tp = fp = tn = fn = 0

    for item in calibration_set:
        review = await reviewer_agent.run(
            f"Review this hypothesis:\n{item['hypothesis']}\n\n"
            f"Evidence:\n{item['evidence']}\n\n"
            f"Decision: accept, revise, or reject."
        )

        predicted = "accept"  # default
        text = review.text.lower()
        if "reject" in text[:100]:
            predicted = "reject"
        elif "revise" in text[:100]:
            predicted = "revise"

        actual = item["ground_truth"]

        # Binary classification: accept vs. not-accept
        if actual == "accept" and predicted == "accept":
            tp += 1
        elif actual == "accept" and predicted != "accept":
            fn += 1
        elif actual != "accept" and predicted != "accept":
            tn += 1
        else:  # actual != "accept" and predicted == "accept"
            fp += 1

    total = len(calibration_set)
    tpr = tp / max(tp + fn, 1)
    tnr = tn / max(tn + fp, 1)
    fpr = fp / max(fp + tn, 1)
    fnr = fn / max(fn + tp, 1)
    accuracy = (tp + tn) / total

    # Determine bias direction
    if fpr > 0.3:
        bias = "permissive"
    elif fnr > 0.3:
        bias = "strict"
    else:
        bias = "balanced"

    return CalibrationResult(
        reviewer_id=reviewer_agent.name,
        true_positive_rate=tpr,
        true_negative_rate=tnr,
        false_positive_rate=fpr,
        false_negative_rate=fnr,
        calibration_score=accuracy,
        bias=bias,
    )

3.2 Adjusting Reviewer Behavior

Calibration results directly inform prompt adjustments. A permissive reviewer (high false positive rate) needs a more adversarial prompt: "Look harder for flaws. Assume the hypothesis is wrong until proven otherwise." A strict reviewer (high false negative rate) needs a more charitable prompt: "Focus on the core contribution. Do not reject work for minor issues that can be fixed in revision."

def adjust_reviewer_prompt(
    base_prompt: str,
    calibration: CalibrationResult,
    target_accuracy: float = 0.75,
) -> str:
    """Adjust a reviewer's system prompt based on calibration results.

    Adds corrective instructions to shift the reviewer toward
    the target accuracy.
    """
    adjustments = []

    if calibration.bias == "permissive":
        adjustments.append(
            "IMPORTANT: In past evaluations, you have been too lenient. "
            "You accepted hypotheses that turned out to be flawed. "
            "Be more skeptical. Ask: 'What evidence would disprove "
            "this hypothesis?' If the authors have not addressed this "
            "question, the work is not ready for acceptance."
        )
    elif calibration.bias == "strict":
        adjustments.append(
            "IMPORTANT: In past evaluations, you have been too strict. "
            "You rejected hypotheses that turned out to be correct. "
            "Focus on fundamental scientific merit, not stylistic "
            "preferences. A hypothesis with a sound mechanism and "
            "testable predictions deserves 'revise', not 'reject', "
            "even if the current evidence is incomplete."
        )

    if calibration.false_positive_rate > 0.2:
        adjustments.append(
            "Pay extra attention to statistical methodology. Check "
            "for multiple comparisons, p-hacking indicators, and "
            "effect sizes that are implausibly large."
        )

    if calibration.false_negative_rate > 0.2:
        adjustments.append(
            "Before rejecting, ask yourself: 'Is this hypothesis "
            "fundamentally wrong, or does it just need more evidence?' "
            "Recommend revision for work that is on the right track."
        )

    if adjustments:
        adjustment_block = "\n\n".join(adjustments)
        return f"{base_prompt}\n\n{adjustment_block}"

    return base_prompt

4. Measuring Review Quality

A review process is only as good as its outcomes. The right metrics capture whether review improves discovery quality, not just whether reviewers agree with each other. Four metrics together provide a complete picture.

4.1 Precision and Recall of the Review Filter

Treating the review process as a binary classifier (accept vs. reject), we compute precision (fraction of accepted hypotheses that are genuinely correct) and recall (fraction of genuinely correct hypotheses that are accepted). The ideal review process has high precision (does not pass bad work) and high recall (does not block good work).

In scientific discovery, precision is usually more important than recall. A false accept (accepting a flawed hypothesis) wastes expensive experimental resources. A false reject (rejecting a correct hypothesis) loses an opportunity but does not waste resources, and the hypothesis can be regenerated or rediscovered. This asymmetry suggests tuning reviewers for high precision even at the cost of lower recall, especially when experiments are expensive.

4.2 Inter-Reviewer Agreement

Fleiss' kappa, introduced in Section 54.1, measures agreement but not accuracy. High agreement (kappa > 0.6) may reflect correct reviewers converging on clear answers, or shared bias producing identical mistakes. Low agreement (kappa < 0.2) surfaces genuine disagreement: valuable when it brings different perspectives, problematic when it signals ambiguous review criteria.

In practice, a useful operating point is moderate agreement (kappa between 0.3 and 0.6), indicating that reviewers bring different perspectives but converge on clear cases. This aligns with empirical findings from human peer review, where inter-reviewer agreement at top ML conferences typically falls in the range of kappa 0.2 to 0.4 (Shah et al., 2018).

4.3 Review Impact Score

The review impact score measures how much the review process changes the final output compared to the unreviewed baseline. If the review process has no impact (every hypothesis passes unchanged), it is wasting tokens. If it changes everything (every hypothesis is heavily revised or rejected), it may be too strict or the proposer may be generating very low-quality hypotheses.

@dataclass
class ReviewImpactMetrics:
    """Measures the impact of the review process on discovery quality."""
    acceptance_rate: float         # Fraction of hypotheses accepted
    revision_rate: float           # Fraction sent back for revision
    rejection_rate: float          # Fraction rejected outright
    avg_revision_rounds: float     # Mean number of revision cycles
    quality_lift: float            # Quality(reviewed) - Quality(unreviewed)
    cost_per_accepted: float       # Total cost / number accepted
    cost_per_correct: float        # Total cost / number correctly accepted


def compute_review_impact(
    reviewed_outcomes: list[dict],
    unreviewed_baseline: list[dict],
    cost_per_review_round: float = 0.50,
) -> ReviewImpactMetrics:
    """Compare discovery quality with and without review.

    Each outcome dict has:
      - "accepted": bool
      - "correct": bool (ground truth, from later validation)
      - "revision_rounds": int
      - "generation_cost": float
    """
    n_reviewed = len(reviewed_outcomes)
    n_accepted = sum(1 for o in reviewed_outcomes if o["accepted"])
    n_revised = sum(
        1 for o in reviewed_outcomes
        if o.get("revision_rounds", 0) > 0
    )
    n_rejected = sum(1 for o in reviewed_outcomes if not o["accepted"])

    # Quality metrics
    reviewed_correct = sum(
        1 for o in reviewed_outcomes
        if o["accepted"] and o["correct"]
    )
    baseline_correct = sum(
        1 for o in unreviewed_baseline if o["correct"]
    )

    reviewed_quality = reviewed_correct / max(n_accepted, 1)
    baseline_quality = baseline_correct / max(len(unreviewed_baseline), 1)

    # Cost metrics
    total_review_cost = sum(
        o.get("revision_rounds", 1) * cost_per_review_round
        for o in reviewed_outcomes
    )
    total_generation_cost = sum(
        o["generation_cost"] for o in reviewed_outcomes
    )
    total_cost = total_review_cost + total_generation_cost

    return ReviewImpactMetrics(
        acceptance_rate=n_accepted / max(n_reviewed, 1),
        revision_rate=n_revised / max(n_reviewed, 1),
        rejection_rate=n_rejected / max(n_reviewed, 1),
        avg_revision_rounds=sum(
            o.get("revision_rounds", 0) for o in reviewed_outcomes
        ) / max(n_reviewed, 1),
        quality_lift=reviewed_quality - baseline_quality,
        cost_per_accepted=total_cost / max(n_accepted, 1),
        cost_per_correct=total_cost / max(reviewed_correct, 1),
    )

Positive impact scores confirm that review is working, but they do not reveal the conditions under which debate itself becomes the bottleneck.

5. When Debate Degrades Discovery

Adversarial debate is not universally beneficial. Three failure modes cause debate to reduce discovery quality:

Conformity pressure. When the proposer observes that the critic consistently rejects novel hypotheses, it learns to propose conservative, incremental ideas that are easier to defend. The debate process optimizes for "surviving review" rather than "maximizing scientific value." Liang et al. (2024) showed that naive debate protocols reduce the diversity of generated hypotheses by 30 to 40% compared to single-agent generation. The fix is to explicitly reward novelty in the proposer's utility function (the \(\alpha\) parameter from Section 54.1) and to include a novelty-focused reviewer alongside the methodology reviewer.

Eloquence over substance. LLMs are better at generating persuasive text than at evaluating scientific arguments. A proposer can "win" a debate by producing eloquent defenses of a flawed hypothesis, especially if the critic lacks domain expertise. The fix is to ground the debate in concrete predictions: instead of asking "is this hypothesis plausible?" ask "what specific experimental outcome would disprove this hypothesis, and has the proposer accounted for it?" Predictions are harder to fake with rhetoric.

Cost without benefit. For tasks where the proposer's accuracy is already high (for example, well-studied domains with abundant training data), review adds cost without meaningful quality improvement. The calibration experiment from Section 54.3 identifies this regime: if the quality lift is near zero, skip the review and invest the saved tokens in generating more hypotheses.

Exercises

1. Conceptual: A three-round proposer-critic debate uses 6 LLM calls per hypothesis. A round-robin review with 3 reviewers uses 3 calls (no iteration). If the proposer-critic debate achieves 85% precision and the round-robin achieves 80% precision, but the round-robin has higher recall (90% vs. 75%), which protocol is better for (a) a domain where experiments cost \$10,000 each, and (b) a domain where experiments cost \$0.10 each? Justify your answer with cost-per-correct-hypothesis calculations.
2. Coding: Implement a DebateMonitor class that tracks debate quality in real time. The monitor should compute: (a) the semantic similarity between successive proposer responses (to detect when the proposer is repeating itself rather than improving), (b) the fraction of critic objections that are substantive vs. stylistic (using a simple keyword classifier), and (c) the cumulative token cost. The monitor should recommend early stopping when the proposer's responses converge (similarity > 0.95) or when the cost exceeds a budget threshold.
3. Analysis: You calibrate three reviewer agents on a test set of 100 hypotheses (50 correct, 50 flawed). Reviewer A achieves 80% accuracy but is permissive (false positive rate = 0.30). Reviewer B achieves 75% accuracy and is strict (false negative rate = 0.35). Reviewer C achieves 70% accuracy and is balanced. If you can only use two reviewers for cost reasons, which pair maximizes the majority-vote accuracy? Compute the expected accuracy for all three pairs, assuming independence.