58.3 Open Problems and What the Field Needs

1. The Benchmark Gap

Without reliable benchmarks, teams have no way to tell whether a discovery system is genuinely advancing science or merely generating plausible outputs that fool human reviewers. The consequence is wasted funding, duplicated compute, and a growing credibility gap that could stall the entire field before it matures.

Machine learning progressed rapidly in part because of shared benchmarks: ImageNet (a large-scale image classification dataset with over 14 million labeled images) for vision, General Language Understanding Evaluation (GLUE) for language understanding (largely saturated by 2020; as of 2024, broader benchmarks such as MMLU and BIG-bench have become the primary yardsticks for large language models), SWE-bench (a benchmark of real GitHub issues that tests whether an AI system can write correct code patches) for software engineering. Discovery AI lacks equivalents. The benchmarks that exist (MLAgentBench, ScienceAgentBench) evaluate narrow slices of the discovery process, typically the experiment execution phase. No benchmark captures the full discovery loop from problem identification through hypothesis generation, experimental design, execution, analysis, and interpretation.

A discovery benchmark is a curated collection of research tasks with known outcomes, structured so that an AI system's ability to perform genuine scientific reasoning can be measured quantitatively and compared across systems. Discovery benchmarks matter because without them, the field has no objective way to distinguish a system that makes real scientific progress from one that merely generates plausible-sounding outputs. They work by presenting a system with a research context (datasets, prior literature, a target question), collecting the system's outputs at each stage of the discovery loop, and scoring those outputs against ground truth established from historical discoveries or expert judgment. Use discovery benchmarks rather than generic ML evaluations whenever the goal is to assess end-to-end scientific reasoning; use component-level evaluations (accuracy, F1, perplexity) only when diagnosing failures within a single stage.

Building a comprehensive discovery benchmark is hard for three reasons:

1. Ground truth is retrospective. We can only evaluate a discovery system against discoveries that have already been made. But evaluating on known discoveries tests rediscovery, not discovery. A system that "discovers" penicillin in 2026 has demonstrated pattern matching, not innovation. The evaluation framework from Chapter 56 addressed this partially through held-out temporal splits, but the fundamental problem remains.
2. Discovery is domain-specific. A benchmark that works for materials science (propose crystal structures, verify stability) has no applicability to social science (propose hypotheses about institutional behavior, test with quasi-experiments). The discovery process is universal in structure but domain-specific in content, making a single universal benchmark unlikely.
3. Success criteria are contested. Is a discovery "successful" when it is novel? When it is correct? When it is useful? When it is published? When it is cited? These criteria often conflict: highly cited papers tend to be incremental, while the most novel work frequently takes years to accumulate citations.

Despite these challenges, the field can make progress by building domain-specific discovery benchmark suites that evaluate the full discovery loop within a constrained domain. The following code implements a framework for constructing and evaluating against such benchmarks.

"""
Discovery Benchmark Framework: infrastructure for constructing,
running, and scoring domain-specific discovery benchmarks that
evaluate the full hypothesis-experiment-interpretation loop.
"""

import json
from dataclasses import dataclass, field
from datetime import datetime
from pathlib import Path
from typing import Any, Optional, Protocol
from enum import Enum


class DiscoveryStage(Enum):
    """Stages of the discovery process that a benchmark can evaluate."""
    PROBLEM_IDENTIFICATION = "problem_identification"
    HYPOTHESIS_GENERATION = "hypothesis_generation"
    EXPERIMENT_DESIGN = "experiment_design"
    EXPERIMENT_EXECUTION = "experiment_execution"
    DATA_ANALYSIS = "data_analysis"
    INTERPRETATION = "interpretation"
    FULL_LOOP = "full_loop"


@dataclass
class BenchmarkTask:
    """
    A single task in a discovery benchmark.

    Each task specifies an input context (what the system is given),
    an expected output type, and evaluation criteria. Tasks can
    target individual discovery stages or the full loop.
    """
    task_id: str
    domain: str
    stage: DiscoveryStage
    description: str

    # What the system receives
    input_context: dict[str, Any] = field(default_factory=dict)

    # Ground truth (for tasks with known answers)
    ground_truth: Optional[dict[str, Any]] = None

    # Evaluation criteria
    evaluation_criteria: list[str] = field(default_factory=list)

    # Metadata
    difficulty: str = "medium"  # "easy", "medium", "hard", "frontier"
    source_paper: Optional[str] = None
    discovery_year: Optional[int] = None


@dataclass
class BenchmarkResult:
    """Result of evaluating a system on a single benchmark task."""
    task_id: str
    system_output: dict[str, Any]
    scores: dict[str, float]
    evaluation_notes: str = ""
    wall_time_seconds: float = 0.0
    compute_cost_usd: float = 0.0


class DiscoveryEvaluator(Protocol):
    """Protocol for domain-specific evaluators."""

    def evaluate(
        self,
        task: BenchmarkTask,
        system_output: dict[str, Any],
    ) -> dict[str, float]:
        """Return a dict of metric_name -> score."""
        ...


class DiscoveryBenchmark:
    """
    A domain-specific discovery benchmark suite.

    Organizes tasks by discovery stage, runs discovery systems
    against them, and produces standardized evaluation reports.

    Example: a materials science benchmark might include:
    - Problem identification: given a set of material property
      databases, identify promising optimization targets
    - Hypothesis generation: given a target property, propose
      candidate compositions
    - Experiment design: given candidates, design a Bayesian
      optimization campaign
    - Full loop: from a vague goal ("better thermoelectrics")
      to a ranked list of verified candidates
    """

    def __init__(
        self,
        name: str,
        domain: str,
        version: str = "1.0",
    ):
        self.name = name
        self.domain = domain
        self.version = version
        self.tasks: list[BenchmarkTask] = []
        self.evaluators: dict[DiscoveryStage, DiscoveryEvaluator] = {}

    def add_task(self, task: BenchmarkTask) -> None:
        """Add a task to the benchmark suite."""
        if task.domain != self.domain:
            raise ValueError(
                f"Task domain '{task.domain}' does not match "
                f"benchmark domain '{self.domain}'"
            )
        self.tasks.append(task)

    def register_evaluator(
        self,
        stage: DiscoveryStage,
        evaluator: DiscoveryEvaluator,
    ) -> None:
        """Register a domain-specific evaluator for a discovery stage."""
        self.evaluators[stage] = evaluator

    def run(
        self,
        discovery_system,  # callable: task -> output
        stages: Optional[list[DiscoveryStage]] = None,
    ) -> list[BenchmarkResult]:
        """
        Run a discovery system against all tasks (or a subset
        filtered by stage) and return scored results.
        """
        target_tasks = self.tasks
        if stages:
            target_tasks = [
                t for t in self.tasks if t.stage in stages
            ]

        results = []
        for task in target_tasks:
            start_time = datetime.now()

            # Run the discovery system on this task
            system_output = discovery_system(task)

            elapsed = (datetime.now() - start_time).total_seconds()

            # Evaluate using the stage-appropriate evaluator
            evaluator = self.evaluators.get(task.stage)
            if evaluator:
                scores = evaluator.evaluate(task, system_output)
            else:
                scores = {"completeness": 1.0 if system_output else 0.0}

            result = BenchmarkResult(
                task_id=task.task_id,
                system_output=system_output,
                scores=scores,
                wall_time_seconds=elapsed,
            )
            results.append(result)

        return results

    def compute_aggregate_scores(
        self, results: list[BenchmarkResult]
    ) -> dict[str, Any]:
        """
        Compute aggregate scores across tasks, grouped by
        discovery stage and difficulty level.
        """
        import numpy as np

        # Group results by stage
        task_map = {t.task_id: t for t in self.tasks}
        by_stage: dict[str, list[dict[str, float]]] = {}
        by_difficulty: dict[str, list[dict[str, float]]] = {}

        for result in results:
            task = task_map.get(result.task_id)
            if not task:
                continue

            stage = task.stage.value
            difficulty = task.difficulty

            by_stage.setdefault(stage, []).append(result.scores)
            by_difficulty.setdefault(difficulty, []).append(result.scores)

        # Aggregate per-stage
        stage_aggregates = {}
        for stage, score_dicts in by_stage.items():
            all_metrics = set()
            for sd in score_dicts:
                all_metrics.update(sd.keys())

            stage_agg = {}
            for metric in all_metrics:
                values = [sd.get(metric, 0.0) for sd in score_dicts]
                stage_agg[metric] = {
                    "mean": float(np.mean(values)),
                    "std": float(np.std(values)),
                    "n": len(values),
                }
            stage_aggregates[stage] = stage_agg

        # Overall composite
        all_scores = []
        for result in results:
            if result.scores:
                all_scores.append(np.mean(list(result.scores.values())))

        return {
            "benchmark": self.name,
            "domain": self.domain,
            "version": self.version,
            "n_tasks": len(results),
            "overall_mean": float(np.mean(all_scores)) if all_scores else 0.0,
            "by_stage": stage_aggregates,
            "total_wall_time": sum(r.wall_time_seconds for r in results),
            "total_compute_cost": sum(r.compute_cost_usd for r in results),
        }

    def save(self, path: str) -> None:
        """Export benchmark definition for sharing."""
        data = {
            "name": self.name,
            "domain": self.domain,
            "version": self.version,
            "n_tasks": len(self.tasks),
            "tasks": [
                {
                    "task_id": t.task_id,
                    "stage": t.stage.value,
                    "description": t.description,
                    "difficulty": t.difficulty,
                    "source_paper": t.source_paper,
                    "evaluation_criteria": t.evaluation_criteria,
                }
                for t in self.tasks
            ],
        }
        Path(path).parent.mkdir(parents=True, exist_ok=True)
        with open(path, "w") as f:
            json.dump(data, f, indent=2)


def build_materials_discovery_benchmark() -> DiscoveryBenchmark:
    """
    Example: construct a materials science discovery benchmark
    with tasks spanning the full discovery loop.

    This is illustrative; a production benchmark would include
    hundreds of tasks with validated ground truth from the
    materials science literature.
    """
    bench = DiscoveryBenchmark(
        name="MatDiscovery-v1",
        domain="materials_science",
        version="1.0",
    )

    # Stage 1: Problem identification
    bench.add_task(BenchmarkTask(
        task_id="mat-prob-001",
        domain="materials_science",
        stage=DiscoveryStage.PROBLEM_IDENTIFICATION,
        description=(
            "Given the Materials Project database summary statistics "
            "and recent review papers on thermoelectric materials, "
            "identify the three most promising optimization targets "
            "for room-temperature thermoelectric performance."
        ),
        input_context={
            "database": "materials_project_summary_2024.json",
            "reviews": ["doi:10.1038/s41578-024-00123-4"],
        },
        evaluation_criteria=[
            "relevance_to_field",
            "specificity_of_target",
            "novelty_vs_known_targets",
        ],
        difficulty="hard",
    ))

    # Stage 2: Hypothesis generation
    bench.add_task(BenchmarkTask(
        task_id="mat-hyp-001",
        domain="materials_science",
        stage=DiscoveryStage.HYPOTHESIS_GENERATION,
        description=(
            "Propose five candidate compositions for a new "
            "half-Heusler thermoelectric material with ZT > 1.5 "
            "at 300K. Justify each proposal with reasoning about "
            "electronic structure and phonon scattering."
        ),
        input_context={
            "target": "half-Heusler, ZT > 1.5, T = 300K",
            "known_best": "ZrNiSn (ZT ~ 1.0 at 700K)",
        },
        ground_truth={
            "verified_candidates": [
                "TiNiSn-based alloys with Hf substitution",
            ],
        },
        evaluation_criteria=[
            "thermodynamic_stability",
            "electronic_structure_reasoning",
            "novelty",
        ],
        difficulty="hard",
        discovery_year=2023,
    ))

    # Stage 3: Experiment design
    bench.add_task(BenchmarkTask(
        task_id="mat-exp-001",
        domain="materials_science",
        stage=DiscoveryStage.EXPERIMENT_DESIGN,
        description=(
            "Design a Bayesian optimization campaign to explore "
            "the composition space Ti(1-x)Hf(x)NiSn(1-y)Sb(y) "
            "for thermoelectric performance. Specify the search "
            "space, surrogate model, acquisition function, initial "
            "design, and stopping criteria."
        ),
        input_context={
            "composition_space": "Ti(1-x)Hf(x)NiSn(1-y)Sb(y)",
            "x_range": [0.0, 1.0],
            "y_range": [0.0, 0.2],
            "budget": "50 DFT calculations",
        },
        evaluation_criteria=[
            "search_space_coverage",
            "surrogate_appropriateness",
            "sample_efficiency",
        ],
        difficulty="medium",
    ))

    # Stage 4: Full loop
    bench.add_task(BenchmarkTask(
        task_id="mat-full-001",
        domain="materials_science",
        stage=DiscoveryStage.FULL_LOOP,
        description=(
            "Starting from the goal 'find a new lead-free "
            "piezoelectric material with d33 > 500 pC/N', "
            "execute the complete discovery loop: identify the "
            "problem space, generate hypotheses, design and "
            "execute a computational screening campaign, analyze "
            "results, and report top candidates with confidence "
            "estimates."
        ),
        input_context={
            "goal": "lead-free piezoelectric, d33 > 500 pC/N",
            "compute_budget": "200 DFT calculations",
            "known_baseline": "BaTiO3 (d33 ~ 190 pC/N)",
        },
        evaluation_criteria=[
            "problem_decomposition",
            "hypothesis_quality",
            "experimental_efficiency",
            "result_validity",
            "uncertainty_calibration",
        ],
        difficulty="frontier",
    ))

    return bench

2. The Theory Gap

Even if the community builds comprehensive benchmarks that evaluate the full discovery loop, those benchmarks can typically only tell us whether a system works, not why it works or when it should be expected to fail.

Discovery AI lacks a theoretical foundation comparable to what statistical learning theory provides for supervised learning. In supervised learning, we have probably approximately correct (PAC) bounds, Vapnik-Chervonenkis (VC) dimension (a measure of the capacity of a hypothesis class, roughly how many data points it can shatter or perfectly classify), Rademacher complexity (a measure of how well a hypothesis class can fit random noise, used to bound generalization error), and other tools that tell us when and why learning is possible. For discovery, we have no analogous theory. We cannot answer fundamental questions:

• Sample complexity of discovery. How many experiments does a discovery agent need to identify a novel scientific law with probability at least \(1 - \delta\)? In supervised learning, the answer depends on the hypothesis class complexity and the desired accuracy. For discovery, we lack even the vocabulary to state the question formally, because the "hypothesis class" is the space of all possible scientific laws, which is not well-characterized.
• Learnability of scientific domains. Are some scientific domains inherently easier for AI to make discoveries in than others? Intuitively, domains with strong mathematical structure (physics, chemistry) seem more amenable than domains with weak structure (social science, ecology). But we have no formal way to measure "structure" or to predict discoverability from domain properties.
• Composition of discovery capabilities. If a system can discover laws in domain A and laws in domain B, can it discover laws that bridge A and B? Supervised learning has no general composition theory either, but the question is especially acute for discovery because many of the most celebrated discoveries in the history of science have connected previously unrelated fields (consider how statistical mechanics bridged thermodynamics and atomic theory).
• Optimality of discovery strategies. Is there an optimal strategy for allocating effort between exploration (trying new approaches) and exploitation (refining known approaches) in a discovery context? The multi-armed bandit framework from Chapter 46 provides partial answers for fixed search spaces, but discovery involves an expanding search space where new arms can be created.

We can make partial progress by formalizing discovery as a particular kind of search problem. Define a discovery space \(\mathcal{H}\) as the set of all hypotheses expressible in a given formal language (e.g., algebraic equations up to a given complexity, or logical formulas over a given vocabulary). A discovery oracle \(\mathcal{O}\) is a function that, given a hypothesis \(h \in \mathcal{H}\) and a dataset \(D\), returns a quality score \(q(h, D) \in [0, 1]\). The discovery problem is to find:

where \(K(h)\) is the description complexity of \(h\) (a measure of the length of the shortest program or formula that specifies \(h\), as introduced in Section 58.1). This formulation connects discovery to program synthesis and to the Minimum Description Length (MDL) principle. The theoretical questions above can then be restated: what is the sample complexity of finding \(h^*\) as a function of \(|\mathcal{H}|\), \(K(h^*)\), and properties of \(q\)?

For restricted hypothesis classes, we can derive bounds. If \(\mathcal{H}\) is the set of polynomial equations of degree at most \(d\) over \(p\) variables, then \(|\mathcal{H}|\) grows as \(\binom{p+d}{d}\), and standard PAC-learning bounds apply to the regression sub-problem. But real discovery involves choosing the right hypothesis class, not just searching within a fixed one, and that meta-learning problem (learning which hypothesis class to search within, rather than just searching a fixed one) remains theoretically uncharted.

"""
Discovery complexity estimation: tools for characterizing
the difficulty of a discovery domain based on properties
of its hypothesis space and data distribution.
"""

import numpy as np
from scipy.stats import entropy
from sklearn.neighbors import NearestNeighbors
from typing import Optional


def estimate_hypothesis_space_complexity(
    hypotheses: list[str],
    embeddings: np.ndarray,
    data_embeddings: np.ndarray,
) -> dict[str, float]:
    """
    Estimate the structural complexity of a discovery domain
    by analyzing the geometry of its hypothesis space relative
    to the data distribution.

    Parameters
    ----------
    hypotheses : list[str]
        Textual descriptions of hypotheses in the domain.
    embeddings : np.ndarray
        Embeddings of the hypotheses, shape (n_hypotheses, dim).
    data_embeddings : np.ndarray
        Embeddings of data points / observations, shape (n_data, dim).

    Returns
    -------
    dict with complexity metrics:
        - effective_dimensionality: intrinsic dimensionality of
          the hypothesis space (higher = harder to search)
        - coverage_ratio: fraction of data space covered by
          nearest hypothesis (higher = better structured domain)
        - hypothesis_entropy: entropy of the hypothesis distribution
          (higher = more uniform / harder to find good hypotheses)
        - alignment: correlation between hypothesis and data
          structure (higher = domain is more learnable)
    """
    n_hyp, dim = embeddings.shape

    # 1. Effective dimensionality via PCA
    centered = embeddings - embeddings.mean(axis=0)
    _, singular_values, _ = np.linalg.svd(centered, full_matrices=False)
    explained_variance = singular_values ** 2
    explained_variance /= explained_variance.sum()
    cumulative = np.cumsum(explained_variance)
    # Dimensionality = number of components for 90% variance
    effective_dim = int(np.searchsorted(cumulative, 0.9)) + 1

    # 2. Coverage ratio: for each data point, distance to nearest hypothesis
    nn = NearestNeighbors(n_neighbors=1, metric="cosine")
    nn.fit(embeddings)
    distances, _ = nn.kneighbors(data_embeddings)
    coverage = float(np.mean(distances < 0.5))

    # 3. Hypothesis entropy (discretized)
    # Cluster hypotheses and compute entropy of cluster sizes
    from sklearn.cluster import MiniBatchKMeans
    n_clusters = min(20, n_hyp // 2) if n_hyp > 4 else 2
    kmeans = MiniBatchKMeans(n_clusters=n_clusters, random_state=42)
    labels = kmeans.fit_predict(embeddings)
    _, counts = np.unique(labels, return_counts=True)
    hyp_entropy = float(entropy(counts / counts.sum()))

    # 4. Alignment: simplified CCA-style correlation between
    #    hypothesis and data manifolds (cosine similarity
    #    between principal components)
    data_centered = data_embeddings - data_embeddings.mean(axis=0)
    _, _, Vh_hyp = np.linalg.svd(centered, full_matrices=False)
    _, _, Vh_data = np.linalg.svd(data_centered, full_matrices=False)
    k = min(5, min(Vh_hyp.shape[0], Vh_data.shape[0]))
    alignment_matrix = Vh_hyp[:k] @ Vh_data[:k].T
    alignment = float(np.mean(np.abs(np.diag(alignment_matrix[:k, :k]))))

    return {
        "effective_dimensionality": effective_dim,
        "coverage_ratio": coverage,
        "hypothesis_entropy": hyp_entropy,
        "alignment": alignment,
        "estimated_difficulty": (
            "easy" if alignment > 0.7 and coverage > 0.6
            else "hard" if alignment < 0.3 or coverage < 0.2
            else "medium"
        ),
    }


def discovery_sample_complexity_bound(
    hypothesis_space_size: int,
    confidence: float = 0.95,
    accuracy: float = 0.1,
    description_length_penalty: float = 1.0,
) -> int:
    """
    Estimate the minimum number of experiments needed to
    identify the best hypothesis with given confidence.

    Uses a PAC-style bound adapted for the discovery setting.
    The bound is:

        n >= (1 / (2 * epsilon^2)) * ln(2 * |H| / delta)

    where epsilon is the accuracy parameter and delta = 1 - confidence.
    The description length penalty adjusts for MDL-regularized
    hypothesis selection.

    This is a LOOSE bound; tighter domain-specific bounds
    require structural assumptions about the hypothesis class
    and the data-generating process.
    """
    delta = 1.0 - confidence
    epsilon = accuracy

    # Standard Hoeffding-based PAC bound
    n_base = (1.0 / (2 * epsilon ** 2)) * np.log(
        2 * hypothesis_space_size / delta
    )

    # Adjust for MDL regularization (tighter because we
    # prefer simpler hypotheses, reducing effective |H|)
    effective_size = hypothesis_space_size ** (1.0 / (1.0 + description_length_penalty))
    n_mdl = (1.0 / (2 * epsilon ** 2)) * np.log(
        2 * effective_size / delta
    )

    return {
        "n_unregularized": int(np.ceil(n_base)),
        "n_mdl_regularized": int(np.ceil(n_mdl)),
        "compression_ratio": n_mdl / n_base if n_base > 0 else 1.0,
        "note": (
            "These are loose upper bounds. Domain-specific structure "
            "(symmetries, conservation laws, smoothness) can dramatically "
            "reduce the actual sample complexity."
        ),
    }

3. The Infrastructure Gap

Discovery AI systems today are built on infrastructure designed for other purposes: machine learning (ML) training frameworks, database systems, workflow orchestrators. The field needs purpose-built infrastructure in four areas.

3.1 Shared Experiment Registries

Chapter 47 built experiment registries for individual projects. The field needs shared registries that allow discovery systems to learn from each other's experiments. If one AI agent has already tested a hypothesis and found it false, another agent should not waste resources re-testing it. This requires standardized experiment description formats, shared ontologies for research domains, and access control mechanisms that respect intellectual property while maximizing collective efficiency.

The technical challenges are significant. Experiment descriptions must capture enough detail to determine whether two experiments test the same hypothesis. Semantic similarity alone falls short: identical words can describe very different experiments in different contexts. Negative results require the same careful recording as positive results, which conflicts with the publication bias that plagues human science. The registry must also handle the combinatorial explosion of possible experiments without becoming unusably slow.

3.2 Reproducibility Infrastructure

Reproducibility in Discovery AI has a dual requirement: the AI system's behavior must be reproducible (given the same inputs and random seeds, it produces the same outputs), and the discoveries it makes must be independently verifiable (another system or human can confirm the result using different methods). Current systems satisfy neither requirement reliably.

The behavior reproducibility problem is in principle addressable by containerization, deterministic random seeds, and version-pinned dependencies, though in practice nondeterminism in GPU floating-point operations and LLM API versioning still introduces variability that these measures alone do not eliminate. The verification problem is harder: it requires a community of independent agents (or humans) willing to invest resources in replication studies. This is the discovery analogue of the "replication crisis" in human science, but with the advantage that AI replications can be automated and scaled.

3.3 Discovery-Native Compute Platforms

Scientific discovery has computational patterns that differ from ML training: long-running simulations with checkpointing needs, heterogeneous hardware requirements (GPUs for ML, CPUs for molecular dynamics, quantum processing units (QPUs) for quantum chemistry), and irregular communication patterns between exploration and exploitation phases. Cloud platforms designed for web services or ML training are suboptimal for these workloads. Discovery-native platforms would provide first-class support for experiment management, result caching, and adaptive resource allocation based on research progress.

3.4 Interoperability Standards

Discovery AI components from different research groups cannot easily interoperate. A hypothesis generator from one lab cannot feed directly into an experiment designer from another. The Model Context Protocol (MCP) introduced in Chapter 12 provides a foundation for tool interoperability, but discovery requires higher-level standards: common formats for hypotheses, experimental protocols, datasets, and results. The scientific community has domain-specific standards (CIF for crystallography, PDB for proteins, NetCDF for climate data), but there is no cross-domain standard for discovery artifacts.

Closing the infrastructure gap is necessary but not sufficient, because shared registries, reproducibility tools, and interoperability standards all depend on a social contract: agreements about who contributes what, who gets credit, and what counts as a verified result.

4. The Norms Gap

Scientific norms, the unwritten rules about credit, authorship, peer review, and intellectual honesty, were designed for human scientists. Discovery AI challenges every one of these norms.

4.1 Credit and Authorship

When an AI system generates a hypothesis that a human scientist validates experimentally, who deserves credit? Current norms require that authors take intellectual responsibility for a publication. An AI system cannot take responsibility in any meaningful legal or ethical sense. But denying all credit to the AI misrepresents the intellectual contribution, especially when the hypothesis was one that no human on the team would have generated.

The emerging consensus (still fragile) is a contributor model rather than an authorship model: AI systems are acknowledged as contributors with specified roles, similar to how specialized equipment or software is acknowledged. But this analogy breaks down when the AI's contribution is intellectual rather than mechanical. The field needs new norms that are honest about the AI's role without anthropomorphizing it.

4.2 Peer Review

If AI systems can generate research ideas that match human novelty ratings (as Si et al., 2024, demonstrated), they can also generate peer reviews. This creates a recursive problem: AI-generated papers reviewed by AI-generated reviews, with no human in the loop who deeply understands the work. The risk is not that the reviews will be low quality (they may be excellent) but that the review process loses its function as a social mechanism for establishing trust and consensus within a research community.

A more constructive direction is to use AI as a pre-review filter: AI systems check for methodological errors, statistical problems, and reproducibility gaps before human reviewers invest their time. This preserves the human social function of peer review while leveraging AI for quality control. The claim validation pipeline from Chapter 41 provides the technical foundation for such pre-review systems.

4.3 Intellectual Honesty

Messeri and Crockett (2024) warn of "illusions of understanding": AI systems that produce plausible-sounding explanations that satisfy human cognitive needs without actually providing genuine understanding. A Discovery AI system that generates a hypothesis, tests it computationally, and reports a positive result may create an illusion of discovery when the hypothesis was trivially true, the test was poorly designed, or the positive result reflects overfitting rather than a real phenomenon.

The antidote is adversarial verification: systematically attempting to falsify AI-generated claims through methods independent of the originating AI. Karl Popper advocated this falsificationist program for all science; Discovery AI makes it urgent. The evaluation framework from Chapter 56 provides the technical tools. The missing norm is a community expectation that AI-generated discoveries carry a higher burden of independent verification than human-generated ones, at least until track records calibrate trust.

5. Scientific Institutions in the Agent Era

Beyond individual norms, the institutional structures of science (universities, funding agencies, journals, professional societies) must adapt to a world where AI agents are active participants in research. Table 58.3 maps the adaptations needed across four institutional pillars.

6. Discovery as Socio-Technical Infrastructure

The institutional adaptations outlined above share a common thread: none of them is purely technical or purely social, and none can succeed in isolation from the others.

Discovery AI is not merely a set of tools. It is socio-technical infrastructure: a system that combines technical components (models, algorithms, compute) with social components (norms, institutions, incentives) in a way that neither can function without the other. A brilliant discovery algorithm deployed without appropriate norms for verification and credit is dangerous. A comprehensive governance framework without capable technical systems is aspirational but inert.

Building this infrastructure is the work of the next decade. The following roadmap tracker provides a concrete framework for monitoring progress across all four dimensions. Figure 58.3.1 illustrates the four-dimensional gap framework for Discovery AI maturity.

"""
Discovery AI Roadmap Tracker: monitors progress across the
four dimensions (benchmarks, theory, infrastructure, norms)
needed for Discovery AI to become reliable socio-technical
infrastructure.
"""

import json
from dataclasses import dataclass, field
from datetime import datetime
from pathlib import Path
from typing import Optional
from enum import Enum


class Dimension(Enum):
    BENCHMARKS = "benchmarks"
    THEORY = "theory"
    INFRASTRUCTURE = "infrastructure"
    NORMS = "norms"


class Maturity(Enum):
    NASCENT = "nascent"           # Problem recognized, no solutions
    EMERGING = "emerging"         # Early solutions, no consensus
    DEVELOPING = "developing"     # Multiple approaches, partial adoption
    ESTABLISHED = "established"   # Consensus solution, broad adoption
    MATURE = "mature"             # Stable, refinement-only changes


@dataclass
class Milestone:
    """A concrete milestone on the Discovery AI roadmap."""
    milestone_id: str
    dimension: Dimension
    title: str
    description: str
    maturity: Maturity
    blocking_issues: list[str] = field(default_factory=list)
    key_references: list[str] = field(default_factory=list)
    estimated_timeline: str = ""
    last_assessed: str = field(
        default_factory=lambda: datetime.now().strftime("%Y-%m")
    )


class DiscoveryAIRoadmap:
    """
    Tracks the field's progress toward reliable Discovery AI
    across benchmarks, theory, infrastructure, and norms.

    Designed to be updated periodically (quarterly or annually)
    as a community resource for identifying high-priority gaps.
    """

    def __init__(self):
        self.milestones: list[Milestone] = []
        self._initialize_milestones()

    def _initialize_milestones(self):
        """Populate with the current state of the field (mid-2026)."""

        # === BENCHMARKS ===
        self.milestones.extend([
            Milestone(
                milestone_id="B1",
                dimension=Dimension.BENCHMARKS,
                title="Component-level discovery benchmarks",
                description=(
                    "Benchmarks for individual discovery stages: "
                    "hypothesis generation, experiment design, "
                    "data analysis. Examples: MLAgentBench, "
                    "ScienceAgentBench."
                ),
                maturity=Maturity.EMERGING,
                blocking_issues=[
                    "Limited domain coverage",
                    "No standardized evaluation metrics",
                ],
                key_references=["MLAgentBench (2024)", "ScienceAgentBench (2024)"],
                estimated_timeline="2025-2027",
            ),
            Milestone(
                milestone_id="B2",
                dimension=Dimension.BENCHMARKS,
                title="Full-loop discovery benchmarks",
                description=(
                    "Benchmarks that evaluate the complete discovery "
                    "cycle from problem identification to verified "
                    "conclusion, within specific domains."
                ),
                maturity=Maturity.NASCENT,
                blocking_issues=[
                    "Ground truth requires retrospective validation",
                    "High cost of benchmark construction",
                    "Domain specificity limits generalization",
                ],
                estimated_timeline="2027-2030",
            ),
            Milestone(
                milestone_id="B3",
                dimension=Dimension.BENCHMARKS,
                title="Cross-domain discovery benchmarks",
                description=(
                    "Benchmarks that test transfer of discovery "
                    "capability across scientific domains."
                ),
                maturity=Maturity.NASCENT,
                blocking_issues=[
                    "No formal definition of cross-domain transfer",
                    "Requires multiple domain-specific benchmarks first",
                ],
                estimated_timeline="2029-2033",
            ),
        ])

        # === THEORY ===
        self.milestones.extend([
            Milestone(
                milestone_id="T1",
                dimension=Dimension.THEORY,
                title="Sample complexity of discovery",
                description=(
                    "Formal bounds on experiments needed to identify "
                    "scientific laws within restricted hypothesis classes."
                ),
                maturity=Maturity.NASCENT,
                blocking_issues=[
                    "Requires formal definition of hypothesis classes",
                    "Discovery involves growing hypothesis spaces",
                ],
                estimated_timeline="2027-2032",
            ),
            Milestone(
                milestone_id="T2",
                dimension=Dimension.THEORY,
                title="Domain learnability characterization",
                description=(
                    "Formal measures of how amenable a scientific "
                    "domain is to AI-driven discovery, based on "
                    "structural properties."
                ),
                maturity=Maturity.NASCENT,
                blocking_issues=[
                    "No agreed-upon formalization of 'domain structure'",
                ],
                estimated_timeline="2028-2035",
            ),
        ])

        # === INFRASTRUCTURE ===
        self.milestones.extend([
            Milestone(
                milestone_id="I1",
                dimension=Dimension.INFRASTRUCTURE,
                title="Shared experiment registries",
                description=(
                    "Cross-institution registries where discovery "
                    "agents share experimental results to avoid "
                    "redundant work."
                ),
                maturity=Maturity.NASCENT,
                blocking_issues=[
                    "No standard experiment description format",
                    "IP and competition concerns",
                    "Negative result recording incentives",
                ],
                estimated_timeline="2027-2030",
            ),
            Milestone(
                milestone_id="I2",
                dimension=Dimension.INFRASTRUCTURE,
                title="Discovery artifact standards",
                description=(
                    "Machine-readable formats for discovery outputs "
                    "that enable interoperability between systems."
                ),
                maturity=Maturity.EMERGING,
                blocking_issues=[
                    "Domain-specific standards exist but no cross-domain",
                    "Community coordination challenge",
                ],
                key_references=["CIF, PDB, NetCDF (domain-specific)"],
                estimated_timeline="2026-2029",
            ),
        ])

        # === NORMS ===
        self.milestones.extend([
            Milestone(
                milestone_id="N1",
                dimension=Dimension.NORMS,
                title="AI contribution disclosure standards",
                description=(
                    "Agreed-upon standards for disclosing AI "
                    "contributions to research publications."
                ),
                maturity=Maturity.EMERGING,
                blocking_issues=[
                    "No consensus on granularity of disclosure",
                    "Enforcement mechanisms unclear",
                ],
                key_references=[
                    "ICML 2024 AI disclosure policy",
                    "Nature AI use policy (2024)",
                ],
                estimated_timeline="2025-2027",
            ),
            Milestone(
                milestone_id="N2",
                dimension=Dimension.NORMS,
                title="AI-generated discovery verification norms",
                description=(
                    "Community expectations for independent "
                    "verification of AI-generated scientific claims."
                ),
                maturity=Maturity.NASCENT,
                blocking_issues=[
                    "Cost of independent verification",
                    "Unclear responsibility for verification",
                ],
                estimated_timeline="2027-2030",
            ),
        ])

    def get_status_report(self) -> dict:
        """Generate a summary of roadmap progress."""
        by_dimension = {}
        for dim in Dimension:
            milestones = [
                m for m in self.milestones if m.dimension == dim
            ]
            maturity_counts = {}
            for m in milestones:
                mat = m.maturity.value
                maturity_counts[mat] = maturity_counts.get(mat, 0) + 1
            by_dimension[dim.value] = {
                "n_milestones": len(milestones),
                "maturity_distribution": maturity_counts,
                "blocking_issues": [
                    issue
                    for m in milestones
                    for issue in m.blocking_issues
                ],
            }

        # Overall maturity score (0-4 scale)
        maturity_scores = {
            Maturity.NASCENT: 0,
            Maturity.EMERGING: 1,
            Maturity.DEVELOPING: 2,
            Maturity.ESTABLISHED: 3,
            Maturity.MATURE: 4,
        }
        all_scores = [
            maturity_scores[m.maturity] for m in self.milestones
        ]
        overall = sum(all_scores) / len(all_scores) if all_scores else 0

        return {
            "assessment_date": datetime.now().strftime("%Y-%m"),
            "overall_maturity": round(overall, 2),
            "overall_maturity_label": (
                "nascent" if overall < 0.5
                else "emerging" if overall < 1.5
                else "developing" if overall < 2.5
                else "established" if overall < 3.5
                else "mature"
            ),
            "by_dimension": by_dimension,
            "highest_priority_gaps": [
                m.title
                for m in self.milestones
                if m.maturity == Maturity.NASCENT
            ],
        }

    def save(self, path: str) -> None:
        """Export roadmap for sharing."""
        data = {
            "milestones": [
                {
                    "id": m.milestone_id,
                    "dimension": m.dimension.value,
                    "title": m.title,
                    "description": m.description,
                    "maturity": m.maturity.value,
                    "blocking_issues": m.blocking_issues,
                    "key_references": m.key_references,
                    "estimated_timeline": m.estimated_timeline,
                    "last_assessed": m.last_assessed,
                }
                for m in self.milestones
            ],
        }
        Path(path).parent.mkdir(parents=True, exist_ok=True)
        with open(path, "w") as f:
            json.dump(data, f, indent=2)


# Usage example
if __name__ == "__main__":
    roadmap = DiscoveryAIRoadmap()
    report = roadmap.get_status_report()

    print(f"Overall maturity: {report['overall_maturity_label']}")
    print(f"  Score: {report['overall_maturity']} / 4.0")
    print(f"\nHighest priority gaps:")
    for gap in report["highest_priority_gaps"]:
        print(f"  - {gap}")

    print(f"\nBy dimension:")
    for dim, info in report["by_dimension"].items():
        print(f"  {dim}: {info['maturity_distribution']}")

7. What This Book Has Built, and What Comes Next

This book began with a simple observation: scientific discovery is a search problem (Chapter 1). From that starting point, the book built an increasingly sophisticated account of how AI systems participate in that search: representing knowledge (Chapter 3), reasoning about it (Chapter 4), building software that embodies it (Chapter 9), learning from data (Chapter 26), generating hypotheses (Chapter 39), designing experiments (Chapter 46), and deploying autonomous agents that do all of this in concert (Chapter 53).

The Discovery Workbench that has grown across all 58 chapters is not a finished system. It is a platform: a set of integrated components that a practitioner can assemble, extend, and deploy for their specific research domain. The workbench includes:

• Knowledge representation and retrieval (Chapters 3, 37, 38)
• Hypothesis generation and validation (Chapters 39, 41)
• Experiment design and optimization (Chapters 45, 46)
• Experiment tracking and provenance (Chapter 47)
• Multi-agent coordination (Chapter 54)
• Evaluation and benchmarking (Chapter 56)
• Responsible deployment safeguards (Chapter 57)
• Innovation scoring and co-discovery management (Chapter 58)

The platform's remaining gaps are what this section has mapped: better benchmarks to know when it works, better theory to know why, shared infrastructure for interoperability, and norms for responsible use. No single research group can close these gaps; they demand coordination across computer science, the natural sciences, the social sciences, and science policy.

The field of Discovery AI is, in the terminology of Section 58.1, at approximately Level 3 on the discovery autonomy scale: conditionally autonomous systems that can execute research programs within human-defined boundaries. The path to Level 4 (problem-finding) and beyond is visible but not yet paved. Paving it will require the benchmarks, theory, infrastructure, and norms described in this section. It will require the co-discovery partnerships described in Section 58.2. And it will require practitioners who understand both the capabilities and the limitations of these systems, which is what this book has aimed to provide. In short: The hardest part of building Discovery AI is not the algorithms; it is the agreements.