Chapter 56: Evaluating Discovery Systems

Overview

Building a system that generates scientific hypotheses, designs experiments, or discovers new materials is only half the challenge. The other half is knowing whether the system actually works. Evaluating discovery systems is harder than evaluating classifiers or translation models because the outputs are inherently open-ended: a novel hypothesis cannot be scored against a fixed ground truth without first verifying it experimentally.

This chapter develops a complete evaluation framework for autonomous discovery systems. We begin by defining what "good discovery" means along five axes: novelty, impact, reproducibility, validity, and efficiency. We survey the major benchmarks that the community has developed for measuring scientific AI capabilities, from ScienceAgentBench and DiscoveryBench to GPQA Diamond and FrontierMath. We then confront the threats that undermine evaluation: data contamination, construct mismatch, and the gap between benchmark performance and real-world discovery. Finally, we build a working evaluation suite for the Discovery Workbench that integrates automated rubrics, human annotation with calibrated inter-rater agreement, statistical power analysis, and CI-driven regression testing.

The evaluation toolkit developed here connects back to Chapter 23: Evaluating AI Coding Agents (which introduced SWE-bench and agent evaluation methodology), Chapter 41: Scientific Claim Validation (which covered evidence assessment), and Chapter 47: Experiment Registries and Scientific Provenance (which provided the provenance infrastructure that makes evaluation reproducible).

Prerequisites

This chapter assumes familiarity with the Discovery Workbench architecture from Chapter 6, the agent evaluation methodology from Chapter 23, and basic statistics (hypothesis testing, confidence intervals). Readers unfamiliar with inter-rater reliability should review Cohen's kappa in Appendix A before Section 56.4.

Learning Outcomes

Define and measure five dimensions of discovery quality: novelty, impact, reproducibility, validity, and efficiency.
Compare the scope, construction, and saturation status of ScienceAgentBench, DiscoveryBench, MLE-bench, SWE-bench Verified, GPQA Diamond, and FrontierMath.
Identify construct validity threats including data contamination, metric gaming, and benchmark-to-deployment gaps.
Design human evaluation protocols with calibrated rubrics and compute required sample sizes via power analysis.
Compute inter-rater agreement using Cohen's kappa and Krippendorff's alpha, and interpret the results.
Build an automated evaluation suite for the Discovery Workbench with task banks, gold standards, and CI integration.

Sections

56.1 What It Means to Discover

Five axes of discovery quality: novelty, impact, reproducibility, validity, and efficiency. Formalizing each as a measurable quantity with rubrics and scoring functions.

56.2 Benchmarks for Discovery

ScienceAgentBench, DiscoveryBench, MLE-bench, SWE-bench Verified/Live, GPQA Diamond, and FrontierMath. Benchmark anatomy, saturation curves, and what each actually measures.

56.3 Validity and Contamination

Construct validity threats, data contamination detection, Goodhart's Law in benchmarks, human evaluation protocols, inter-rater agreement (Cohen's kappa, Krippendorff's alpha), and power analysis for evaluation studies.

56.4 Building a Discovery Evaluation Suite

Recipe: a complete evaluation harness for the Discovery Workbench with task types, rubrics, gold-standard curation, automated scoring, human annotation pipelines, and CI-driven regression comparison.

What's Next

With a rigorous evaluation framework in place, Chapter 57: Responsible Discovery AI turns to the ethical, safety, and governance dimensions of autonomous discovery. Evaluation measures whether the system works; responsible deployment asks whether it should work in a given context, who benefits, and what guardrails prevent harm. The evaluation infrastructure from this chapter directly supports the auditing and monitoring techniques introduced there.

Bibliography

Benchmark Papers

Chen, Z., et al. (2024). ScienceAgentBench: Toward rigorous assessment of language agents for data-driven scientific discovery. NeurIPS 2024 Datasets and Benchmarks Track.

Introduces 102 tasks spanning data analysis, visualization, and scientific computation, requiring agents to write and execute code for data-driven discovery. The gold standard for evaluating end-to-end scientific agents.

Majumder, B. P., et al. (2024). DiscoveryBench: Towards data-driven discovery with large language models. arXiv:2407.01725.

A benchmark of 264 tasks requiring hypothesis generation and verification from real scientific datasets. Distinguishes between hypothesis-driven and data-driven discovery modes.

Chan, J. S., et al. (2024). MLE-bench: Evaluating machine learning agents on machine learning engineering. arXiv:2410.07095.

OpenAI's benchmark of 75 Kaggle competitions for evaluating ML engineering agents. Provides medal-based scoring that maps to real competitive performance tiers.

Jimenez, C. E., et al. (2024). SWE-bench: Can language models resolve real-world GitHub issues? ICLR 2024.

The foundational software engineering benchmark. SWE-bench Verified (500 human-validated instances) and SWE-bench Live (continuously updated from new issues) address contamination and staleness.

Rein, D., et al. (2024). GPQA: A graduate-level Google-proof question-answering benchmark. ICML 2024.

448 expert-written questions in biology, chemistry, and physics that domain PhDs answer at only 65% accuracy. The Diamond subset (198 questions) is approaching saturation by frontier models.

Glazer, E., et al. (2024). FrontierMath: A benchmark for evaluating advanced mathematical reasoning in AI. arXiv:2411.04872.

Hundreds of original, research-level mathematics problems where frontier models score below 2%. Designed to remain unsaturated for years by requiring genuine mathematical insight.

Evaluation Methodology

Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37-46.

Introduced Cohen's kappa for measuring inter-rater agreement beyond chance. The standard reliability metric for two annotators in NLP and AI evaluation.

Krippendorff, K. (2004). Content Analysis: An Introduction to Its Methodology, 2nd ed. Sage.

Defines Krippendorff's alpha, a reliability coefficient that handles any number of annotators, missing data, and multiple measurement scales.

Cohen, J. (1992). A power primer. Psychological Bulletin, 112(1), 155-159.

The classic reference for statistical power analysis. Essential for determining how many evaluation instances are needed to detect a meaningful difference between systems.

Contamination and Validity

Oren, Y., et al. (2024). Proving test set contamination in black-box language models. ICLR 2024.

Methods for detecting whether benchmark data has leaked into training sets, including membership inference and order-sensitivity tests.

Zhang, H., et al. (2024). Careful examination of large language model performance on graduate-level science questions. arXiv:2402.01781.

Demonstrates that frontier model performance on GPQA depends heavily on whether questions appeared in public pretraining corpora, motivating the Diamond subset.

Tools and Libraries

MLflow

Open-source platform for managing the ML lifecycle. Used in this chapter to track evaluation runs, compare system versions, and log metrics with full reproducibility.

SciPy

Scientific computing library providing the statistical functions (bootstrap confidence intervals, power analysis, agreement coefficients) used throughout this chapter.

Label Studio

Open-source data labeling platform used for structuring human annotation of discovery outputs with configurable rubrics and annotation interfaces.