56.1 What It Means to Discover

1. The Evaluation Problem in Discovery

What do you call a system that generates ten thousand hypotheses overnight, every one of them novel by any metric you choose, yet not a single one worth testing? Answering that question forces us to confront a problem far harder than standard model evaluation: there is no answer key for discoveries that nobody has made yet. Evaluating a discovery system is fundamentally harder than scoring a classifier for three reasons.

First, ground truth is unavailable at evaluation time. If we already knew all the valid discoveries, we would not need a discovery system. A system that proposes a novel catalyst for CO2 reduction cannot be scored against a label set because the correct answer may not yet be known to anyone.

Second, discovery is multi-dimensional. A hypothesis can be novel but wrong, correct but trivial, impactful but irreproducible. Collapsing these dimensions into a single score obscures the failure modes that matter most.

Third, discovery value is context-dependent. A finding that is old news in one field may be a breakthrough when transferred to another. The significance of a result depends on the state of knowledge at the time of evaluation, which changes continuously. In short: Discovery evaluation requires a vector, not a scalar, because every collapsed score hides a failure mode.

2. The Five Axes of Discovery Quality

Without a principled way to decompose "good discovery," teams routinely ship systems that optimize a single axis (typically novelty) while silently failing on others. In one widely cited cautionary case, a pharmaceutical team reported that an AI tool generated hundreds of novel molecular candidates, none of which survived the first round of wet-lab validation, consuming months of synthesis time.

Five axes together capture the principal dimensions of discovery quality, as illustrated in Figure 56.1. Each axis is independent: a system can score high on one and low on another. These axes typically mirror the criteria that peer reviewers at top scientific journals apply, recast for automated and semi-automated measurement.

2.1 Novelty: Is It New?

Novelty measures the degree to which a discovery output differs from existing knowledge. A hypothesis that restates a well-known result has zero novelty; a hypothesis that proposes a previously unconsidered mechanism has high novelty. We formalize novelty using information-theoretic distance (a family of measures that quantify how different two representations are in terms of their information content).

Novelty scoring quantifies how different a candidate discovery is from everything already known. The score ranges from 0 (identical to prior work) to 1 (completely unprecedented). Without a formal novelty measure, a discovery system can game evaluations by rephrasing known results, producing outputs that look productive but add nothing to the knowledge frontier. The scorer encodes both the candidate and every item in the existing knowledge base as vectors in a shared embedding space, then computes the cosine distance to the nearest neighbor. A large minimum distance means the candidate occupies a region of concept space that no prior work has reached. Use embedding-based novelty when you have a large, well-indexed knowledge corpus and need fast, automated scoring; switch to expert panel review when the domain is too narrow for general-purpose embeddings to capture meaningful distinctions.

Let \(\mathcal{K}\) denote the corpus of existing knowledge (papers, databases, known results) and let \(d\) denote a candidate discovery. We define novelty as the minimum semantic distance between \(d\) and any element of \(\mathcal{K}\):

In practice, we embed both \(d\) and elements of \(\mathcal{K}\) into a shared vector space using a scientific language model and compute cosine distance (where cosine distance between two vectors is 1 minus the cosine of the angle between them, ranging from 0 for identical directions to 1 for orthogonal ones). A discovery is novel when it lies far from every known result in embedding space.

import numpy as np
from dataclasses import dataclass
from typing import List, Optional


@dataclass
class DiscoveryOutput:
    """A single output from a discovery system."""
    hypothesis: str
    evidence: List[str]
    methodology: str
    predicted_effect_size: Optional[float] = None
    compute_cost_seconds: float = 0.0
    token_cost: int = 0


def compute_novelty(
    discovery_embedding: np.ndarray,
    knowledge_embeddings: np.ndarray,
) -> float:
    """Compute novelty as max cosine distance to nearest known result.

    Args:
        discovery_embedding: (d,) vector for the candidate discovery
        knowledge_embeddings: (n, d) matrix of known results

    Returns:
        Novelty score in [0, 1] where 1 is maximally novel
    """
    # Normalize to unit vectors
    d_norm = discovery_embedding / (np.linalg.norm(discovery_embedding) + 1e-10)
    k_norms = knowledge_embeddings / (
        np.linalg.norm(knowledge_embeddings, axis=1, keepdims=True) + 1e-10
    )

    # Cosine similarity with every known result
    similarities = k_norms @ d_norm  # shape (n,)

    # Novelty = 1 - max_similarity (farther from known = more novel)
    max_similarity = np.max(similarities)
    return float(1.0 - max_similarity)

A subtlety: novelty computed over a static knowledge corpus becomes stale as new papers appear. The Discovery Workbench addresses this by periodically re-indexing the knowledge base (see Chapter 37) and recomputing novelty scores against the updated corpus.

2.2 Impact: Does It Matter?

Novelty without impact is trivia. A discovery is impactful when it changes what scientists believe, what engineers build, or what clinicians prescribe. Impact is inherently forward-looking and difficult to measure at the time of discovery. We approximate it using two proxies.

Predicted downstream utility: How many existing open problems would this discovery help resolve? We measure this by counting the number of known open questions (from a curated problem bank or from research agendas) whose solution would be advanced by the discovery.

Information gain: How much does this discovery reduce uncertainty about the world? We formalize this as the KL divergence between the posterior belief after incorporating the discovery and the prior belief before it:

where \(\theta\) represents the parameters of a scientific model. A discovery that shifts the posterior substantially relative to the prior carries high information gain.

2.3 Reproducibility: Can Others Verify It?

A discovery that cannot be reproduced is, for practical purposes, not a discovery. The replication crisis in psychology, biomedicine, and social science has made reproducibility a first-class evaluation criterion. For AI discovery systems, reproducibility has two layers.

Computational reproducibility: Given the same inputs, code, and random seeds, does the system produce the same output? This is a prerequisite that Chapter 47 addresses through experiment registries and provenance tracking.

Scientific reproducibility: If the proposed experiment is actually conducted, do the results match the prediction? This requires either wet-lab validation or simulation with independently coded models.

import hashlib
import json


def check_computational_reproducibility(
    system_fn,
    inputs: dict,
    expected_hash: str,
    n_runs: int = 3,
) -> dict:
    """Verify that a discovery system produces identical outputs across runs.

    Args:
        system_fn: callable that takes inputs and returns a JSON-serializable result
        inputs: the input configuration
        expected_hash: SHA-256 hash of the canonical output
        n_runs: number of independent runs to check

    Returns:
        Dict with reproducibility verdict and details
    """
    hashes = []
    for run_idx in range(n_runs):
        result = system_fn(**inputs)
        result_json = json.dumps(result, sort_keys=True, default=str)
        run_hash = hashlib.sha256(result_json.encode()).hexdigest()
        hashes.append(run_hash)

    all_match = len(set(hashes)) == 1
    matches_expected = all_match and hashes[0] == expected_hash

    return {
        "computationally_reproducible": all_match,
        "matches_gold_standard": matches_expected,
        "unique_hashes": list(set(hashes)),
        "n_runs": n_runs,
    }

2.4 Validity: Is It Correct?

Reproducibility confirms that a result can be obtained again, but it says nothing about whether the result is right. Validity asks whether the discovery is actually true. A novel, impactful, reproducible finding that happens to be wrong is worse than useless because it consumes resources to refute. We decompose validity into three sub-dimensions.

Internal validity: Does the reasoning chain from evidence to conclusion hold? Are the statistical tests appropriate? Are the assumptions stated and justified? This can be partially automated by checking logical consistency and statistical methodology.

The three faces of validity

External validity: Does the finding generalize beyond the specific dataset or conditions? A pattern discovered in one cell line may not hold in others. External validity requires testing on held-out data or independent datasets.

Construct validity: Does the evaluation actually measure what it claims to measure? This meta-level concern, covered in depth in Section 56.3, is the most insidious because a flawed evaluation can make a broken system look excellent.

from enum import Enum


class ValidityLevel(Enum):
    """Levels of validation for a discovery output."""
    UNVALIDATED = 0        # No validation performed
    SELF_CONSISTENT = 1    # Passes internal logic checks
    COMPUTATIONALLY_VERIFIED = 2  # Verified by independent simulation
    EXPERIMENTALLY_CONFIRMED = 3  # Confirmed by wet-lab or field experiment
    INDEPENDENTLY_REPLICATED = 4  # Replicated by an independent group


def score_internal_validity(discovery: DiscoveryOutput) -> dict:
    """Check internal validity of a discovery output.

    Examines the reasoning chain for common statistical and logical errors.
    Returns a checklist of validity criteria with pass/fail status.
    """
    checks = {}

    # Check 1: evidence is provided
    checks["has_evidence"] = len(discovery.evidence) > 0

    # Check 2: methodology is specified
    checks["has_methodology"] = len(discovery.methodology.strip()) > 0

    # Check 3: effect size is quantified (not just "significant")
    checks["quantified_effect"] = discovery.predicted_effect_size is not None

    # Check 4: hypothesis is falsifiable (contains a testable prediction)
    falsifiable_markers = [
        "predict", "expect", "should", "will", "would increase",
        "would decrease", "correlate", "cause", "lead to",
    ]
    checks["falsifiable"] = any(
        marker in discovery.hypothesis.lower()
        for marker in falsifiable_markers
    )

    # Overall internal validity score
    checks["score"] = sum(checks.values()) / len(
        [k for k in checks if k != "score"]
    )

    return checks

2.5 Efficiency: At What Cost?

Two systems that produce equally novel, impactful, reproducible, and valid discoveries are not equally good if one costs a thousand times more. Efficiency measures the resource cost per unit of discovery quality. Resources include compute time, API token expenditure, wet-lab reagent cost, human expert time, and wall-clock latency.

We define a composite efficiency metric that normalizes discovery quality by cost:

where \(Q(d)\) is an aggregate quality score (a weighted combination of the other four axes) and \(C(d)\) is the total cost of producing discovery \(d\). This is analogous to the cost-performance Pareto frontier (the set of configurations where no axis can be improved without worsening another) described in Chapter 45.

@dataclass
class EvaluationCost:
    """Track all costs associated with producing a discovery."""
    compute_seconds: float = 0.0
    api_tokens: int = 0
    api_cost_usd: float = 0.0
    human_hours: float = 0.0
    lab_cost_usd: float = 0.0
    wall_clock_seconds: float = 0.0

    @property
    def total_cost_usd(self) -> float:
        """Estimate total cost in USD.

        Assumes $0.10/hr amortized compute and $150/hr for human experts.
        """
        compute_cost = self.compute_seconds / 3600 * 0.10
        human_cost = self.human_hours * 150
        return compute_cost + self.api_cost_usd + human_cost + self.lab_cost_usd


def compute_efficiency(
    quality_scores: dict,
    cost: EvaluationCost,
    quality_weights: Optional[dict] = None,
) -> float:
    """Compute cost-normalized discovery efficiency.

    Args:
        quality_scores: dict with keys novelty, impact, reproducibility, validity
        cost: EvaluationCost tracking all resource expenditures
        quality_weights: optional weighting; defaults to uniform

    Returns:
        Efficiency score (quality per dollar)
    """
    if quality_weights is None:
        quality_weights = {
            "novelty": 0.25,
            "impact": 0.25,
            "reproducibility": 0.25,
            "validity": 0.25,
        }

    aggregate_quality = sum(
        quality_weights.get(k, 0) * quality_scores.get(k, 0)
        for k in quality_weights
    )

    total_cost = max(cost.total_cost_usd, 0.01)  # avoid division by zero
    return aggregate_quality / total_cost

3. The Discovery Quality Vector

Combining all five axes, we define the discovery quality vector for a system \(S\) evaluated on a set of tasks \(\mathcal{T}\):

where each overbar denotes the mean score for that axis across all tasks in \(\mathcal{T}\). Reporting the full vector, rather than a single aggregate, lets practitioners identify the specific failure mode: a system that scores \([0.9, 0.8, 0.3, 0.7, 0.6]\) has a reproducibility problem that would be invisible in a weighted average. Figure 56.1.1 illustrates Five-axis Discovery Quality Vector.

@dataclass
class DiscoveryQualityVector:
    """The five-axis quality assessment for a discovery system."""
    novelty: float      # [0, 1]
    impact: float       # [0, 1]
    reproducibility: float  # [0, 1]
    validity: float     # [0, 1]
    efficiency: float   # [0, inf), but typically normalized to [0, 1]

    def to_array(self) -> np.ndarray:
        return np.array([
            self.novelty, self.impact,
            self.reproducibility, self.validity,
            self.efficiency,
        ])

    def weighted_aggregate(self, weights: Optional[np.ndarray] = None) -> float:
        """Compute weighted aggregate score. Use with caution."""
        if weights is None:
            weights = np.ones(5) / 5
        return float(weights @ self.to_array())

    def radar_plot_data(self) -> dict:
        """Return data formatted for radar/spider plot visualization."""
        labels = ["Novelty", "Impact", "Reproducibility", "Validity", "Efficiency"]
        values = self.to_array().tolist()
        return {"labels": labels, "values": values}


def evaluate_system(
    system_fn,
    tasks: list,
    knowledge_embeddings: np.ndarray,
    embedding_fn,
) -> DiscoveryQualityVector:
    """Run a discovery system on a task bank and compute the quality vector.

    This is the top-level evaluation function that orchestrates
    all five axis scorers.
    """
    novelty_scores = []
    validity_scores = []
    costs = []

    for task in tasks:
        # Run the system
        output = system_fn(task)
        cost = EvaluationCost(
            compute_seconds=output.compute_cost_seconds,
            api_tokens=output.token_cost,
        )
        costs.append(cost)

        # Score novelty
        emb = embedding_fn(output.hypothesis)
        nov = compute_novelty(emb, knowledge_embeddings)
        novelty_scores.append(nov)

        # Score internal validity
        val_result = score_internal_validity(output)
        validity_scores.append(val_result["score"])

    return DiscoveryQualityVector(
        novelty=float(np.mean(novelty_scores)),
        impact=0.0,  # requires human or LLM-judge scoring
        reproducibility=0.0,  # requires multi-run checking
        validity=float(np.mean(validity_scores)),
        efficiency=float(np.mean([
            1.0 / max(c.total_cost_usd, 0.01) for c in costs
        ])),
    )

import mlflow

with mlflow.start_run(run_name="discovery-system-v2.1"):
    mlflow.log_param("model", "claude-opus-4-20250514")
    mlflow.log_param("task_bank_version", "2024-Q4")

    # Log the full quality vector
    qv = evaluate_system(system_fn, tasks, knowledge_embs, embed_fn)
    mlflow.log_metric("novelty", qv.novelty)
    mlflow.log_metric("impact", qv.impact)
    mlflow.log_metric("reproducibility", qv.reproducibility)
    mlflow.log_metric("validity", qv.validity)
    mlflow.log_metric("efficiency", qv.efficiency)
    mlflow.log_metric("aggregate", qv.weighted_aggregate())

4. Rubric Design for Human Evaluation

The quality vector and its automated scorers handle axes like novelty and reproducibility reasonably well, but impact and validity often require human judgment. The challenge is making that judgment consistent across raters. We achieve this through calibrated rubrics (structured scoring guides that pair each quality level with a concrete anchor example so that independent raters converge on the same score).

Table 56.1 shows a rubric for the impact axis. Each level has a verbal description and a concrete anchor example that raters study during calibration sessions before they begin scoring.

Section 56.3 develops the statistical machinery for ensuring raters apply these rubrics consistently (Cohen's kappa, a statistic measuring inter-rater agreement adjusted for chance, Krippendorff's alpha, its generalization to multiple raters and variable scales, and power analysis), and Section 56.4 integrates rubric-based human evaluation into the full evaluation suite.

5. From Axes to Benchmarks

With the five axes defined and both automated scorers and human rubrics in hand, we have the vocabulary for discovery quality but not yet a shared test suite for cross-group comparison. A benchmark fills that gap: a standardized task set, gold-standard answers where available, and a scoring protocol that makes results comparable. The next section surveys the benchmarks the community has built for scientific discovery evaluation, examines what each measures, and identifies where gaps remain.