23.2 Validity and Contamination

The Big Picture

A benchmark score is only meaningful if it measures what it claims to measure. For coding agent benchmarks, the claim is: "this score predicts how well the agent will perform on new, unseen coding tasks." Every validity threat weakens this predictive link. Data contamination means the agent has seen the answers during training. Weak test patches mean incorrect solutions get counted as correct. Retry-based strategies inflate resolve rates at the cost of latency and dollars. And the structural differences between benchmark repositories and real-world codebases mean that even a perfectly valid benchmark score may not transfer to your deployment context. This section equips you to detect and quantify each of these threats. Figure 23.2.1 illustrates validity threat layers between benchmark score and production performance.

Figure 23.2 maps the four validity threats that stand between a raw benchmark score and a trustworthy capability estimate. Each threat acts as a filter: if any one is unaddressed, the score that reaches the right side of the diagram may bear little relation to real-world performance. The sections that follow examine each threat in detail.

1. Data Contamination and Training-Set Overlap

Imagine an agent that solves a Django cache bug in four seconds flat, producing a patch identical to the one merged eighteen months earlier. Is that brilliance, or is it recall from a training set that included the pull request? This is the problem of data contamination: the model has already seen the benchmark's tasks, answers, or closely related content during pre-training. SWE-bench tasks come from public GitHub issues and pull requests. Large language models train on massive corpora of public code, so some degree of overlap is inevitable. The question is how much overlap exists and how severely it affects scores.

When contamination goes undetected, teams ship agents whose "state of the art" scores collapse the moment they face a codebase that was never in the training set, wasting months of integration work on a capability that was really just recall.

Data contamination occurs when evaluation data (questions, answers, or both) appears inside a model's training corpus, inflating measured performance beyond true generalization ability. It matters because a contaminated benchmark cannot distinguish a model that solves problems from one that retrieves memorized solutions. The entire evaluation becomes uninformative. Large-scale web crawls that feed pre-training pipelines ingest public repositories, issue trackers, and forums where benchmark tasks originate. The model's parameters then encode those examples as retrievable patterns. Apply contamination analysis whenever you evaluate on a public benchmark; switch to private-repo or post-cutoff evaluations (described in Section 7) when you cannot rule out contamination.

Contamination operates at three levels, each progressively harder to detect:

• Direct memorization: the model has seen the exact issue-and-fix pair in its training data. Given the issue text, it can recall the gold patch (the reference correct patch, as defined in Section 23.1) verbatim. This is the most severe form and the easiest to test for.
• Indirect exposure: the model has not seen the specific issue, but it has seen the same bug pattern discussed in blog posts, Stack Overflow answers, or related issues in the same repository. It can generalize from these examples to the benchmark task.
• Distributional familiarity: the model has seen thousands of similar codebases, similar bug patterns, and similar fix patterns. It has not memorized anything specific, but its prior knowledge gives it an advantage that would not exist on truly novel code. This level is arguably not contamination at all; it is the model doing what it was trained to do.

The boundary between "contamination" and "useful pre-training" is blurry. A human developer who has fixed hundreds of Django cache bugs brings that experience to every new issue; no one calls that contamination. But a model that has memorized the specific diff for django__django-16379 is demonstrating recall, not problem-solving. The practical test: can the model solve new tasks from the same distribution? In short: a benchmark that cannot tell recall apart from reasoning measures nothing worth optimizing for.

"""
Contamination detection: check whether a model can reproduce
parts of the gold patch when prompted with just the issue text.
Uses n-gram overlap as a proxy for memorization.
"""
from collections import Counter


def ngram_overlap(
    text_a: str,
    text_b: str,
    n: int = 5
) -> float:
    """Compute n-gram overlap ratio between two texts.

    Returns the fraction of n-grams in text_a that also appear
    in text_b. High overlap suggests memorization when text_a
    is a model's output and text_b is the gold patch.

    Args:
        text_a: The model's generated patch.
        text_b: The gold (reference) patch.
        n: N-gram size. Larger n requires more exact reproduction.

    Returns:
        Overlap ratio in [0, 1]. Values above 0.5 for n=5
        strongly suggest memorization.
    """
    tokens_a = text_a.split()
    tokens_b = text_b.split()

    if len(tokens_a) < n or len(tokens_b) < n:
        return 0.0

    ngrams_a = Counter(
        tuple(tokens_a[i:i+n]) for i in range(len(tokens_a) - n + 1)
    )
    ngrams_b = set(
        tuple(tokens_b[i:i+n]) for i in range(len(tokens_b) - n + 1)
    )

    overlap = sum(
        count for ng, count in ngrams_a.items() if ng in ngrams_b
    )
    return overlap / sum(ngrams_a.values())


def detect_contamination(
    model_output: str,
    gold_patch: str,
    threshold: float = 0.4,
    n: int = 5
) -> dict:
    """Flag potential contamination in a model's output.

    Args:
        model_output: The patch the model generated.
        gold_patch: The reference gold patch.
        threshold: Overlap ratio above which contamination is suspected.
        n: N-gram size for overlap computation.

    Returns:
        Dictionary with overlap score and contamination flag.
    """
    overlap = ngram_overlap(model_output, gold_patch, n=n)
    return {
        "ngram_overlap": round(overlap, 4),
        "contaminated": overlap > threshold,
        "ngram_size": n,
        "threshold": threshold,
        "model_patch_tokens": len(model_output.split()),
        "gold_patch_tokens": len(gold_patch.split()),
    }


# Example usage
model_patch = """
--- a/django/core/cache/backends/filebased.py
+++ b/django/core/cache/backends/filebased.py
@@ -89,6 +89,8 @@ class FileBasedCache(BaseCache):
     def has_key(self, key, version=None):
         fname = self._key_to_file(key, version)
-        return os.path.exists(fname)
+        if os.path.exists(fname):
+            with open(fname, "rb") as f:
+                return not self._is_expired(f)
+        return False
"""

gold = """
--- a/django/core/cache/backends/filebased.py
+++ b/django/core/cache/backends/filebased.py
@@ -89,6 +89,8 @@ class FileBasedCache(BaseCache):
     def has_key(self, key, version=None):
         fname = self._key_to_file(key, version)
-        return os.path.exists(fname)
+        if os.path.exists(fname):
+            with open(fname, "rb") as f:
+                return not self._is_expired(f)
+        return False
"""

result = detect_contamination(model_patch, gold)
print(f"Overlap: {result['ngram_overlap']:.1%}, "
      f"Contaminated: {result['contaminated']}")
# Overlap: 100.0%, Contaminated: True

2. Test Adequacy and the False-Pass Problem

Even without contamination, an agent can achieve a "pass" on a benchmark task by producing a patch that is incorrect but happens to satisfy the test oracle, where the test oracle is the automated verdict (pass or fail) produced by running the task's test suite against a candidate patch. This is the false-pass problem, and it arises from weak test patches that do not fully specify the correct behavior.

Consider a task where the gold patch fixes an off-by-one error in a loop boundary. The test patch adds one test case that checks the output for a specific input. An agent that "fixes" the issue by hardcoding the expected output for that specific input will pass the test oracle while leaving the underlying bug unfixed. The test patch is technically correct (it fails on the base commit and passes on the gold patch) but it is not adequate: it does not distinguish the gold patch from incorrect alternatives.

The false-pass rate varies significantly across benchmark subsets. Mundler et al. (2024) found that adding additional test cases to SWE-bench tasks revealed that roughly 10-15% of "resolved" instances in some agent submissions were false passes: the agent's patch satisfied the original tests but failed on the expanded test suite. SWE-bench Verified reduces this rate through human curation, but it cannot eliminate it entirely.

"""
Test adequacy analysis: measure how well a test patch
discriminates correct patches from plausible-but-wrong alternatives.
Uses mutation testing on the gold patch to estimate adequacy.
"""
import subprocess
import random
from dataclasses import dataclass


@dataclass
class AdequacyResult:
    """Result of test adequacy analysis for a benchmark task."""
    instance_id: str
    total_mutants: int
    killed_mutants: int
    survived_mutants: int

    @property
    def mutation_score(self) -> float:
        """Fraction of mutants killed. Higher is better."""
        if self.total_mutants == 0:
            return 0.0
        return self.killed_mutants / self.total_mutants

    @property
    def is_adequate(self) -> bool:
        """A test patch is adequate if mutation score exceeds 0.8."""
        return self.mutation_score >= 0.8


def generate_patch_mutants(
    gold_patch: str,
    num_mutants: int = 20,
    seed: int = 42
) -> list[str]:
    """Generate mutated versions of a gold patch.

    Applies simple syntactic mutations to the gold patch to create
    plausible-but-wrong alternatives. If the test oracle cannot
    distinguish these from the gold patch, the tests are inadequate.

    Args:
        gold_patch: The correct (gold) patch as a unified diff.
        num_mutants: Number of mutants to generate.
        seed: Random seed for reproducibility.

    Returns:
        List of mutated patch strings.
    """
    rng = random.Random(seed)
    mutants = []

    lines = gold_patch.split("\n")
    added_lines = [
        (i, line) for i, line in enumerate(lines)
        if line.startswith("+") and not line.startswith("+++")
    ]

    mutations = [
        # Boundary mutations
        lambda l: l.replace(" < ", " <= "),
        lambda l: l.replace(" > ", " >= "),
        lambda l: l.replace(" <= ", " < "),
        lambda l: l.replace(" >= ", " > "),
        lambda l: l.replace(" == ", " != "),
        # Arithmetic mutations
        lambda l: l.replace(" + 1", " + 0"),
        lambda l: l.replace(" - 1", " - 0"),
        # Logic mutations
        lambda l: l.replace(" and ", " or "),
        lambda l: l.replace(" or ", " and "),
        lambda l: l.replace(" not ", " "),
        # Return value mutations
        lambda l: l.replace("return True", "return False"),
        lambda l: l.replace("return False", "return True"),
        lambda l: l.replace("return None", "return 0"),
    ]

    for _ in range(num_mutants):
        if not added_lines:
            break
        target_idx, target_line = rng.choice(added_lines)
        mutation_fn = rng.choice(mutations)
        mutated_line = mutation_fn(target_line)
        if mutated_line != target_line:  # Mutation actually changed something
            mutant_lines = lines.copy()
            mutant_lines[target_idx] = mutated_line
            mutants.append("\n".join(mutant_lines))

    return mutants


def assess_test_adequacy(
    task: "TaskInstance",
    repo_path: str,
    num_mutants: int = 20
) -> AdequacyResult:
    """Assess whether a task's test patch is adequate.

    Generates mutated versions of the gold patch and checks
    whether the test oracle rejects them. A high mutation score
    means the tests distinguish correct from incorrect patches.

    Args:
        task: The benchmark task to assess.
        repo_path: Path to a clean clone of the repository.
        num_mutants: Number of mutants to test.

    Returns:
        AdequacyResult with mutation score and kill counts.
    """
    mutants = generate_patch_mutants(task.gold_patch, num_mutants)
    killed = 0
    survived = 0

    for mutant_patch in mutants:
        # evaluate_patch is from Section 23.1
        result = evaluate_patch(task, mutant_patch, repo_path)
        if result.passed:
            survived += 1  # Test oracle failed to catch the mutant
        else:
            killed += 1    # Test oracle correctly rejected the mutant

    return AdequacyResult(
        instance_id=task.instance_id,
        total_mutants=len(mutants),
        killed_mutants=killed,
        survived_mutants=survived
    )

The mutation-based approach above connects test adequacy to the retry problem that follows: if a task's tests are too weak to catch mutant patches, they are also too weak to distinguish a lucky retry from a genuinely correct fix. Strengthening the test oracle addresses both threats simultaneously.

3. Metric Gaming and Retry Strategies

Weak tests are not the only way an inflated score slips through; even with adequate test oracles, the evaluation protocol itself can be exploited.

Beyond contamination and test adequacy, agents can inflate their benchmark scores through retry strategies that exploit the evaluation protocol. The simplest strategy is majority voting (running the agent multiple times and selecting the patch that passes the most internal checks): run the agent \(k\) times on each task, collect all \(k\) patches, and submit the one that passes the most internal checks. With \(k = 10\), an agent that has a 20% per-attempt success rate achieves a roughly 89% chance of producing at least one correct patch:

At \(p = 0.2\) and \(k = 10\): \(P = 1 - 0.8^{10} \approx 0.893\). This is a legitimate engineering strategy (redundancy improves reliability), but it obscures the agent's per-attempt capability. Reporting "89% resolve rate" without disclosing "\(k = 10\) with majority voting" is misleading. The honest report includes the per-attempt rate, the number of attempts, the selection strategy, and the total cost.

More sophisticated gaming strategies include:

• Test-guided patching: the agent applies the test patch, runs the tests to see the failure, and uses the failure message to guide patch generation. This is arguably a valid strategy (real developers run tests while debugging), but it means the agent is not solving the task from the issue description alone.
• Patch ensembling: generate multiple patches and combine their changes. This works when different attempts fix different parts of a multi-part bug, but it can also introduce new bugs through conflicting changes.
• Targeted submission: only submit patches on tasks where the agent's internal confidence is high, abstaining on the rest. This inflates the resolve rate on attempted tasks while hiding failures through non-submission.

4. The Benchmark-to-Production Gap

Even a perfectly valid benchmark with no contamination, adequate tests, and honest reporting still has a fundamental limitation: it measures performance on a specific distribution of tasks. Production deployment involves a different distribution, and the gap between these distributions can be large.

Several factors drive this gap:

Repository structure. SWE-bench draws from well-organized Python libraries with conventional layouts, comprehensive docstrings, and mature test infrastructure. Many production codebases are less organized: monorepos with complex build systems, sparse documentation, unconventional patterns, and test suites that take minutes to run.

Task distribution. SWE-bench tasks are derived from issues that were actually resolved by human developers and merged with test additions. In production, many bugs are never filed as issues, many issues are under-specified, and many fixes are merged without test additions. The tasks an agent encounters in production tend to be harder on average than those in the benchmark because benchmark tasks are pre-filtered for resolvability and test coverage.

Context availability. In SWE-bench, the agent receives the complete issue description. In production, bugs arrive as Slack messages ("the dashboard is broken"), Sentry alerts (a stack trace with no context), or user complaints ("it was working yesterday"). The context engineering skills from Chapter 11 become critical for closing this gap.

Interaction model. SWE-bench is a one-shot evaluation: the agent produces a patch and is scored. In production, agents typically operate in a loop with human review: they propose a patch, receive feedback, and iterate. An agent's ability to incorporate feedback (which SWE-bench does not measure) may matter more than its first-attempt accuracy.

"""
Gap analysis between benchmark and production task distributions.
Compares structural properties of benchmark tasks against
a sample of production issues from the same repository.
"""
from dataclasses import dataclass


@dataclass
class TaskProfile:
    """Structural properties of a coding task."""
    description_words: int
    has_stack_trace: bool
    has_repro_script: bool
    files_changed: int
    lines_changed: int
    cross_module: bool          # Changes span multiple packages
    requires_domain_knowledge: bool
    has_test_additions: bool


def compute_distribution_gap(
    benchmark_tasks: list[TaskProfile],
    production_tasks: list[TaskProfile]
) -> dict:
    """Quantify the structural gap between benchmark and production tasks.

    Args:
        benchmark_tasks: Profiles of tasks in the benchmark suite.
        production_tasks: Profiles of recent production issues.

    Returns:
        Dictionary with per-feature gap statistics showing where
        benchmark and production distributions diverge.
    """
    def feature_stats(tasks: list[TaskProfile], attr: str) -> dict:
        values = [getattr(t, attr) for t in tasks]
        if isinstance(values[0], bool):
            return {"fraction_true": sum(values) / len(values)}
        return {
            "mean": sum(values) / len(values),
            "min": min(values),
            "max": max(values)
        }

    features = [
        "description_words", "has_stack_trace", "has_repro_script",
        "files_changed", "lines_changed", "cross_module",
        "requires_domain_knowledge", "has_test_additions"
    ]

    gaps = {}
    for feat in features:
        bench_stats = feature_stats(benchmark_tasks, feat)
        prod_stats = feature_stats(production_tasks, feat)

        # Compute the key gap metric per feature type
        if "fraction_true" in bench_stats:
            gap_value = abs(
                bench_stats["fraction_true"] - prod_stats["fraction_true"]
            )
        else:
            bench_mean = bench_stats["mean"]
            prod_mean = prod_stats["mean"]
            gap_value = abs(bench_mean - prod_mean) / max(bench_mean, 1)

        gaps[feat] = {
            "benchmark": bench_stats,
            "production": prod_stats,
            "relative_gap": round(gap_value, 3)
        }

    return gaps


# Example: typical gaps observed in practice
# Benchmark tasks tend to have longer descriptions, more stack traces,
# fewer files changed, and always include test additions.
# Production tasks are noisier, broader, and often lack tests.

5. Cost, Latency, and Reliability as First-Class Metrics

The benchmark-to-production gap shows that resolve rate alone cannot capture deployment readiness; equally important are the operational costs of achieving that rate.

Resolve rate answers "can the agent fix bugs?" but practitioners also need "at what cost?", "how fast?", and "how consistently?". These operational metrics determine whether an agent is deployable, regardless of its resolve rate.

5.1 The Cost-Accuracy Frontier

Different agent architectures occupy different points on the cost-accuracy frontier (an instance of the Pareto frontier, where the Pareto frontier is the set of solutions for which no alternative is simultaneously better on every objective, from multi-objective optimization). A simple single-pass agent (one prompt, one response) is cheap but has limited accuracy. A multi-step agent with tool use (file browsing, test execution, iterative repair) is more accurate but consumes more tokens. A multi-agent system with planning, coding, and reviewing agents is even more capable but multiplies the cost.

The cost-accuracy frontier can be mapped empirically by evaluating the same agent architecture with different resource budgets:

An agent with 40% resolve rate at \$0.50/task has a cost efficiency of 0.8 resolved tasks per dollar. An agent with 55% at \$3.00/task has an efficiency of 0.18. The first agent resolves more bugs per dollar, even though its headline resolve rate is lower. For batch processing (running the agent overnight on a backlog of issues), cost efficiency may matter more than peak accuracy.

5.2 Latency Profiles

Latency is not a single number but a distribution. An agent might resolve 60% of tasks in under 2 minutes and the remaining 40% in 5 to 15 minutes, with occasional outliers exceeding 30 minutes. The latency distribution matters because different deployment contexts have different tolerance:

• Continuous Integration/Continuous Deployment (CI/CD) gate: maximum tolerance of 5 minutes. The agent must produce a patch quickly or yield to human review.
• Pull request (PR) assistant: tolerance of 15-30 minutes. The agent works asynchronously while the developer moves to other tasks.
• Batch overnight repair: tolerance of hours. Peak accuracy matters more than speed.

"""
Cost-accuracy frontier analysis for comparing agent architectures.
Plots the trade-off between resolve rate and cost per task.
"""
from dataclasses import dataclass
import statistics


@dataclass
class AgentProfile:
    """Performance profile of a coding agent architecture."""
    name: str
    resolve_rate: float
    cost_per_task_usd: float
    median_latency_seconds: float
    p95_latency_seconds: float
    consistency: float  # Fraction of tasks with stable outcomes

    @property
    def cost_efficiency(self) -> float:
        """Resolved tasks per dollar spent."""
        if self.cost_per_task_usd == 0:
            return float("inf")
        return self.resolve_rate / self.cost_per_task_usd

    @property
    def latency_ratio(self) -> float:
        """p95/median ratio. Values >> 2 indicate a heavy tail."""
        if self.median_latency_seconds == 0:
            return float("inf")
        return self.p95_latency_seconds / self.median_latency_seconds


def pareto_frontier(agents: list[AgentProfile]) -> list[AgentProfile]:
    """Find agents on the Pareto frontier of resolve rate vs. cost.

    An agent is Pareto-optimal if no other agent achieves both
    higher resolve rate AND lower cost per task.

    Args:
        agents: List of agent profiles to compare.

    Returns:
        Agents on the Pareto frontier, sorted by cost.
    """
    sorted_agents = sorted(agents, key=lambda a: a.cost_per_task_usd)
    frontier = []
    max_resolve = -1

    for agent in sorted_agents:
        if agent.resolve_rate > max_resolve:
            frontier.append(agent)
            max_resolve = agent.resolve_rate

    return frontier


# Example: comparing four architectures
agents = [
    AgentProfile("Single-pass", 0.22, 0.08, 15, 30, 0.85),
    AgentProfile("ReAct loop", 0.38, 0.45, 90, 300, 0.72),
    AgentProfile("Multi-step + tools", 0.48, 1.20, 180, 600, 0.68),
    AgentProfile("Multi-agent team", 0.55, 3.50, 300, 900, 0.60),
]

frontier = pareto_frontier(agents)
print("Pareto-optimal agents:")
for a in frontier:
    print(f"  {a.name}: {a.resolve_rate:.0%} resolve, "
          f"${a.cost_per_task_usd:.2f}/task, "
          f"efficiency={a.cost_efficiency:.2f} resolved/$")

6. A Validity Checklist for Benchmark Reports

When reading a benchmark report (or writing one), use this checklist to assess the validity of the claimed results. Each item maps to a specific threat discussed in this section.

7. Designing Contamination-Resistant Evaluations

Given the difficulty of eliminating contamination from public benchmarks, the most reliable evaluation strategy is to construct tasks the model cannot have seen. Three approaches achieve this:

Private repositories. Tasks drawn from your organization's proprietary codebase are typically absent from public training data. This is the approach we take in Section 23.3, where we build a five-issue benchmark from your own repository.

Post-cutoff tasks. Tasks from issues created after the model's training data cutoff cannot be memorized. The SWE-bench harness supports filtering by date, allowing temporal decontamination. The downside is a shrinking pool of valid tasks as the cutoff approaches the present.

Synthetic tasks. Programmatically generated tasks (inject a known bug into a codebase, generate the issue description, create the test oracle) eliminate contamination by construction. The risk is that synthetic tasks may not represent the difficulty distribution of real bugs. Combining synthetic bugs with real repository structure (inject artificial bugs into a real codebase) balances contamination resistance with ecological validity, where ecological validity is the degree to which evaluation conditions match the real-world conditions the system will face in deployment.

In practice, the best evaluation combines all three: public benchmarks for comparability with the broader community, private repo benchmarks for deployment-specific assessment, and post-cutoff or synthetic tasks for contamination-free measurement. The next section shows how to build the private-repo component.