23.1 SWE-bench and Task Suites

1. The Anatomy of a Benchmark Task

Without a rigorous way to measure whether a coding agent actually fixes bugs, teams ship tools that look impressive in demos but silently produce broken patches in production. The discipline of benchmark construction turns "does the agent work?" from a subjective impression into a repeatable experiment.

Imagine handing an AI the same GitHub issue a human developer resolved last Tuesday, freezing the repository at the exact commit before the fix, and asking: does the AI's patch make the tests pass? A coding agent benchmark task has four components that make this experiment precise and repeatable, each serving a distinct role in the evaluation. Figure 23.1 illustrates how these four components interact during a single evaluation run.

A benchmark task packages everything needed to test whether a coding agent can fix a real software bug. Without a standardized task format, comparing agents is like comparing test-takers who each took a different exam; the results are meaningless. The task freezes a repository at the moment a bug existed, hands the agent only the bug report, and checks the proposed fix against test cases from the actual human-written solution. Use benchmark tasks for objective, reproducible comparisons between agents or configurations. Use informal manual testing for quick sanity checks on a single prompt.

The Four Components

The repository snapshot is a frozen checkout of the project at a specific commit, typically the commit immediately before the fix was merged. This snapshot defines the "world" the agent operates in: the codebase it can read, the dependencies it can install, and the test suite it can run. Freezing at a specific commit ensures reproducibility; every agent sees exactly the same starting state.

The issue description is the natural-language prompt the agent receives. In SWE-bench, this is the actual GitHub issue body, including any stack traces, error messages, or reproduction steps the original reporter provided. The quality and specificity of this description varies enormously across tasks, which is both a feature (it mirrors real-world variation) and a challenge (it makes task difficulty hard to calibrate).

The test patch contains the test additions from the pull request (PR) that fixed the issue. These tests fail on the base commit (confirming the bug exists) and pass after the fix is applied (confirming the fix is correct). The test patch is the oracle, where a test oracle is an automated criterion that determines whether a given output is correct: an agent's proposed patch is judged correct if and only if it causes the test patch's tests to pass. This is a critical design choice, because it means the evaluation is only as good as the test patch. We will examine the consequences of this choice in Section 23.2. In short: A benchmark is only as honest as its test oracle; everything else is theatre.

The gold patch is the actual code change from the merged pull request. It is not shown to the agent and is not used for grading. Its role is diagnostic: when an agent fails a task, comparing its attempted patch against the gold patch reveals whether the agent was on the right track or completely lost. The gold patch also helps benchmark maintainers assess task difficulty.

2. How SWE-bench Constructs Tasks

The four components define what a task contains. The next question is sourcing: where do all those snapshots, issue descriptions, test patches, and gold patches come from?

The SWE-bench construction pipeline mines GitHub repositories for pull requests that meet specific criteria. Understanding this pipeline reveals which kinds of tasks end up in the benchmark and, more importantly, which kinds are excluded. The pipeline has four stages: repository selection, pull request filtering, task validation, and difficulty stratification. Figure 23.1.1 illustrates the SWE-bench task construction pipeline.

2.1 Repository Selection

SWE-bench draws from well-maintained open-source Python projects with comprehensive test suites: libraries like Django, Flask, scikit-learn, sympy, matplotlib, and requests. These projects share several properties that make them suitable for benchmarking. They have mature Continuous Integration/Continuous Deployment (CI/CD) pipelines, so test infrastructure exists. They follow conventional project layouts, so agents can navigate the codebase. They have extensive issue trackers with detailed bug reports. And they are popular enough that many issues have been resolved by experienced maintainers, producing high-quality gold patches.

This selection introduces a systematic bias: SWE-bench measures agent performance on well-maintained Python libraries with conventional structures. Performance on a disorganized monorepo, a polyglot microservice architecture, or a domain-specific scientific codebase may differ substantially. We return to this gap in Section 23.2.

2.2 Pull Request Filtering

From each repository, the pipeline selects merged pull requests that satisfy three conditions: (1) the PR resolves at least one GitHub issue, (2) the PR includes at least one new test, and (3) the new tests fail on the base commit and pass on the merge commit. The third condition is crucial: it ensures the tests actually verify the fix rather than testing unrelated functionality.

"""
Simplified SWE-bench-style task construction from a Git repository.
Demonstrates the core logic of extracting evaluation tasks from
merged pull requests that include test additions.
"""
import subprocess
import json
from dataclasses import dataclass, field, asdict
from pathlib import Path


@dataclass
class TaskInstance:
    """A single evaluation task extracted from a pull request."""
    instance_id: str
    repo: str
    base_commit: str          # Commit before the fix
    issue_text: str           # GitHub issue body (the agent's prompt)
    test_patch: str           # Diff of added/modified test files
    gold_patch: str           # Diff of the actual fix (not shown to agent)
    test_cmd: str             # Command to run the relevant tests
    created_at: str           # ISO timestamp of the PR merge
    hints: list[str] = field(default_factory=list)  # Optional hints


def extract_task_from_pr(
    repo_path: str,
    merge_commit: str,
    issue_text: str,
    test_files: list[str],
    instance_id: str
) -> TaskInstance | None:
    """Extract a benchmark task from a merged pull request.

    Args:
        repo_path: Path to the local clone of the repository.
        merge_commit: The merge commit SHA of the pull request.
        issue_text: The body of the linked GitHub issue.
        test_files: Paths to test files added or modified in the PR.
        instance_id: Unique identifier for this task.

    Returns:
        A TaskInstance if the PR qualifies, None otherwise.
    """
    # Step 1: Find the base commit (parent of the merge)
    base_commit = subprocess.check_output(
        ["git", "rev-parse", f"{merge_commit}^"],
        cwd=repo_path, text=True
    ).strip()

    # Step 2: Extract the test patch (changes to test files only)
    test_patch = subprocess.check_output(
        ["git", "diff", base_commit, merge_commit, "--"] + test_files,
        cwd=repo_path, text=True
    )
    if not test_patch.strip():
        return None  # No test changes, skip this PR

    # Step 3: Extract the gold patch (changes to non-test files)
    all_files = subprocess.check_output(
        ["git", "diff", "--name-only", base_commit, merge_commit],
        cwd=repo_path, text=True
    ).strip().split("\n")
    src_files = [f for f in all_files if f not in test_files]
    gold_patch = subprocess.check_output(
        ["git", "diff", base_commit, merge_commit, "--"] + src_files,
        cwd=repo_path, text=True
    )

    # Step 4: Validate that tests fail on base and pass on merge
    # (In production, this runs inside Docker for isolation)
    if not _validate_test_behavior(
        repo_path, base_commit, merge_commit, test_files
    ):
        return None

    return TaskInstance(
        instance_id=instance_id,
        repo=Path(repo_path).name,
        base_commit=base_commit,
        issue_text=issue_text,
        test_patch=test_patch,
        gold_patch=gold_patch,
        test_cmd=f"pytest {' '.join(test_files)} -x",
        created_at=subprocess.check_output(
            ["git", "log", "-1", "--format=%aI", merge_commit],
            cwd=repo_path, text=True
        ).strip()
    )


def _validate_test_behavior(
    repo_path: str,
    base_commit: str,
    merge_commit: str,
    test_files: list[str]
) -> bool:
    """Verify tests fail on base commit and pass on merge commit."""
    test_cmd = ["pytest"] + test_files + ["-x", "--tb=no", "-q"]

    # Tests should FAIL on the base commit
    subprocess.run(["git", "checkout", base_commit], cwd=repo_path,
                   capture_output=True)
    base_result = subprocess.run(test_cmd, cwd=repo_path,
                                capture_output=True)

    # Tests should PASS on the merge commit
    subprocess.run(["git", "checkout", merge_commit], cwd=repo_path,
                   capture_output=True)
    merge_result = subprocess.run(test_cmd, cwd=repo_path,
                                 capture_output=True)

    return base_result.returncode != 0 and merge_result.returncode == 0

2.3 Task Validation and Difficulty

Not every filtered PR produces a valid benchmark task. The validation stage checks three conditions: the repository installs from the base commit, the test suite runs without infrastructure failures, and the test patch fails deterministically (not flakily, meaning the test does not alternate between passing and failing across runs due to timing, ordering, or resource sensitivity). The pipeline then stratifies validated tasks by difficulty using several proxies. These include the number of files the gold patch changes, lines added and removed, whether the fix spans multiple modules, and whether the issue description includes a reproduction script.

SWE-bench Lite (300 tasks) and SWE-bench Verified (500 tasks) are curated subsets with human review applied to task selection. Verified in particular uses human annotators to confirm that the issue description is sufficiently clear, the test patch is adequate, and the task is solvable from the information given. This human curation addresses many of the validity concerns we explore in Section 23.2, but it also means these subsets are small enough that statistical comparisons between agents require careful confidence interval estimation.

3. Running an Evaluation Harness

An evaluation harness (the infrastructure that executes benchmark tasks in isolated, reproducible environments and records results) must isolate each task in its own environment (so side effects from one task cannot affect another), enforce resource limits (so an agent cannot run indefinitely), and capture detailed telemetry (so you can diagnose failures). The SWE-bench harness uses Docker containers for isolation, with each task running in a fresh container built from the repository's dependency specification.

The following code implements a minimal evaluation harness. It does not match the full SWE-bench harness in robustness (the production harness handles conda environments, complex build systems, and timeout enforcement), but it captures the essential evaluation loop: set up the environment, apply the agent's patch, run the test oracle, and record the result.

"""
Minimal evaluation harness for coding agent benchmarks.
Runs a single task instance: applies the agent's patch,
executes the test oracle, and records pass/fail with timing.
"""
import subprocess
import tempfile
import shutil
import time
from dataclasses import dataclass
from pathlib import Path


@dataclass
class EvalResult:
    """Result of evaluating an agent's patch on a single task."""
    instance_id: str
    passed: bool
    duration_seconds: float
    test_output: str
    patch_applied: bool
    error: str | None = None


def evaluate_patch(
    task: "TaskInstance",
    agent_patch: str,
    repo_path: str,
    timeout: int = 300
) -> EvalResult:
    """Evaluate an agent's patch against a task's test oracle.

    Args:
        task: The benchmark task instance.
        agent_patch: The agent's proposed patch as a unified diff
            (a standard format showing added and removed lines
            with @@ range headers).
        repo_path: Path to a clean clone of the repository.
        timeout: Maximum seconds for test execution.

    Returns:
        An EvalResult with pass/fail status and diagnostics.
    """
    start = time.time()

    # Work in a temporary copy to preserve the original
    work_dir = tempfile.mkdtemp(prefix="eval_")
    try:
        shutil.copytree(repo_path, f"{work_dir}/repo", dirs_exist_ok=True)
        work_repo = f"{work_dir}/repo"

        # Step 1: Checkout the base commit
        subprocess.run(
            ["git", "checkout", task.base_commit],
            cwd=work_repo, capture_output=True, check=True
        )

        # Step 2: Apply the test patch (the oracle)
        test_apply = subprocess.run(
            ["git", "apply", "--allow-empty"],
            input=task.test_patch, cwd=work_repo,
            capture_output=True, text=True
        )
        if test_apply.returncode != 0:
            return EvalResult(
                instance_id=task.instance_id,
                passed=False,
                duration_seconds=time.time() - start,
                test_output="",
                patch_applied=False,
                error=f"Test patch failed to apply: {test_apply.stderr}"
            )

        # Step 3: Apply the agent's patch
        agent_apply = subprocess.run(
            ["git", "apply", "--allow-empty"],
            input=agent_patch, cwd=work_repo,
            capture_output=True, text=True
        )
        patch_applied = agent_apply.returncode == 0

        if not patch_applied:
            return EvalResult(
                instance_id=task.instance_id,
                passed=False,
                duration_seconds=time.time() - start,
                test_output="",
                patch_applied=False,
                error=f"Agent patch failed to apply: {agent_apply.stderr}"
            )

        # Step 4: Run the test oracle
        test_result = subprocess.run(
            task.test_cmd.split(),
            cwd=work_repo, capture_output=True, text=True,
            timeout=timeout
        )

        return EvalResult(
            instance_id=task.instance_id,
            passed=test_result.returncode == 0,
            duration_seconds=time.time() - start,
            test_output=test_result.stdout + test_result.stderr,
            patch_applied=True
        )

    except subprocess.TimeoutExpired:
        return EvalResult(
            instance_id=task.instance_id,
            passed=False,
            duration_seconds=timeout,
            test_output="",
            patch_applied=True,
            error="Test execution timed out"
        )
    finally:
        shutil.rmtree(work_dir, ignore_errors=True)

4. Repo-Level Task Suites

SWE-bench draws tasks from multiple repositories, which tests an agent's ability to generalize across codebases. But a complementary evaluation strategy is the repo-level task suite: a collection of tasks drawn from a single repository, designed to measure an agent's performance on a specific codebase. This approach is particularly valuable for teams evaluating whether to adopt a coding agent for their own project, because it directly measures the metric that matters: "how well does this agent handle our code?"

A good repo-level suite samples tasks across several dimensions:

• Bug type: logic errors, edge cases, type mismatches, concurrency issues, performance regressions.
• Scope: single-function fixes, cross-module changes, API modifications with downstream updates.
• Description quality: tasks with full reproduction scripts, tasks with only a stack trace, tasks with only a natural-language description.
• Difficulty: one-line fixes, multi-file refactors, changes requiring domain knowledge.

The sampling strategy matters because agents have uneven capability profiles. An agent might excel at single-file bug fixes with clear reproduction steps but fail completely at cross-module refactors described only in natural language. A suite that over-represents one task type produces misleading aggregate scores.

Research Frontier

Recent work extends the SWE-bench paradigm well beyond Python bug fixing. SWE-bench Multimodal (2024) adds tasks involving visual outputs where the test oracle includes image comparison. SWE-bench for JavaScript, Java, and Rust are under active development. CrossCodeBench (Zhang et al., 2025) constructs tasks that require changes across multiple programming languages in the same repository. A significant methodological advance is SWE-bench+ (Aleithan et al., 2025), which audits the original SWE-bench tasks and finds that many test patches are too weak: patches that are clearly incorrect (for example, hardcoding a return value) can still pass the oracle tests. SWE-bench+ strengthens the test suites with additional assertions, reducing false-positive resolve rates for several leading agents by 5 to 15 percentage points. This finding underscores that benchmark quality depends critically on the test oracle, not just on task selection. For scientific discovery applications, SciCode (Tian et al., 2024) evaluates agents on scientific computing tasks drawn from computational physics, chemistry, and biology codebases, where correct patches require domain knowledge beyond software engineering. The methodology from this section applies directly to constructing such domain-specific evaluation suites. As of mid-2025, several leading coding agents report resolve rates above 50% on SWE-bench Verified, up from single-digit percentages when the benchmark launched in 2023; this rapid score growth has intensified interest in harder successors and domain-specific suites.

5. Human Review Calibration

The task suites above rely entirely on automated test oracles, but a passing test suite cannot tell you whether the task itself was fair or whether the test patch was strong enough to catch a subtly wrong fix.

Automated evaluation (did the tests pass?) is necessary but not sufficient for understanding agent capabilities. Human review adds two dimensions that automated metrics miss: patch quality (is the fix elegant, maintainable, and correct beyond the test cases?) and task calibration (is the task actually solvable from the information given?).

SWE-bench Verified uses a structured human review protocol. For each task, a human reviewer answers three questions: (1) Is the issue description clear enough that a competent developer could solve the task without additional context? (2) Does the test patch adequately verify the fix, or could an incorrect patch pass the tests? (3) Is the gold patch the only reasonable fix, or are there multiple valid approaches? Tasks where the answer to (1) is "no" are excluded as ambiguous. Tasks where the answer to (2) is "could pass incorrectly" are flagged for test enhancement.

For internal benchmarks, a lighter-weight calibration works: have a developer who did not create the task solve it from the issue description alone, without seeing the gold patch. If the developer cannot solve it, the description needs improvement. If the developer finds an alternative fix that also passes, the test patch needs strengthening. This "red team" step, analogous to adversarial testing (Chapter 18), catches ambiguous tasks before they contaminate evaluation results.

"""
Human review calibration protocol for benchmark tasks.
Tracks reviewer assessments and flags tasks that need
revision before inclusion in the benchmark.
"""
from dataclasses import dataclass
from enum import Enum


class Clarity(Enum):
    CLEAR = "clear"              # Solvable from description alone
    AMBIGUOUS = "ambiguous"      # Requires assumptions not in description
    INSUFFICIENT = "insufficient"  # Missing critical information


class TestAdequacy(Enum):
    ADEQUATE = "adequate"        # Only correct patches pass
    WEAK = "weak"                # Incorrect patches could pass
    FLAKY = "flaky"              # Tests pass/fail non-deterministically


@dataclass
class TaskReview:
    """Human review of a benchmark task's quality."""
    instance_id: str
    reviewer: str
    clarity: Clarity
    test_adequacy: TestAdequacy
    alternative_fixes: int      # Number of valid fixes beyond the gold
    estimated_difficulty: int    # 1-5 scale (1=trivial, 5=expert)
    notes: str = ""

    @property
    def is_valid(self) -> bool:
        """A task is valid if it's clear and has adequate tests."""
        return (
            self.clarity == Clarity.CLEAR
            and self.test_adequacy == TestAdequacy.ADEQUATE
        )


def calibrate_suite(
    tasks: list["TaskInstance"],
    reviews: list[TaskReview]
) -> dict:
    """Compute calibration statistics for a task suite.

    Returns:
        Dictionary with validity rate, difficulty distribution,
        and flagged tasks needing revision.
    """
    review_map = {r.instance_id: r for r in reviews}
    valid = [t for t in tasks if review_map.get(t.instance_id,
             TaskReview(t.instance_id, "", Clarity.INSUFFICIENT,
                       TestAdequacy.WEAK, 0, 3)).is_valid]
    flagged = [t.instance_id for t in tasks
               if t.instance_id in review_map
               and not review_map[t.instance_id].is_valid]

    difficulties = [review_map[t.instance_id].estimated_difficulty
                    for t in tasks if t.instance_id in review_map]

    return {
        "total_tasks": len(tasks),
        "valid_tasks": len(valid),
        "validity_rate": len(valid) / len(tasks) if tasks else 0,
        "flagged_for_revision": flagged,
        "difficulty_distribution": {
            d: difficulties.count(d) for d in range(1, 6)
        },
        "mean_difficulty": sum(difficulties) / len(difficulties)
            if difficulties else 0
    }

6. Metrics Beyond Pass Rate

The headline metric for SWE-bench is the resolve rate, where resolve rate is the fraction of tasks where the agent's patch causes all oracle tests to pass. This single number dominates leaderboard discussions, but it obscures critical operational details. Two agents with identical 40% resolve rates may differ dramatically in cost, latency, reliability, and failure modes.

A complete evaluation captures at least five metrics:

• Resolve rate: fraction of tasks where all oracle tests pass. The headline metric.
• Patch application rate: fraction of tasks where the agent produces a syntactically valid patch that applies cleanly. Measures basic code generation competence.
• Cost per task: total API cost (input + output tokens) per task attempt. Varies by model, prompt strategy, and number of retries.
• Latency per task: wall-clock time from task start to patch submission. Includes all agent reasoning, tool use, and retry loops.
• Consistency: if you run the same agent on the same task five times, how often does it produce the same outcome? High variance indicates fragile strategies.

The cost and latency metrics connect directly to the operational monitoring in Chapter 22: MLOps, LLMOps, and AgentOps. In production, an agent that costs \$2 per task and takes 3 minutes is operationally very different from one that costs \$0.10 and takes 30 seconds, even if both achieve the same resolve rate. The right choice depends on your deployment context: a CI/CD gate needs low latency; a batch overnight repair system can tolerate higher latency for better accuracy.

"""
Multi-dimensional evaluation metrics for coding agent benchmarks.
Captures resolve rate, cost, latency, and consistency
across a suite of evaluation tasks.
"""
import statistics
from dataclasses import dataclass


@dataclass
class TaskMetrics:
    """Metrics for a single task evaluation."""
    instance_id: str
    resolved: bool
    patch_applied: bool
    cost_usd: float
    latency_seconds: float
    input_tokens: int
    output_tokens: int
    num_tool_calls: int
    num_retries: int


@dataclass
class SuiteMetrics:
    """Aggregate metrics across an entire evaluation suite."""
    agent_name: str
    total_tasks: int
    resolved: int
    patches_applied: int
    total_cost_usd: float
    total_latency_seconds: float

    @property
    def resolve_rate(self) -> float:
        return self.resolved / self.total_tasks if self.total_tasks else 0

    @property
    def apply_rate(self) -> float:
        return self.patches_applied / self.total_tasks if self.total_tasks else 0

    @property
    def cost_per_task(self) -> float:
        return self.total_cost_usd / self.total_tasks if self.total_tasks else 0

    @property
    def mean_latency(self) -> float:
        return self.total_latency_seconds / self.total_tasks if self.total_tasks else 0


def compute_suite_metrics(
    agent_name: str,
    task_metrics: list[TaskMetrics]
) -> SuiteMetrics:
    """Aggregate per-task metrics into suite-level statistics."""
    return SuiteMetrics(
        agent_name=agent_name,
        total_tasks=len(task_metrics),
        resolved=sum(1 for t in task_metrics if t.resolved),
        patches_applied=sum(1 for t in task_metrics if t.patch_applied),
        total_cost_usd=sum(t.cost_usd for t in task_metrics),
        total_latency_seconds=sum(t.latency_seconds for t in task_metrics),
    )


def compute_consistency(
    runs: list[list[TaskMetrics]]
) -> dict[str, float]:
    """Measure outcome consistency across repeated runs.

    Args:
        runs: Multiple runs of the same suite, each a list of TaskMetrics.
              All runs must cover the same tasks in the same order.

    Returns:
        Per-task consistency scores (fraction of runs with same outcome).
    """
    if not runs or not runs[0]:
        return {}

    consistency = {}
    for task_idx in range(len(runs[0])):
        instance_id = runs[0][task_idx].instance_id
        outcomes = [run[task_idx].resolved for run in runs]
        # Consistency = fraction of runs agreeing with the majority
        majority = sum(outcomes) > len(outcomes) / 2
        agreement = sum(1 for o in outcomes if o == majority)
        consistency[instance_id] = agreement / len(outcomes)

    return consistency

# Download task instances
python -m swebench.harness.prepare --dataset_name princeton-nlp/SWE-bench_Verified

# Run evaluation on agent predictions
python -m swebench.harness.run_evaluation \
    --predictions_path ./predictions.json \
    --swe_bench_tasks princeton-nlp/SWE-bench_Verified \
    --log_dir ./eval_logs \
    --testbed /tmp/testbed

7. From Public Benchmarks to Internal Evaluation

Public benchmarks like SWE-bench answer a general question: which agent architectures are most capable at resolving GitHub issues across well-known Python libraries? But the practitioner's question is more specific: which agent will work best on my codebase, with my coding conventions, my test infrastructure, and my deployment constraints?

The gap between these questions motivates building internal benchmarks. In Section 23.3, we construct a complete five-issue benchmark from your own repository. The task construction pipeline from this section provides the methodology: extract tasks from your own merged PRs, validate test behavior, calibrate with human review, and run evaluations with multi-dimensional metrics. The statistical tools in Section 23.3 (bootstrap confidence intervals and Wilcoxon signed-rank tests, statistical methods for estimating uncertainty and comparing paired outcomes) then let you make rigorous comparisons between agent workflows rather than relying on single-number resolve rates.

Before building that internal benchmark, however, we need to understand what can go wrong with evaluation. Section 23.2 examines the validity threats that plague both public and internal benchmarks: data contamination, test leakage, metric gaming, and the gap between benchmark scores and production utility.