Overview
Building an AI coding agent is only half the problem. The other half is knowing whether it actually works. Leaderboard positions shift weekly, vendor claims multiply, and yet the practitioner's core question remains stubbornly simple: if I deploy this agent on my codebase, will it reliably fix my bugs at a cost and latency I can justify? This chapter provides the measurement framework to answer that question with statistical rigor rather than anecdote.
We begin with SWE-bench and its family of task suite benchmarks, dissecting how real GitHub issues become reproducible evaluation instances and what design choices determine whether a benchmark measures genuine problem-solving or pattern matching. We then confront the validity threats that plague every public benchmark: data contamination, test leakage, metric gaming, and the gap between benchmark performance and production utility. Finally, we build a complete internal benchmark from scratch, constructing a five-issue evaluation suite from your own repository, running two competing agent workflows against it, and reporting the results with bootstrap confidence intervals and Wilcoxon signed-rank tests.
The evaluation methodology in this chapter connects to several threads running through the book. The search framework of Chapter 1 provides the conceptual grounding: evaluating an agent means measuring how efficiently it searches the space of possible code edits to find correct patches. The testing infrastructure from Chapter 18: AI-Assisted Testing and QA supplies the oracle: passing test suites tell us whether a patch is correct. The multi-agent architectures of Chapter 17 are the systems we evaluate. And the MLOps/AgentOps monitoring from Chapter 22 provides the production telemetry that complements benchmark scores with real-world reliability data.
Prerequisites
Readers should have completed Chapter 18: AI-Assisted Testing and QA (which introduced pytest, property-based testing, and mutation analysis) and Chapter 22: MLOps, LLMOps, and AgentOps (which covered operational metrics for deployed AI systems). Familiarity with Chapter 17: Multi-Agent Software Teams is helpful for understanding the agent architectures being evaluated. Basic statistics (means, confidence intervals, hypothesis tests) and comfort with Python's subprocess and git libraries are assumed.
Learning Outcomes
- Explain how SWE-bench constructs evaluation tasks from GitHub issues and pull requests, and identify the design decisions that affect benchmark validity.
- Detect and mitigate benchmark contamination, test leakage, and metric gaming in agent evaluations.
- Measure agent performance across three dimensions: correctness (pass rate), efficiency (cost and latency), and reliability (variance and failure modes).
- Compute bootstrap confidence intervals for pass rates and apply Wilcoxon signed-rank tests to compare paired agent evaluations.
- Build a five-issue internal benchmark from your own repository, evaluate two agent workflows, and produce a statistically grounded comparison report.
- Integrate agent evaluation into CI/CD pipelines using GitHub Actions and pytest.
Sections
23.1 SWE-bench and Task Suites
How SWE-bench constructs evaluation instances from GitHub issues. The anatomy of a task: repository snapshot, issue description, test patch, gold patch. Repo-level task suites. Human review calibration. Tools: SWE-bench harness, Docker, pytest. Recipe: running a single SWE-bench task end to end.
23.2 Validity and Contamination
Benchmark leakage and data contamination. Training-set overlap detection. Test adequacy and the false-pass problem. Gaming metrics with retry loops. The gap between benchmark scores and production utility. Cost, latency, and reliability as first-class metrics.
23.3 Building an Agent Benchmark
Recipe: constructing a five-issue internal benchmark from your own repository. Task selection criteria. Harness design with Docker isolation. Evaluating two agent workflows. Bootstrap confidence intervals. Wilcoxon signed-rank tests. Reporting with statistical rigor. Discovery Workbench integration.
Bibliography
Foundational Papers
The original SWE-bench paper, establishing the methodology for constructing evaluation tasks from GitHub issues and pull requests. Defines the task format, evaluation harness, and baseline results that anchor this chapter.
Introduces human-verified filtering of SWE-bench tasks to remove ambiguous or under-specified instances, addressing concerns about test adequacy raised in Section 23.2.
Introduces the Agent-Computer Interface (ACI) concept and demonstrates how tool design affects agent performance, motivating the interface-aware evaluation approach in Section 23.1.
Benchmarks and Datasets
Extends SWE-bench with additional test cases to reduce false positives, directly relevant to the test adequacy discussion in Section 23.2.
Combines code search with LLM reasoning for automated bug fixing, providing a concrete agent architecture to evaluate with the framework in Section 23.3.
Statistical Methods
The foundational paper on bootstrap resampling for confidence interval estimation, the core statistical technique used in Section 23.3.
Introduces the signed-rank test for paired comparisons without normality assumptions, used in Section 23.3 for comparing agent workflows.
Tools and Libraries
The official SWE-bench repository containing the evaluation harness, Docker infrastructure, and task instances used throughout this chapter.
The standard Python testing framework used as the test oracle in agent evaluation harnesses.
CI/CD platform used to automate agent evaluation pipelines in Section 23.3.
Research Frontiers
Comprehensive survey of code evaluation methodologies, covering benchmarks from HumanEval through SWE-bench and identifying open challenges in evaluation design.
Analysis of scaling behavior in agent evaluation, showing how pass rates change with task difficulty and repository complexity.