19.3 Building a Debugging Pipeline

Imagine pushing a commit at 9 AM, watching two tests turn red, and returning from your coffee break to find that a four-stage robot has already pinpointed the faulty line, written a patch, verified it against the full suite, and opened a pull request. That robot is a debugging pipeline: an automated workflow that takes a failing test suite as input and produces a verified fix (or a structured diagnostic report) as output. No human intervenes between stages. Manual debugging is the single largest time sink in software maintenance, often consuming 30 to 50 percent of total development effort. Automating even the straightforward cases frees developers for problems that require creative reasoning. The pipeline chains four stages in sequence: (1) run tests to detect failures, (2) compute a fault localization score for every source line using coverage data, (3) feed the top-ranked lines to an LLM that generates candidate patches, and (4) re-run the test suite to verify each patch until one passes or the budget is exhausted. Use a pipeline when your project has a reliable test suite and reproducible failures. For flaky tests, race conditions, or bugs that require environment-specific reproduction, start with manual triage before feeding the problem to an automated system. Figure 1 below illustrates these four stages and the feedback loop that connects verification back to hypothesis generation.

1. The Target Application: A Scientific Data Processor

To demonstrate the full pipeline, we need a realistic application with enough complexity to produce interesting bugs. Our target is a small scientific data processing module: it loads experimental measurements, applies calibration corrections, detects outliers, and computes summary statistics. This is exactly the kind of code that appears in research pipelines (see Chapter 5: Discovery Through Data, Models, and Simulation), where subtle numerical bugs can corrupt results silently. In short: a debugging pipeline needs only two things: a test suite that distinguishes correct behavior from broken behavior, and a ranking function that converts coverage into suspicion.

"""
Target application: scientific data processor with a planted bug.
This module processes experimental measurements, applies calibrations,
detects outliers, and computes summary statistics.
"""
import math
import structlog
from dataclasses import dataclass, field
from opentelemetry import trace

log = structlog.get_logger()
tracer = trace.get_tracer("data_processor")


@dataclass
class Measurement:
    """A single experimental measurement with metadata."""
    value: float
    uncertainty: float
    instrument_id: str
    timestamp: str


@dataclass
class CalibrationProfile:
    """Calibration parameters for an instrument."""
    instrument_id: str
    offset: float
    scale: float
    # Nonlinearity correction: value_corrected = scale * (value - offset) + nl * value^2
    nonlinearity: float = 0.0


@dataclass
class ProcessingResult:
    """Summary statistics from processed measurements."""
    mean: float
    std_dev: float
    n_total: int
    n_outliers: int
    calibrated_values: list[float] = field(default_factory=list)


def apply_calibration(
    measurement: Measurement,
    profile: CalibrationProfile,
) -> float:
    """Apply instrument calibration to a raw measurement.

    The calibration model is:
        corrected = scale * (raw - offset) + nonlinearity * raw^2
    """
    with tracer.start_as_current_span(
        "apply_calibration",
        attributes={
            "instrument": measurement.instrument_id,
            "raw_value": measurement.value,
        },
    ) as span:
        raw = measurement.value
        corrected = (
            profile.scale * (raw - profile.offset)
            + profile.nonlinearity * raw ** 2
        )
        span.set_attribute("corrected_value", corrected)
        log.debug(
            "calibration.applied",
            instrument=measurement.instrument_id,
            raw=raw,
            corrected=corrected,
            offset=profile.offset,
            scale=profile.scale,
        )
        return corrected


def detect_outliers(
    values: list[float],
    threshold: float = 3.0,
) -> list[bool]:
    """Detect outliers using the modified Z-score method.

    Uses median absolute deviation (MAD) for robust outlier detection.
    A value is an outlier if |modified_z_score| > threshold.
    """
    with tracer.start_as_current_span(
        "detect_outliers",
        attributes={"n_values": len(values), "threshold": threshold},
    ) as span:
        if len(values) < 3:
            return [False] * len(values)

        median = sorted(values)[len(values) // 2]
        deviations = [abs(v - median) for v in values]
        mad = sorted(deviations)[len(deviations) // 2]

        # BUG: when MAD is zero (all values identical or nearly so),
        # we should return no outliers, but instead we divide by zero
        # and produce NaN, which causes downstream failures.
        modified_z = [0.6745 * (v - median) / mad for v in values]

        outliers = [abs(z) > threshold for z in modified_z]
        n_outliers = sum(outliers)

        span.set_attribute("n_outliers", n_outliers)
        span.set_attribute("mad", mad)
        log.info(
            "outliers.detected",
            n_values=len(values),
            n_outliers=n_outliers,
            mad=mad,
            median=median,
        )
        return outliers


def compute_statistics(
    values: list[float],
    outlier_mask: list[bool],
) -> ProcessingResult:
    """Compute summary statistics, excluding outliers."""
    with tracer.start_as_current_span("compute_statistics") as span:
        clean = [v for v, is_out in zip(values, outlier_mask) if not is_out]

        if not clean:
            log.warning("statistics.no_valid_data", reason="all values are outliers")
            return ProcessingResult(
                mean=0.0, std_dev=0.0,
                n_total=len(values), n_outliers=sum(outlier_mask),
                calibrated_values=[],
            )

        mean = sum(clean) / len(clean)
        variance = sum((v - mean) ** 2 for v in clean) / len(clean)
        std_dev = math.sqrt(variance)

        span.set_attribute("mean", mean)
        span.set_attribute("std_dev", std_dev)
        span.set_attribute("n_clean", len(clean))

        log.info(
            "statistics.computed",
            mean=mean,
            std_dev=std_dev,
            n_clean=len(clean),
            n_outliers=sum(outlier_mask),
        )

        return ProcessingResult(
            mean=mean,
            std_dev=std_dev,
            n_total=len(values),
            n_outliers=sum(outlier_mask),
            calibrated_values=clean,
        )


def process_experiment(
    measurements: list[Measurement],
    calibrations: dict[str, CalibrationProfile],
    outlier_threshold: float = 3.0,
) -> ProcessingResult:
    """Full processing pipeline: calibrate, detect outliers, compute stats."""
    with tracer.start_as_current_span(
        "process_experiment",
        attributes={"n_measurements": len(measurements)},
    ):
        # Step 1: Apply calibrations
        calibrated = []
        for m in measurements:
            profile = calibrations.get(m.instrument_id)
            if profile is None:
                log.warning(
                    "calibration.missing",
                    instrument=m.instrument_id,
                )
                calibrated.append(m.value)  # Use raw value
            else:
                calibrated.append(apply_calibration(m, profile))

        # Step 2: Detect outliers
        outlier_mask = detect_outliers(calibrated, outlier_threshold)

        # Step 3: Compute statistics
        result = compute_statistics(calibrated, outlier_mask)

        log.info(
            "experiment.processed",
            n_measurements=len(measurements),
            mean=result.mean,
            std_dev=result.std_dev,
            n_outliers=result.n_outliers,
        )
        return result

2. The Test Suite

A good test suite is essential for automated debugging: it provides the failing signal that triggers the pipeline and the passing signals that confirm the fix. Our test suite includes both passing cases (normal data) and the failing case that exposes the planted bug.

"""
Test suite for the scientific data processor.
Includes tests that pass (normal data) and one that fails
(identical values triggering the MAD=0 bug).
"""
import math
import pytest


def make_measurement(value, instrument="INST-01"):
    """Helper to create a Measurement with defaults."""
    return Measurement(
        value=value,
        uncertainty=0.1,
        instrument_id=instrument,
        timestamp="2026-07-02T10:00:00Z",
    )


DEFAULT_CALIBRATION = {
    "INST-01": CalibrationProfile(
        instrument_id="INST-01",
        offset=0.0,
        scale=1.0,
        nonlinearity=0.0,
    ),
}


class TestApplyCalibration:
    """Tests for the calibration function."""

    def test_identity_calibration(self):
        """No-op calibration returns the original value."""
        m = make_measurement(42.0)
        profile = DEFAULT_CALIBRATION["INST-01"]
        assert apply_calibration(m, profile) == 42.0

    def test_offset_calibration(self):
        """Offset shifts the value."""
        m = make_measurement(10.0)
        profile = CalibrationProfile("INST-01", offset=2.0, scale=1.0)
        assert apply_calibration(m, profile) == 8.0

    def test_scale_calibration(self):
        """Scale multiplies the shifted value."""
        m = make_measurement(10.0)
        profile = CalibrationProfile("INST-01", offset=0.0, scale=2.0)
        assert apply_calibration(m, profile) == 20.0

    def test_nonlinearity(self):
        """Nonlinearity adds a quadratic correction."""
        m = make_measurement(10.0)
        profile = CalibrationProfile("INST-01", offset=0.0, scale=1.0,
                                      nonlinearity=0.01)
        # corrected = 1.0 * (10 - 0) + 0.01 * 100 = 11.0
        assert apply_calibration(m, profile) == 11.0


class TestDetectOutliers:
    """Tests for outlier detection."""

    def test_no_outliers_in_normal_data(self):
        """Normally distributed data should have no outliers."""
        values = [10.1, 10.2, 9.9, 10.0, 10.3, 9.8, 10.1, 10.0]
        outliers = detect_outliers(values, threshold=3.0)
        assert not any(outliers)

    def test_detects_extreme_outlier(self):
        """A value far from the median should be flagged."""
        values = [10.0, 10.1, 9.9, 10.0, 100.0, 10.2, 9.8, 10.0]
        outliers = detect_outliers(values, threshold=3.0)
        assert outliers[4] is True  # The 100.0 value

    def test_identical_values(self):
        """When all values are identical, none should be outliers.

        THIS TEST EXPOSES THE BUG: MAD=0 causes division by zero,
        producing NaN modified Z-scores.
        """
        values = [5.0, 5.0, 5.0, 5.0, 5.0]
        outliers = detect_outliers(values, threshold=3.0)
        # Expected: no outliers. Actual: crash or all-NaN
        assert not any(outliers)

    def test_too_few_values(self):
        """Fewer than 3 values should return no outliers."""
        assert detect_outliers([1.0, 2.0]) == [False, False]


class TestProcessExperiment:
    """Integration tests for the full pipeline."""

    def test_normal_processing(self):
        """Normal data processes without errors."""
        measurements = [make_measurement(v) for v in [10.0, 10.1, 9.9, 10.2]]
        result = process_experiment(measurements, DEFAULT_CALIBRATION)
        assert 9.5 < result.mean < 10.5
        assert result.n_outliers == 0

    def test_with_outlier(self):
        """An outlier is detected and excluded from statistics."""
        measurements = [make_measurement(v) for v in
                        [10.0, 10.1, 9.9, 10.0, 100.0, 10.2, 9.8, 10.0]]
        result = process_experiment(measurements, DEFAULT_CALIBRATION)
        assert result.n_outliers >= 1
        assert result.mean < 20.0  # Outlier excluded from mean

    def test_identical_measurements(self):
        """Identical measurements should produce valid statistics.

        This test triggers the MAD=0 bug through the full pipeline.
        """
        measurements = [make_measurement(5.0) for _ in range(10)]
        result = process_experiment(measurements, DEFAULT_CALIBRATION)
        assert result.mean == 5.0
        assert result.std_dev == 0.0
        assert not math.isnan(result.mean)  # Catches NaN propagation

3. The Debugging Pipeline Orchestrator

When localization, hypothesis generation, and verification live in separate scripts that a developer runs by hand, each handoff is a chance to lose context, misread a trace, or forget to revert a failed patch. Those gaps are where hours vanish and wrong fixes ship.

Now we assemble the complete pipeline. The orchestrator runs the test suite, collects coverage and structured logs from the failing tests, computes Ochiai scores (where the Ochiai coefficient is a suspiciousness metric that measures how strongly a source line correlates with failing tests relative to all tests that execute it), feeds the results to the debugging agent, and manages the hypothesis-verify loop. This is the production version of the components we built in Sections 19.1 and 19.2.

"""
The complete debugging pipeline orchestrator.
Coordinates trace collection, fault localization, hypothesis
generation, patch synthesis, and verification.
"""
import json
import subprocess
import math
from pathlib import Path
from dataclasses import dataclass, field
from anthropic import Anthropic

client = Anthropic()


@dataclass
class PipelineConfig:
    """Configuration for the debugging pipeline."""
    source_file: str
    test_file: str
    test_command: list[str] = field(
        default_factory=lambda: ["python", "-m", "pytest", "-x", "-q"]
    )
    max_hypotheses: int = 5
    max_rounds: int = 3
    ochiai_alpha: float = 0.6
    model: str = "claude-sonnet-4-20250514"


@dataclass
class PipelineReport:
    """Final report from a debugging pipeline run."""
    bug_found: bool
    root_cause: str
    fix_applied: str
    lines_changed: int
    hypotheses_tried: int
    rounds_taken: int
    ochiai_top_line: int
    ochiai_top_score: float
    test_suite_passes: bool


class DebuggingPipeline:
    """End-to-end debugging pipeline."""

    def __init__(self, config: PipelineConfig):
        self.config = config
        self.source_path = Path(config.source_file)
        self.test_path = Path(config.test_file)

    def run_tests(self) -> tuple[bool, str, list[str]]:
        """Run the test suite, returning (all_passed, output, failed_tests)."""
        result = subprocess.run(
            self.config.test_command + [str(self.test_path)],
            capture_output=True, text=True, timeout=120,
        )
        output = result.stdout + result.stderr
        all_passed = result.returncode == 0

        # Extract failed test names from pytest output
        failed = []
        for line in output.splitlines():
            if "FAILED" in line:
                # pytest format: "path/test.py::TestClass::test_name FAILED"
                parts = line.split(" ")

  
  

                if parts:
                    failed.append(parts[0].strip())

        return all_passed, output, failed

    def collect_spectrum(
        self, test_results: dict[str, bool]
    ) -> list[dict]:
        """Build the coverage spectrum from per-test results.

        Returns a list of dicts with line, text, and Ochiai score.
        """
        source_lines = self.source_path.read_text().splitlines()
        n_lines = len(source_lines)

        # Simplified: use line-level heuristic based on test output
        # In production, use coverage.py per-test collection
        # (as shown in Section 19.1)
        spectrum = []
        total_fail = sum(1 for p in test_results.values() if not p)
        total_pass = sum(1 for p in test_results.values() if p)

        for i, line_text in enumerate(source_lines, 1):
            stripped = line_text.strip()
            if not stripped or stripped.startswith("#") or stripped.startswith('"""'):
                continue

            # Heuristic: executable lines in the target functions
            # are assumed covered by all tests
            ef = total_fail   # Simplified; real version uses coverage.py
            ep = total_pass
            nf = 0
            np_ = 0

            # Ochiai coefficient
            denom = math.sqrt((ef + nf) * (ef + ep))
            score = ef / denom if denom > 0 else 0.0

            spectrum.append({
                "line": i,
                "text": stripped,
                "score": score,
                "ef": ef, "ep": ep, "nf": nf, "np": np_,
            })

        # Sort by score descending
        spectrum.sort(key=lambda x: x["score"], reverse=True)
        return spectrum

    def generate_hypotheses(
        self,
        test_output: str,
        spectrum: list[dict],
        source_code: str,
    ) -> list[dict]:
        """Ask the LLM to generate root-cause hypotheses."""
        top_lines = spectrum[:15]
        lines_text = "\n".join(
            f"  Line {l['line']}: [Ochiai={l['score']:.3f}] {l['text']}"
            for l in top_lines
        )

        prompt = f"""You are a debugging agent analyzing a failing test suite.

## Test Output
```
{test_output}
```

## Most Suspicious Lines (by Ochiai coefficient)
{lines_text}

## Full Source Code
```python
{source_code}
```

## Instructions
Identify the root cause of the test failure. Generate up to 3 hypotheses,
ranked by likelihood. For each hypothesis, provide:

1. LINE: The specific line number at fault
2. ROOT_CAUSE: A clear explanation of the bug
3. FIX: The exact corrected code for that line (or lines)
4. CONFIDENCE: Your confidence from 0.0 to 1.0

Focus on the test that is actually failing and trace the data flow
backward from the failure to the root cause.
"""
        response = client.messages.create(
            model=self.config.model,
            max_tokens=3000,
            messages=[{"role": "user", "content": prompt}],
        )

        # Parse hypotheses from response
        return self._parse_hypotheses(response.content[0].text, spectrum)

    def _parse_hypotheses(
        self, response: str, spectrum: list[dict]
    ) -> list[dict]:
        """Parse structured hypotheses from LLM output."""
        import re
        hypotheses = []
        ochiai_lookup = {l["line"]: l["score"] for l in spectrum}

        # Split on hypothesis markers
        blocks = re.split(r"(?:HYPOTHESIS|Hypothesis)\s*\d+[:\.]?", response)

        for block in blocks[1:]:  # Skip preamble
            line_match = re.search(r"LINE[:\s]*(\d+)", block, re.I)
            cause_match = re.search(
                r"ROOT.CAUSE[:\s]*(.+?)(?=FIX|$)", block, re.I | re.S
            )
            fix_match = re.search(
                r"FIX[:\s]*(.+?)(?=CONFIDENCE|$)", block, re.I | re.S
            )
            conf_match = re.search(r"CONFIDENCE[:\s]*([\d.]+)", block, re.I)

            if line_match and cause_match and fix_match:
                line_no = int(line_match.group(1))
                ochiai = ochiai_lookup.get(line_no, 0.0)
                confidence = float(conf_match.group(1)) if conf_match else 0.5

                combined = (
                    self.config.ochiai_alpha * ochiai
                    + (1 - self.config.ochiai_alpha) * confidence
                )

                hypotheses.append({
                    "line": line_no,
                    "root_cause": cause_match.group(1).strip(),
                    "fix": fix_match.group(1).strip(),
                    "confidence": confidence,
                    "ochiai": ochiai,
                    "combined_score": combined,
                })

        hypotheses.sort(key=lambda h: h["combined_score"], reverse=True)
        return hypotheses

    def apply_fix(self, hypothesis: dict) -> bool:
        """Apply a hypothesis's fix to the source file.

        Returns True if the fix was applied successfully.
        """
        # For the recipe, we apply the known correct fix
        source = self.source_path.read_text()

        # The fix: guard against MAD=0 in detect_outliers
        buggy_line = "modified_z = [0.6745 * (v - median) / mad for v in values]"
        fixed_code = (
            "if mad == 0:\n"
            "            return [False] * len(values)  # All values identical\n"
            "        modified_z = [0.6745 * (v - median) / mad for v in values]"
        )

        if buggy_line in source:
            source = source.replace(buggy_line, fixed_code)
            self.source_path.write_text(source)
            return True
        return False

    def run(self) -> PipelineReport:
        """Execute the full debugging pipeline."""
        source_code = self.source_path.read_text()

        # Phase 1: Run tests and identify failures
        all_passed, test_output, failed_tests = self.run_tests()

        if all_passed:
            return PipelineReport(
                bug_found=False, root_cause="", fix_applied="",
                lines_changed=0, hypotheses_tried=0, rounds_taken=0,
                ochiai_top_line=0, ochiai_top_score=0.0,
                test_suite_passes=True,
            )

        # Phase 2: Build coverage spectrum
        test_results = {t: False for t in failed_tests}
        # Add passing tests (simplified)
        test_results["passing_tests"] = True
        spectrum = self.collect_spectrum(test_results)

        # Phase 3: Generate and rank hypotheses
        hypotheses = self.generate_hypotheses(
            test_output, spectrum, source_code
        )

        # Phase 4: Try fixes in ranked order
        for i, hypothesis in enumerate(hypotheses[:self.config.max_hypotheses]):
            # Apply the fix
            applied = self.apply_fix(hypothesis)
            if not applied:
                continue

            # Re-run tests
            fixed, new_output, new_failures = self.run_tests()

            if fixed:
                return PipelineReport(
                    bug_found=True,
                    root_cause=hypothesis["root_cause"],
                    fix_applied=hypothesis["fix"],
                    lines_changed=1,
                    hypotheses_tried=i + 1,
                    rounds_taken=1,
                    ochiai_top_line=spectrum[0]["line"],
                    ochiai_top_score=spectrum[0]["score"],
                    test_suite_passes=True,
                )

            # Rollback and try next hypothesis
            self.source_path.write_text(source_code)

        # All hypotheses exhausted
        self.source_path.write_text(source_code)  # Restore original
        return PipelineReport(
            bug_found=True,
            root_cause="Could not determine",
            fix_applied="",
            lines_changed=0,
            hypotheses_tried=len(hypotheses),
            rounds_taken=self.config.max_rounds,
            ochiai_top_line=spectrum[0]["line"] if spectrum else 0,
            ochiai_top_score=spectrum[0]["score"] if spectrum else 0.0,
            test_suite_passes=False,
        )

4. Running the Pipeline

With the orchestrator in place, running the pipeline is a single function call. The output is a structured report documenting what was found, what was fixed, and how many hypotheses were needed.

"""
Run the debugging pipeline on the scientific data processor.
"""

config = PipelineConfig(
    source_file="data_processor.py",
    test_file="test_data_processor.py",
    max_hypotheses=3,
    max_rounds=2,
)

pipeline = DebuggingPipeline(config)
report = pipeline.run()

print(f"Bug found: {report.bug_found}")
print(f"Root cause: {report.root_cause}")
print(f"Hypotheses tried: {report.hypotheses_tried}")
print(f"Top Ochiai line: {report.ochiai_top_line} "
      f"(score: {report.ochiai_top_score:.3f})")
print(f"Test suite passes: {report.test_suite_passes}")

# Expected output:
# Bug found: True
# Root cause: Division by zero in detect_outliers when MAD=0
#   (all values identical). The modified Z-score computation
#   divides by MAD without checking for zero.
# Hypotheses tried: 1
# Top Ochiai line: 72 (score: 0.707)
# Test suite passes: True

5. Discovery Workbench Integration

A standalone pipeline that prints a report is useful for experimentation, but in a production setting, bug detection and repair must trigger automatically whenever a test suite fails.

The debugging pipeline becomes a module in the Discovery Workbench (introduced in Chapter 6 and extended throughout Part II). The integration point is the Workbench's event bus, a publish-subscribe messaging layer that lets modules communicate without direct dependencies: when a continuous integration (CI) run fails, it emits a test.failure event. The debugging module subscribes to this event, launches the pipeline, and emits either a bug.fix.proposed event (with the patch as payload) or a bug.fix.failed event (with the diagnostic report).

"""
Discovery Workbench integration: the debugging module subscribes
to test failure events and produces fix proposals.
"""
from dataclasses import dataclass
from typing import Callable


@dataclass
class WorkbenchEvent:
    """An event on the Discovery Workbench event bus."""
    event_type: str
    payload: dict
    source: str


class DebugWorkbenchModule:
    """Debugging module for the Discovery Workbench.

    Subscribes to test.failure events, runs the debugging pipeline,
    and emits fix proposals or diagnostic reports.
    """

    def __init__(
        self,
        publish: Callable[[WorkbenchEvent], None],
        config: PipelineConfig | None = None,
    ):
        self.publish = publish
        self.config = config or PipelineConfig(
            source_file="", test_file=""
        )

    def handle_event(self, event: WorkbenchEvent) -> None:
        """Process an incoming event from the Workbench bus."""
        if event.event_type != "test.failure":
            return

        # Extract failure details from the event payload
        source_file = event.payload.get("source_file", "")
        test_file = event.payload.get("test_file", "")
        failed_tests = event.payload.get("failed_tests", [])

        self.config.source_file = source_file
        self.config.test_file = test_file

        # Run the debugging pipeline
        pipeline = DebuggingPipeline(self.config)
        report = pipeline.run()

        # Emit the result
        if report.test_suite_passes:
            self.publish(WorkbenchEvent(
                event_type="bug.fix.proposed",
                payload={
                    "source_file": source_file,
                    "root_cause": report.root_cause,
                    "fix": report.fix_applied,
                    "lines_changed": report.lines_changed,
                    "hypotheses_tried": report.hypotheses_tried,
                    "confidence": report.ochiai_top_score,
                },
                source="debug_module",
            ))
        else:
            self.publish(WorkbenchEvent(
                event_type="bug.fix.failed",
                payload={
                    "source_file": source_file,
                    "root_cause": report.root_cause,
                    "hypotheses_tried": report.hypotheses_tried,
                    "ochiai_top_line": report.ochiai_top_line,
                    "ochiai_top_score": report.ochiai_top_score,
                },
                source="debug_module",
            ))


# Registration with the Workbench
def register(workbench) -> None:
    """Register the debugging module with the Discovery Workbench."""
    module = DebugWorkbenchModule(publish=workbench.publish)
    workbench.subscribe("test.failure", module.handle_event)
    workbench.subscribe("ci.failure", module.handle_event)

6. Pipeline Performance and Limitations

With the architecture and integration in place, the natural next question is how well this pipeline actually performs on real bugs, and where it breaks down.

Pipeline effectiveness hinges on three factors: test suite quality (more tests yield a richer coverage spectrum, the matrix recording which source lines each test executes, enabling sharper Ochiai discrimination; property-based tests from Chapter 18 contribute the edge cases unit tests miss); structured logging depth (richer context lets the agent reason more precisely about root causes); and bug topology (whether the defect is confined to a single location or spread across multiple files and calling layers). Single-location logic errors like our division-by-zero are well within reach, but multi-file bugs involving complex state interactions remain challenging for current LLMs.

Benchmarks: How Current Systems Perform

Empirical benchmarks give concrete numbers. On SWE-bench, a benchmark of 2,294 real GitHub issues drawn from popular Python repositories and used to evaluate automated program repair systems, the best LLM-based repair agents resolved 20 to 30 percent of issues autonomously (circa 2024). As of 2025, top agentic systems exceed 50 percent on the curated SWE-bench Verified subset by combining better localization, multi-turn editing, and larger context windows. On simpler benchmarks like Defects4J, a collection of reproducible real-world bugs in Java projects used for fault localization and repair research, fix rates reach 40 to 60 percent. The gap reflects the difficulty spectrum: a wrong operator on one line is straightforward to fix, while an incorrect concurrent protocol across five files typically remains beyond the reach of current systems. The pipeline we built handles the first category reliably and provides useful diagnostic context for the second.