19.2 Self-Debugging Agents

1. The Hypothesis-Rank-Verify Loop

When a test suite turns red at 2 a.m., no experienced developer reads the codebase from line one. She scans the traceback, forms a mental shortlist of suspects, and tests her top theory before touching anything else. That instinct (hypothesis first, evidence second) is exactly the protocol an effective self-debugging agent follows. The agent replaces intuition with structured evidence from the fault localizer and replaces experience with the LLM's training distribution.

In production systems, a single undetected regression can cascade into hours of downtime or corrupt data that takes days to recover. Automating the first pass of diagnosis turns that exposure from hours into minutes.

A self-debugging agent is an LLM-powered program that autonomously diagnoses and repairs software faults. It executes a structured loop: observe, hypothesize, patch, verify. Manual debugging typically consumes 30-50% of total engineering effort according to industry surveys, so an agent that handles routine faults frees developers to focus on design and architecture. The agent receives a curated context window: failing test output, ranked suspicious lines, and structured logs. It then generates candidate fixes, each tested against the real test suite in a sandboxed environment. This approach works best for well-tested codebases where failures produce clear, reproducible test output. For exploratory code with no test suite, interactive debugging or logging-based diagnosis remains more appropriate. In short: If your test suite can catch a bug, a self-debugging agent can close the loop from failure to fix without a human in the chair.

The Four Stages

The loop has four stages, illustrated in Figure 19.2: Figure 19.2.1 illustrates the hypothesis-rank-verify debugging loop.

1. Observe: Collect the failing test output, the traceback, the structured logs around the failure point, and the ranked list of suspicious lines produced by the Ochiai coefficient (a statistical measure that scores how strongly each source line correlates with failing tests) from Section 19.1.
2. Hypothesize: Ask the LLM to generate \(k\) candidate root-cause explanations, each tied to a specific suspicious line or code region.
3. Rank: Score each hypothesis by combining the Ochiai suspiciousness of the implicated lines with the LLM's own confidence estimate and the semantic plausibility of the explanation.
4. Verify: For the top-ranked hypothesis, generate a targeted patch, apply it, re-run the failing test (plus the full suite to check for regressions), and observe the outcome. If the test passes, the bug is fixed. If it fails, update the ranking and try the next hypothesis.

"""
Self-debugging agent: the hypothesis-rank-verify loop.
Reads fault localization output and structured logs, generates
root-cause hypotheses, and iteratively patches until tests pass.
"""
from dataclasses import dataclass, field
from anthropic import Anthropic


@dataclass
class DebugHypothesis:
    """A candidate root-cause explanation."""
    line_number: int
    source_text: str
    ochiai_score: float
    explanation: str
    confidence: float  # LLM's self-assessed confidence [0, 1]
    combined_score: float = 0.0

    def compute_combined_score(self, alpha: float = 0.6) -> None:
        """Blend Ochiai suspiciousness with LLM confidence.

        alpha controls the weight of the statistical signal vs.
        the LLM's semantic judgment. Higher alpha trusts the
        coverage data more; lower alpha trusts the LLM more.
        """
        self.combined_score = (
            alpha * self.ochiai_score
            + (1 - alpha) * self.confidence
        )


@dataclass
class DebugContext:
    """All evidence available to the debugging agent."""
    failing_test: str          # Test function source code
    test_output: str           # Captured stdout/stderr + traceback
    suspicious_lines: list[dict]  # From fault localizer (line, text, score)
    log_events: list[dict]     # Structured log entries from the failing run
    source_context: dict[int, list[str]]  # line_no -> surrounding lines


def format_debug_prompt(ctx: DebugContext) -> str:
    """Build the debugging prompt from structured evidence."""
    lines_section = "\n".join(
        f"  Line {l['line']}: (Ochiai={l['score']:.3f}) {l['text']}"
        for l in ctx.suspicious_lines[:10]
    )

    logs_section = "\n".join(
        f"  [{ev.get('level', 'info')}] {ev.get('event', '')} "
        f"| {', '.join(f'{k}={v}' for k, v in ev.items() if k not in ('level', 'event', 'timestamp'))}"
        for ev in ctx.log_events[-20:]  # Last 20 log events
    )

    context_section = ""
    for line_no, lines in ctx.source_context.items():
        context_section += f"\n  --- Around line {line_no} ---\n"
        for i, line in enumerate(lines):
            marker = " >> " if i == len(lines) // 2 else "    "
            context_section += f"  {marker}{line}\n"

    return f"""You are a debugging agent. A test is failing and you must identify the root cause.

## Failing Test
```python
{ctx.failing_test}
```

## Test Output (including traceback)
```
{ctx.test_output}
```

## Suspicious Lines (ranked by Ochiai coefficient)
{lines_section}

## Structured Log Events (from the failing execution)
{logs_section}

## Source Context (around suspicious lines)
{context_section}

## Your Task
Generate exactly 3 root-cause hypotheses. For each hypothesis:
1. Identify the specific line(s) at fault.
2. Explain WHY this line causes the observed failure.
3. Propose a concrete fix (the exact code change).
4. Rate your confidence from 0.0 to 1.0.

Format each hypothesis as:
HYPOTHESIS N:
LINE: 
EXPLANATION: 
FIX: 
CONFIDENCE: <0.0 to 1.0>
"""

2. Parsing and Ranking Hypotheses

The LLM returns hypotheses as structured text. A parser extracts each one, computes a combined score blending Ochiai suspiciousness with the LLM's confidence, and sorts by decreasing score. The parameter \(\alpha\) controls this tradeoff: higher values favor the coverage data, lower values favor the LLM's reasoning. At \(\alpha = 0.6\), the statistical signal takes slight precedence, but the LLM can still promote a low-Ochiai line when semantic evidence is compelling.

"""
Parse LLM-generated hypotheses and rank by combined score.
"""
import re


def parse_hypotheses(
    llm_response: str,
    suspicious_lines: list[dict],
) -> list[DebugHypothesis]:
    """Extract structured hypotheses from LLM output."""
    hypotheses = []

    # Build a lookup from line number to Ochiai score
    ochiai_lookup = {l["line"]: l["score"] for l in suspicious_lines}

    # Parse each HYPOTHESIS block
    blocks = re.split(r"HYPOTHESIS\s+\d+:", llm_response)[1:]

    for block in blocks:
        line_match = re.search(r"LINE:\s*(\d+)", block)
        expl_match = re.search(r"EXPLANATION:\s*(.+?)(?=FIX:|$)", block, re.S)
        fix_match = re.search(r"FIX:\s*(.+?)(?=CONFIDENCE:|$)", block, re.S)
        conf_match = re.search(r"CONFIDENCE:\s*([\d.]+)", block)

        if not all([line_match, expl_match, fix_match, conf_match]):
            continue

        line_no = int(line_match.group(1))
        hypothesis = DebugHypothesis(
            line_number=line_no,
            source_text=ochiai_lookup.get(line_no, {}).get("text", ""),
            ochiai_score=ochiai_lookup.get(line_no, 0.0),
            explanation=expl_match.group(1).strip(),
            confidence=min(1.0, max(0.0, float(conf_match.group(1)))),
        )
        hypothesis.compute_combined_score(alpha=0.6)
        hypotheses.append(hypothesis)

    # Sort by combined score, descending
    hypotheses.sort(key=lambda h: h.combined_score, reverse=True)
    return hypotheses

3. Automated Patch Generation and Verification

Once hypotheses are ranked, the agent applies the top-ranked fix, runs the test suite, and observes the result. If the fix resolves the failing test without introducing regressions, the debugging session succeeds. If the fix fails (the test still fails, or new tests break), the agent moves to the next hypothesis. This verify-or-retry loop runs until either a fix succeeds or the hypothesis budget is exhausted.

"""
Patch-and-verify loop: apply each hypothesis's fix and test it.
"""
import subprocess
import shutil
from pathlib import Path


@dataclass
class PatchResult:
    """Outcome of applying and testing a patch."""
    hypothesis: DebugHypothesis
    patch_applied: bool
    target_test_passed: bool
    regression_tests_passed: bool
    error_output: str = ""


def apply_and_verify(
    source_file: Path,
    hypotheses: list[DebugHypothesis],
    test_command: list[str],
    target_test: str,
    max_attempts: int = 3,
) -> PatchResult | None:
    """Try each hypothesis's fix until one passes all tests.

    Args:
        source_file: Path to the file being debugged.
        hypotheses: Ranked list of hypotheses with proposed fixes.
        test_command: Command to run the full test suite.
        target_test: Specific test ID that is failing.
        max_attempts: Maximum number of hypotheses to try.

    Returns:
        PatchResult for the successful fix, or None if all fail.
    """
    # Save the original file for rollback
    backup = source_file.with_suffix(".bak")
    shutil.copy2(source_file, backup)

    for hypothesis in hypotheses[:max_attempts]:
        # Restore original before each attempt
        shutil.copy2(backup, source_file)

        # Apply the patch
        original_lines = source_file.read_text().splitlines()
        line_idx = hypothesis.line_number - 1  # 0-indexed

        if line_idx < 0 or line_idx >= len(original_lines):
            continue

        # Replace the faulty line with the fix
        patched_lines = original_lines.copy()
        patched_lines[line_idx] = hypothesis.explanation  # Simplified
        source_file.write_text("\n".join(patched_lines) + "\n")

        # Run the specific failing test first (fast feedback)
        target_result = subprocess.run(
            ["python", "-m", "pytest", target_test, "-x", "-q"],
            capture_output=True, text=True, timeout=60,
        )
        target_passed = target_result.returncode == 0

        if not target_passed:
            continue  # This fix did not resolve the failure

        # Run full suite to check for regressions
        full_result = subprocess.run(
            test_command,
            capture_output=True, text=True, timeout=300,
        )
        regression_passed = full_result.returncode == 0

        result = PatchResult(
            hypothesis=hypothesis,
            patch_applied=True,
            target_test_passed=target_passed,
            regression_tests_passed=regression_passed,
            error_output=full_result.stderr if not regression_passed else "",
        )

        if target_passed and regression_passed:
            backup.unlink()  # Clean up backup
            return result

    # All attempts failed; restore original
    shutil.copy2(backup, source_file)
    backup.unlink()
    return None

4. Programmatic Debugging: Inspecting Runtime State

The patch-and-verify loop confirms whether a fix is correct, but when every generated patch keeps failing, the agent needs to look deeper into what the program is actually doing at runtime. Sometimes the agent needs more than static traces and coverage data. It needs to inspect runtime state: the values of variables at a specific point in execution, the contents of a data structure just before a crash, the return value of a function that behaves unexpectedly. Python offers several mechanisms for programmatic inspection. The built-in sys.settrace function lets you install a callback that fires on every line execution, providing access to local and global variables without requiring any external library. For richer interactive sessions, the debugpy library implements the Debug Adapter Protocol (DAP), a standard interface that lets editors and tools control debuggers programmatically, supporting breakpoints, stepping, and watch expressions. Both approaches let the agent inspect variables from Python code rather than a graphical user interface (GUI). The example below uses sys.settrace for lightweight, dependency-free inspection (a trace function is a Python callable that the interpreter invokes before executing each line, giving it access to the current stack frame and local variables).

"""
Programmatic debugging with debugpy: inspect runtime state from code.
The agent sets breakpoints and reads variable values without a GUI.
"""
import debugpy
import threading
import subprocess
import json


def inspect_at_breakpoint(
    script_path: str,
    breakpoint_line: int,
    variables_to_inspect: list[str],
    script_args: list[str] | None = None,
) -> dict:
    """Run a script with a breakpoint, inspecting variables when hit.

    This launches the script under debugpy, sets a breakpoint, and
    captures variable values when execution reaches that line.

    Args:
        script_path: Path to the Python script to debug.
        breakpoint_line: Line number where to pause execution.
        variables_to_inspect: Names of variables to capture.
        script_args: Optional command-line arguments for the script.

    Returns:
        Dict mapping variable names to their string representations.
    """
    captured = {}
    breakpoint_hit = threading.Event()

    def on_breakpoint(frame, event, arg):
        """Trace function that captures variables at the breakpoint."""
        if event == "line" and frame.f_lineno == breakpoint_line:
            for var_name in variables_to_inspect:
                if var_name in frame.f_locals:
                    captured[var_name] = repr(frame.f_locals[var_name])
                elif var_name in frame.f_globals:
                    captured[var_name] = repr(frame.f_globals[var_name])
                else:
                    captured[var_name] = ""
            breakpoint_hit.set()
        return on_breakpoint

    # Use sys.settrace for lightweight inspection
    import sys
    import runpy

    original_trace = sys.gettrace()
    sys.settrace(on_breakpoint)
    try:
        # Run the target script
        runpy.run_path(script_path, run_name="__main__")
    except Exception as exc:
        captured["__exception__"] = repr(exc)
    finally:
        sys.settrace(original_trace)

    return captured


# Example usage: inspect variables at a suspicious line
results = inspect_at_breakpoint(
    script_path="app/order_processor.py",
    breakpoint_line=42,
    variables_to_inspect=["total", "discount", "item_count"],
)
# results might be:
# {"total": "135.0", "discount": "15.0", "item_count": "12"}

5. Multi-Agent Debugging Teams

Programmatic inspection gives a single agent sharper eyes, but some bugs are too tangled for one agent to localize, explain, patch, and verify alone. The single-agent loop from sections 1 through 4 packs localization, explanation, patching, and verification into one prompt, which can exceed the model's effective attention span on complex faults. Complex bugs benefit from specialized perspectives. Drawing on the multi-agent patterns from Chapter 17: Multi-Agent Software Teams, debugging decomposes into four specialized roles, each handled by a separate agent with its own system prompt and tool access.

• Localizer: Runs the fault localizer, collects the execution spectrum (a matrix recording which source lines each test executed), computes Ochiai scores, and produces the ranked list of suspicious lines.
• Explainer: Reads the suspicious lines, structured logs, and source context. Generates natural-language hypotheses about the root cause.
• Patcher: Takes a hypothesis and generates a concrete code patch. Has access to the full source file and the coding conventions of the project.
• Verifier: Applies the patch, runs the test suite, and reports whether the fix resolves the failure without regressions.

"""
Multi-agent debugging team: four specialized agents coordinate
to localize, explain, patch, and verify a bug fix.
"""
from dataclasses import dataclass
from anthropic import Anthropic

client = Anthropic()


@dataclass
class AgentRole:
    """Configuration for a specialized debugging agent."""
    name: str
    system_prompt: str
    model: str = "claude-sonnet-4-20250514"


DEBUGGING_TEAM = {
    "localizer": AgentRole(
        name="Localizer",
        system_prompt="""You are a fault localization specialist. Given test
results and coverage data, you identify the most suspicious source lines.
You produce ranked lists with Ochiai scores and brief justifications.
You never propose fixes; that is the Patcher's job.""",
    ),
    "explainer": AgentRole(
        name="Explainer",
        system_prompt="""You are a root-cause analyst. Given suspicious lines,
structured logs, and source context, you explain WHY the bug occurs.
You trace data flow, identify incorrect assumptions, and produce clear
causal chains from the root cause to the observed symptom.
You never write code; that is the Patcher's job.""",
    ),
    "patcher": AgentRole(
        name="Patcher",
        system_prompt="""You are a code repair specialist. Given a root-cause
explanation and the surrounding source code, you generate a minimal,
correct patch. You follow the project's coding style. You change as
few lines as possible. You add a comment explaining the fix.""",
    ),
    "verifier": AgentRole(
        name="Verifier",
        system_prompt="""You are a verification specialist. You apply patches,
run test suites, and analyze results. You check for both fix correctness
(does the failing test pass?) and regression safety (do all other tests
still pass?). You report pass/fail with detailed evidence.""",
    ),
}


def run_debugging_team(
    failing_test: str,
    test_output: str,
    source_file: str,
    source_code: str,
    coverage_data: dict,
    log_events: list[dict],
    max_rounds: int = 3,
) -> dict:
    """Orchestrate the multi-agent debugging team.

    The agents communicate through a shared context dictionary,
    each adding their contribution in sequence.
    """
    context = {
        "failing_test": failing_test,
        "test_output": test_output,
        "source_file": source_file,
        "source_code": source_code,
        "coverage_data": coverage_data,
        "log_events": log_events,
        "round": 0,
        "status": "in_progress",
    }

    for round_num in range(max_rounds):
        context["round"] = round_num + 1

        # Step 1: Localizer identifies suspicious lines
        loc_response = client.messages.create(
            model=DEBUGGING_TEAM["localizer"].model,
            max_tokens=2000,
            system=DEBUGGING_TEAM["localizer"].system_prompt,
            messages=[{
                "role": "user",
                "content": (
                    f"Test output:\n{context['test_output']}\n\n"
                    f"Coverage data:\n{context['coverage_data']}\n\n"
                    f"Source code:\n{context['source_code']}"
                ),
            }],
        )
        context["localization"] = loc_response.content[0].text

        # Step 2: Explainer forms root-cause hypothesis
        exp_response = client.messages.create(
            model=DEBUGGING_TEAM["explainer"].model,
            max_tokens=2000,
            system=DEBUGGING_TEAM["explainer"].system_prompt,
            messages=[{
                "role": "user",
                "content": (
                    f"Suspicious lines:\n{context['localization']}\n\n"
                    f"Log events:\n{context['log_events']}\n\n"
                    f"Source:\n{context['source_code']}"
                ),
            }],
        )
        context["explanation"] = exp_response.content[0].text

        # Step 3: Patcher generates a fix
        patch_response = client.messages.create(
            model=DEBUGGING_TEAM["patcher"].model,
            max_tokens=2000,
            system=DEBUGGING_TEAM["patcher"].system_prompt,
            messages=[{
                "role": "user",
                "content": (
                    f"Root cause:\n{context['explanation']}\n\n"
                    f"Source:\n{context['source_code']}"
                ),
            }],
        )
        context["patch"] = patch_response.content[0].text

        # Step 4: Verifier tests the fix
        # (In production, this agent would apply the patch and run pytest)
        ver_response = client.messages.create(
            model=DEBUGGING_TEAM["verifier"].model,
            max_tokens=1000,
            system=DEBUGGING_TEAM["verifier"].system_prompt,
            messages=[{
                "role": "user",
                "content": (
                    f"Proposed patch:\n{context['patch']}\n\n"
                    f"Failing test:\n{context['failing_test']}\n\n"
                    "Analyze whether this patch correctly addresses "
                    "the root cause and is unlikely to cause regressions."
                ),
            }],
        )
        context["verification"] = ver_response.content[0].text

        # Check if verifier approved
        if "APPROVED" in context["verification"].upper():
            context["status"] = "fixed"
            break

        # Feed failure info back for next round
        context["test_output"] += (
            f"\n\n--- Round {round_num + 1} failed ---\n"
            f"Patch attempted:\n{context['patch']}\n"
            f"Verification:\n{context['verification']}"
        )

    return context

6. Root-Cause Extraction from Structured Logs

Whether the debugging work is split across a team of agents or handled by one, every agent in the pipeline depends on the quality of the evidence it receives, and the richest vein of evidence is often the execution log itself. The structured logs from Section 19.1 become particularly valuable when the agent can query them semantically. Instead of scanning raw text for keywords, the agent asks questions like "what was the value of discount in the last order.bulk_discount_applied event before the failure?" This requires a thin query layer over the log store.

"""
Semantic log querying: let the debugging agent ask structured
questions about log events from the failing execution.
"""
from datetime import datetime


class LogQueryEngine:
    """Query structured log events for debugging context."""

    def __init__(self, events: list[dict]):
        self.events = events

    def filter_by_event(self, event_name: str) -> list[dict]:
        """Get all log entries matching an event name pattern."""
        return [
            e for e in self.events
            if event_name in e.get("event", "")
        ]

    def get_field_values(
        self, event_name: str, field: str
    ) -> list:
        """Extract a specific field from matching events."""
        return [
            e[field]
            for e in self.filter_by_event(event_name)
            if field in e
        ]

    def events_before_error(
        self, n: int = 10
    ) -> list[dict]:
        """Get the last N events before the first error-level event."""
        for i, event in enumerate(self.events):
            if event.get("level") in ("error", "critical", "exception"):
                start = max(0, i - n)
                return self.events[start:i + 1]
        return self.events[-n:]  # No error found; return last N

    def trace_field_evolution(
        self, field: str
    ) -> list[tuple[str, object]]:
        """Track how a field's value changes across events.

        Returns list of (event_name, field_value) showing
        the progression of a variable through the execution.
        """
        evolution = []
        for event in self.events:
            if field in event:
                evolution.append((event.get("event", "unknown"), event[field]))
        return evolution


# Example: trace how 'total' evolves through an order processing run
engine = LogQueryEngine(log_events)
total_history = engine.trace_field_evolution("total")
# [("order.item_added", 50.0), ("order.item_added", 100.0),
#  ("order.bulk_discount_applied", 90.0), ("order.completed", 90.0)]

pre_error = engine.events_before_error(n=5)
# The 5 events leading up to the error, giving temporal context