9.3 Verification and Repair

1. The Verification Stack

What if your code passes every unit test you wrote, yet silently corrupts data the moment a real user supplies an input you never considered? Example-based tests only check the specific inputs you thought of. Property-based tests only check the properties you defined. Type checkers only catch type errors, not logic errors. Each tool has blind spots that the others compensate for. The verification stack arranges these tools in a deliberate order, from fastest and cheapest to slowest and most thorough, as shown in Figure 9.3.1.

A verification stack is a fixed sequence of automated checking tools, ordered by speed and cost. It runs against every version of the generated code. No single tool catches all categories of defect: a linter sees style violations but not logic errors, while a test suite sees logic errors but not type mismatches. Each layer runs independently, produces a pass/fail verdict with structured details, and the runner aggregates all results into one report before any repair begins. Use a verification stack whenever you iterate on AI-generated code (the vibe coding loop). For a one-off script you will never modify again, pytest alone may suffice. For anything that will evolve across multiple generate/repair cycles, the full stack pays for itself within two or three iterations by catching errors that would otherwise cascade.

Layer 1: Static analysis (mypy and pyright, which are static type checkers that verify type annotations without running the code, along with ruff, a fast Python linter that checks style and catches common errors) runs in seconds, requires no test inputs, and catches type mismatches, unused imports, unreachable code, and style violations. Static analysis is the cheapest verification layer because it requires no test infrastructure and provides immediate feedback. In the vibe coding loop, static analysis runs first: if the generated code has type errors, there is no point running the test suite.

Layer 2: Unit tests (pytest) run in seconds to minutes and verify specific input-output pairs. Each test encodes one behavioral decision from the specification. Unit tests are the backbone of the executable contract because they are easy to write, easy to understand, and produce clear pass/fail signals that the model can act on.

Layer 3: Property-based tests (Hypothesis) run in seconds to minutes and verify invariants (properties that must hold true for all valid inputs, such as "the output list is always sorted" or "no input element is lost") across randomly generated inputs. Property tests are more powerful than unit tests because they explore the input space automatically, finding edge cases the developer never considered. They are also harder to write because they require the developer to articulate general properties rather than specific examples.

Layer 4: Integration and end-to-end tests (Playwright, httpx) run in seconds to minutes and verify that components work together. End-to-end tests are the most expensive layer but also the most realistic: they test the system as a user would experience it. Section 9.4 covers browser-based testing with Playwright. In short: Each layer in the stack exists because every other layer has a blind spot.

import subprocess
import json
from dataclasses import dataclass

@dataclass
class VerificationResult:
    """Result from one layer of the verification stack."""
    layer: str
    passed: bool
    details: str
    duration_seconds: float

def run_verification_stack(
    source_file: str,
    test_file: str,
) -> list[VerificationResult]:
    """
    Run the full verification stack against generated code.

    Layers run in order; each layer runs regardless of whether
    previous layers passed (to collect all issues at once).
    """
    results = []
    import time

    # Layer 1: Static analysis with ruff (linting)
    t0 = time.time()
    ruff_result = subprocess.run(
        ["ruff", "check", source_file, "--output-format=json"],
        capture_output=True, text=True,
    )
    results.append(VerificationResult(
        layer="ruff",
        passed=ruff_result.returncode == 0,
        details=ruff_result.stdout[:500],
        duration_seconds=time.time() - t0,
    ))

    # Layer 2: Type checking with mypy
    t0 = time.time()
    mypy_result = subprocess.run(
        ["mypy", source_file, "--ignore-missing-imports"],
        capture_output=True, text=True,
    )
    results.append(VerificationResult(
        layer="mypy",
        passed=mypy_result.returncode == 0,
        details=mypy_result.stdout[:500],
        duration_seconds=time.time() - t0,
    ))

    # Layer 3: pytest (unit + property-based tests)
    t0 = time.time()
    pytest_result = subprocess.run(
        ["pytest", test_file, "-v", "--tb=short"],
        capture_output=True, text=True,
    )
    results.append(VerificationResult(
        layer="pytest",
        passed=pytest_result.returncode == 0,
        details=pytest_result.stdout[-1000:],  # last 1000 chars for failures
        duration_seconds=time.time() - t0,
    ))

    return results

def format_for_repair(results: list[VerificationResult]) -> str:
    """
    Format verification results as a repair prompt for the model.

    Only includes failing layers, with actionable details.
    """
    failures = [r for r in results if not r.passed]
    if not failures:
        return "All verification layers passed."

    lines = ["The following verification layers failed:\n"]
    for f in failures:
        lines.append(f"## {f.layer} (failed in {f.duration_seconds:.1f}s)")
        lines.append(f.details)
        lines.append("")

    lines.append("Please fix all issues. Do not change the test file.")
    return "\n".join(lines)

2. Property-Based Testing with Hypothesis

Example-based tests verify that specific inputs produce specific outputs. Property-based tests verify that all inputs (within a defined domain) satisfy a general property. The distinction is profound: an example test says "for input [3, 1, 2], the output is [1, 2, 3]"; a property test says "for any list of integers, the output is a sorted permutation of the input." The property test subsumes infinitely many example tests.

Mental Model

Think of property-based testing like a building inspector versus a homeowner. A homeowner (example-based testing) checks that the kitchen faucet works, the bedroom light switch flips, and the front door locks. An inspector (property-based testing) checks that every faucet delivers water within a pressure range, every switch controls the correct circuit, and every door meets fire-code clearance. The inspector does not enumerate each fixture individually; instead, she states a rule ("all outlets must be grounded") and then walks through the house testing outlets she encounters, including ones the homeowner forgot existed. Similarly, Hypothesis states a property ("the output is always sorted") and then generates inputs the developer never considered, including the empty list, the single-element list, and the list with all identical values.

Hypothesis is the standard property-based testing library for Python. It generates random inputs according to strategies (descriptions of valid input domains), runs the function under test, and checks that the specified property holds. When Hypothesis finds a failing input, it automatically shrinks it (that is, it reduces the failing input to the smallest example that still triggers the failure), making the bug report maximally informative.

from hypothesis import given, settings, assume
from hypothesis import strategies as st

# --- The function under test (generated by AI) ---
def merge_sorted(a: list[int], b: list[int]) -> list[int]:
    """Merge two sorted lists into one sorted list."""
    result = []
    i = j = 0
    while i < len(a) and j < len(b):
        if a[i] <= b[j]:
            result.append(a[i])
            i += 1
        else:
            result.append(b[j])
            j += 1
    result.extend(a[i:])
    result.extend(b[j:])
    return result

# --- Property-based tests ---

@given(
    a=st.lists(st.integers(min_value=-1000, max_value=1000)),
    b=st.lists(st.integers(min_value=-1000, max_value=1000)),
)
def test_merge_preserves_elements(a: list[int], b: list[int]):
    """Property: merging preserves all elements (no drops, no duplicates)."""
    a_sorted = sorted(a)
    b_sorted = sorted(b)
    merged = merge_sorted(a_sorted, b_sorted)
    assert sorted(merged) == sorted(a_sorted + b_sorted)

@given(
    a=st.lists(st.integers(min_value=-1000, max_value=1000)),
    b=st.lists(st.integers(min_value=-1000, max_value=1000)),
)
def test_merge_output_is_sorted(a: list[int], b: list[int]):
    """Property: output is always sorted."""
    a_sorted = sorted(a)
    b_sorted = sorted(b)
    merged = merge_sorted(a_sorted, b_sorted)
    for i in range(len(merged) - 1):
        assert merged[i] <= merged[i + 1], (
            f"Output not sorted at index {i}: {merged[i]} > {merged[i+1]}"
        )

@given(
    a=st.lists(st.integers(min_value=-1000, max_value=1000)),
    b=st.lists(st.integers(min_value=-1000, max_value=1000)),
)
def test_merge_length_is_sum(a: list[int], b: list[int]):
    """Property: output length equals sum of input lengths."""
    a_sorted = sorted(a)
    b_sorted = sorted(b)
    merged = merge_sorted(a_sorted, b_sorted)
    assert len(merged) == len(a_sorted) + len(b_sorted)

The three properties above form a near-complete characterization (a set of properties that, taken together, constrain the function's behavior so tightly that any implementation satisfying all of them is typically correct) of the merge function: any function that preserves elements, produces sorted output, and preserves total length is, in practice, a correct merge. This characterization does not cover every conceivable property (for example, it does not verify stability of equal elements from different lists), but it is more powerful than any finite set of example tests because it covers the entire input domain for the properties it does specify.

That theoretical completeness becomes a practical superpower once the code under test is generated by a model whose implicit assumptions you cannot inspect. In the vibe coding loop, property-based tests serve a special role: they catch bugs that arise from the implicit assumptions component of the specification gap (Section 9.1). The developer may not think to test what happens when both input lists are empty, or when they contain duplicate values, or when one list is much longer than the other. Hypothesis generates all of these cases automatically.

import numpy as np
from hypothesis import given
from hypothesis import strategies as st
from hypothesis.extra.numpy import arrays

@given(
    v=arrays(
        dtype=np.float64,
        shape=st.integers(min_value=1, max_value=100),
        elements=st.floats(
            min_value=-1e308, max_value=1e308,
            allow_nan=False, allow_infinity=False,
        ),
    )
)
def test_normalize_unit_length(v):
    """Property: normalized vector has unit length (or input is zero)."""
    norm = np.linalg.norm(v)
    if norm == 0:
        # Zero vector: function should return zero vector or raise
        try:
            result = normalize(v)
            assert np.allclose(result, 0.0)
        except ValueError:
            pass  # raising is also acceptable
    else:
        result = normalize(v)
        result_norm = np.linalg.norm(result)
        assert abs(result_norm - 1.0) < 1e-6, (
            f"Expected unit norm, got {result_norm}"
        )

3. Schema Validation with Pydantic

Property-based tests and example-based tests both verify that code behaves correctly, but an entire category of defect lies outside their reach: violations in the shape and type of the data itself. Where the previous two sections focused on whether code produces the right answers, schema validation asks a different question: does the data flowing between components have the right structure? In vibe coding for data-intensive applications (which most scientific applications are), the most common bugs are structural: a function returns a dictionary with the wrong keys, a nested field has the wrong type, or a numeric value falls outside its valid range. Pydantic, a Python library for data validation using type annotations and declarative field constraints, catches all of these automatically.

A single Pydantic model acts as type annotation (readable by mypy), runtime validator, JSON serializer, and documentation artifact (generates JSON Schema). One definition delivers all four functions.

from pydantic import BaseModel, Field, field_validator
from typing import Optional
from datetime import datetime

class ExperimentResult(BaseModel):
    """
    Schema contract for a single experiment result.

    This model serves as the executable contract between the data
    ingestion pipeline (AI-generated) and the analysis pipeline
    (human-written). Any violation raises ValidationError with
    a precise error message that can be fed back to the model.
    """
    experiment_id: str = Field(
        pattern=r"^EXP-\d{6}$",
        description="Experiment ID in format EXP-NNNNNN",
    )
    timestamp: datetime = Field(
        description="When the measurement was taken (UTC)",
    )
    measurement: float = Field(
        ge=-273.15,  # absolute zero in Celsius
        le=10000.0,
        description="Temperature measurement in Celsius",
    )
    uncertainty: float = Field(
        gt=0.0,
        description="Measurement uncertainty (must be positive)",
    )
    instrument: str = Field(
        min_length=1,
        max_length=100,
        description="Instrument identifier",
    )
    notes: Optional[str] = Field(
        default=None,
        max_length=1000,
        description="Optional free-text notes",
    )

    @field_validator("measurement")
    @classmethod
    def measurement_precision(cls, v: float) -> float:
        """Measurements should not have more than 4 decimal places."""
        if round(v, 4) != v:
            raise ValueError(
                f"Measurement {v} has excessive precision; "
                f"round to 4 decimal places"
            )
        return v

# Valid: passes all constraints
valid = ExperimentResult(
    experiment_id="EXP-000042",
    timestamp="2025-06-15T10:30:00Z",
    measurement=23.5,
    uncertainty=0.1,
    instrument="Thermocouple-A7",
)
print(valid.model_dump_json(indent=2))

# Invalid: multiple violations caught at once
try:
    invalid = ExperimentResult(
        experiment_id="EXP42",         # wrong format
        timestamp="2025-06-15",
        measurement=-300.0,            # below absolute zero
        uncertainty=-0.1,              # negative uncertainty
        instrument="",                 # too short
    )
except Exception as e:
    print(f"Validation errors:\n{e}")
    # Pydantic reports ALL violations, not just the first one

In the vibe coding loop, Pydantic models serve as the interface contract between components. When you ask the model to generate a data ingestion function, you provide the Pydantic model as part of the specification: "The function must return a list of ExperimentResult objects." If the generated function returns data that violates the schema, Pydantic's error message is detailed enough to serve directly as steering feedback. This technique extends naturally to MCP server development (Chapter 12), where Pydantic models define the tool input/output schemas.

4. The Repair Loop: Feeding Failures Back

Verification without repair is diagnosis without treatment. The repair loop takes verification failures, formats them as actionable feedback, and feeds them back to the model. The quality of this formatting directly affects repair efficiency (the \(\alpha_t\) parameter from Section 9.2, where \(\alpha_t\) measures the fraction of remaining defects eliminated per iteration).

Three principles govern effective repair feedback:

Principle 1: Show the full failure context. Do not paraphrase error messages. Include the complete stack trace, the failing assertion, the actual versus expected values, and the test name. The model needs the raw data, not your interpretation of it.

Structuring the Repair Message

Principle 2: Separate structural failures from behavioral failures. A type error ("expected int, got str") requires a different repair strategy than a logic error ("expected 42, got 41"). Presenting them separately helps the model prioritize. Fix structural issues first (they often cause cascading behavioral failures), then address behavioral issues.

Principle 3: Constrain the repair scope. Tell the model which files it may modify and which it must not. "Fix the implementation in pipeline.py. Do not modify test_pipeline.py." This prevents the model from "fixing" a test failure by weakening the test, which is a common failure mode when the repair prompt is unconstrained.

from dataclasses import dataclass

@dataclass
class RepairPrompt:
    """Structured repair prompt for the vibe coding loop."""
    original_spec: str
    source_file: str
    test_file: str
    failures: list[VerificationResult]

    def render(self) -> str:
        """Render the repair prompt as a string for the model."""
        sections = []

        # Remind the model of the original specification
        sections.append("## Original Specification")
        sections.append(self.original_spec)
        sections.append("")

        # Separate structural and behavioral failures
        structural = [f for f in self.failures if f.layer in ("ruff", "mypy")]
        behavioral = [f for f in self.failures if f.layer in ("pytest",)]

        if structural:
            sections.append("## Structural Issues (fix these first)")
            for f in structural:
                sections.append(f"### {f.layer}")
                sections.append(f.details)
                sections.append("")

        if behavioral:
            sections.append("## Behavioral Issues")
            for f in behavioral:
                sections.append(f"### {f.layer}")
                sections.append(f.details)
                sections.append("")

        # Constrain the repair scope
        sections.append("## Constraints")
        sections.append(f"- Only modify `{self.source_file}`")
        sections.append(f"- Do NOT modify `{self.test_file}`")
        sections.append("- Do NOT weaken any test assertions")
        sections.append("- All existing passing tests must continue to pass")

        return "\n".join(sections)

What does a complete repair cycle look like in practice? The developer runs the verification pipeline, receives a report showing that mypy found a type error on line 14 and pytest found one failing assertion in test_normalize. The RepairPrompt renders these into a structured message. The developer pastes (or the tool automatically feeds) this message into the next model turn, along with the current source file. The model reads the structured failures, identifies that the type error (returning str instead of float) is also the root cause of the test failure, and produces a corrected version. The developer runs the pipeline again. If all layers pass, the cycle is complete; if new failures appear, the cycle repeats. In practice, most single-function repairs converge within one to three iterations when the repair prompt follows the three principles above.

The scope constraint ("do not modify the test file") deserves emphasis. Without it, the model frequently "repairs" a bug by changing the test's expected value to match the buggy output. This is formally correct (all tests pass) but semantically wrong (the specification has been weakened). In the search framework, weakening tests is equivalent to expanding the solution space \(S^*\) rather than navigating to a better point within it. It feels like progress (the pass rate goes up) but represents regression.

5. Putting It Together: The Automated Verification Pipeline

The following class combines all verification layers into a single automated pipeline that can be invoked from the command line or integrated into a continuous integration / continuous deployment (CI/CD) workflow. This pipeline is the workhorse of the "observe" phase in the vibe coding loop.

import subprocess
import time
import json
from pathlib import Path
from dataclasses import dataclass, field

@dataclass
class VerificationPipeline:
    """
    Automated verification pipeline for vibe-coded projects.

    Runs all verification layers in sequence, collects results,
    and generates a repair prompt if any layer fails.
    """
    source_files: list[str]
    test_files: list[str]
    results: list[VerificationResult] = field(default_factory=list)

    def run_layer(
        self, name: str, cmd: list[str],
    ) -> VerificationResult:
        """Run a single verification layer and capture the result."""
        t0 = time.time()
        proc = subprocess.run(cmd, capture_output=True, text=True, timeout=120)
        elapsed = time.time() - t0
        return VerificationResult(
            layer=name,
            passed=proc.returncode == 0,
            details=(proc.stdout + proc.stderr)[-1500:],
            duration_seconds=elapsed,
        )

    def run_all(self) -> list[VerificationResult]:
        """Run the full verification stack."""
        self.results = []

        # Layer 1: Linting
        self.results.append(self.run_layer(
            "ruff", ["ruff", "check"] + self.source_files,
        ))

        # Layer 2: Type checking
        self.results.append(self.run_layer(
            "mypy", ["mypy", "--ignore-missing-imports"] + self.source_files,
        ))

        # Layer 3: Tests (unit + property-based)
        self.results.append(self.run_layer(
            "pytest",
            ["pytest"] + self.test_files + ["-v", "--tb=short", "-x"],
        ))

        return self.results

    @property
    def all_passed(self) -> bool:
        return all(r.passed for r in self.results)

    def summary(self) -> str:
        """One-line summary for logging."""
        total = len(self.results)
        passed = sum(1 for r in self.results if r.passed)
        elapsed = sum(r.duration_seconds for r in self.results)
        return f"{passed}/{total} layers passed in {elapsed:.1f}s"


# Usage in the vibe coding loop
pipeline = VerificationPipeline(
    source_files=["src/pipeline.py"],
    test_files=["tests/test_pipeline.py"],
)
results = pipeline.run_all()
print(pipeline.summary())

if not pipeline.all_passed:
    repair = RepairPrompt(
        original_spec="...",  # from Section 9.1
        source_file="src/pipeline.py",
        test_file="tests/test_pipeline.py",
        failures=[r for r in results if not r.passed],
    )
    print(repair.render())

6. Repair Anti-Patterns

Not all repairs move the session toward convergence. Three anti-patterns are common enough to warrant explicit warnings.

Test weakening occurs when the model changes the test to match the buggy output rather than fixing the code. Guard against this by making the test file read-only or by including a "do not modify tests" constraint in every repair prompt.

Overfitting to failures occurs when the model adds special-case logic to pass specific failing tests without addressing the underlying bug. The result is code that passes the current test suite but fails on any new input. Property-based tests are the primary defense: they generate new inputs on every run, making overfitting significantly harder because the model cannot anticipate which specific inputs Hypothesis will produce next.

Regression cascades occur when fixing one test breaks several others, triggering a chain of fixes and breaks that never stabilizes. This is the "oscillating pass rate" pathology from Section 9.2. The root cause is usually tight coupling between components. The fix is to decompose the function into smaller, independently testable units and repair each unit separately.

Exercises

1. (Conceptual) For each layer of the verification stack (static analysis, unit tests, property tests, end-to-end tests), give one example of a bug it would catch that the layers below it would miss. Then give one example of a bug that no layer in the stack would catch.
2. (Coding) Write a Hypothesis property test for a function parse_csv_row(row: str) -> list[str] that parses a single line of CSV. Your property tests should verify: (1) round-trip consistency (formatting the parsed result back to CSV produces the original input), (2) the number of fields is consistent for all rows of the same structure, and (3) quoted fields preserve commas and newlines. Use st.text() and st.lists() to generate test inputs.
3. (Analysis) Implement the VerificationPipeline and run it against a small Python project (your own or a tutorial project). Record the pass/fail results for each layer. Which layer catches the most issues? How many of the issues caught by static analysis would also be caught by the test suite?