18.1 Test Generation Strategies

1. The Three Sources of Test Cases

A 200-test suite passes every night for months, then a production pipeline crashes on real data because no one thought to test a flat input signal where every value is identical. Where should that missing test have come from: the specification, the source code, or the failure itself? Each source produces tests with different strengths. Specification-based tests verify intent. Code-based tests achieve coverage. Failure-based tests prevent regressions. A comprehensive test suite draws from all three.

1.1 Specification-Based Test Generation

The simplest form of AI test generation starts with a function's contract: its name, type signature, docstring, and any formal preconditions or postconditions. The LLM reads the contract and generates test cases that exercise the specified behavior, including boundary conditions, error cases, and typical usage patterns. This approach mirrors how a human tester reads a requirements document and writes acceptance tests, but operates at the speed of token generation.

A function's specification encompasses every promise it makes to callers: input types and constraints (preconditions), output guarantees (postconditions), and declared exceptions. Specification-based test generation anchors tests to intended behavior rather than implementation details, so the tests survive refactoring. The LLM parses the signature, docstring, and type annotations, then samples inputs that exercise each documented behavior, boundary, and error path. Prefer specification-based generation whenever a function has a clear contract; switch to code-based generation (Section 1.2) when the specification is incomplete or you need to target uncovered branches. In short: tell the model what the function promises, and it will find the promises you forgot to keep.

"""
Specification-based test generation using an LLM.
We provide the function signature and docstring, then ask
the model to generate pytest test cases.
"""
import json
from anthropic import Anthropic

def generate_tests_from_spec(
    function_source: str,
    model: str = "claude-sonnet-4-20250514"
) -> str:
    """Generate pytest test cases from a function's specification.

    Args:
        function_source: The complete source code of the function,
            including its docstring and type annotations.
        model: The Claude model to use for generation.

    Returns:
        A string containing valid pytest test code.
    """
    client = Anthropic()

    prompt = f"""Analyze this Python function and generate a comprehensive
pytest test suite. Include:
1. Happy-path tests for typical inputs
2. Boundary tests (empty inputs, single elements, maximum sizes)
3. Error-handling tests (invalid types, out-of-range values)
4. Edge cases suggested by the function's logic

Return ONLY valid Python code, starting with imports.

Function:
```python
{function_source}
```"""

    response = client.messages.create(
        model=model,
        max_tokens=4096,
        messages=[{"role": "user", "content": prompt}]
    )
    return response.content[0].text


# Example: generate tests for a scientific data normalization function
target_function = '''
def normalize_spectrum(
    wavelengths: list[float],
    intensities: list[float],
    method: str = "min-max"
) -> list[float]:
    """Normalize spectral intensity values to [0, 1] range.

    Args:
        wavelengths: Wavelength values in nanometers (must be sorted).
        intensities: Raw intensity values (same length as wavelengths).
        method: Normalization method, one of "min-max", "z-score", "peak".

    Returns:
        Normalized intensity values.

    Raises:
        ValueError: If inputs have different lengths or are empty.
        ValueError: If method is not recognized.
    """
'''

generated_tests = generate_tests_from_spec(target_function)
print(generated_tests)

The generated tests typically include cases the developer might overlook: what happens when all intensities are identical (min equals max in min-max normalization), when wavelengths are not sorted despite the docstring requiring it, or when the intensity list contains negative values. The LLM appears to infer these edge cases from its training on large corpora of similar functions, applying what resembles analogical reasoning that Chapter 4: Reasoning for Discovery would recognize as case-based reasoning.

1.2 Code-Based Test Generation

Code-based generation analyzes the implementation itself: its branches, loops, exception handlers, and data transformations. The goal is to achieve high structural coverage by generating inputs that exercise each execution path. Where specification-based generation asks "what should this function do?", code-based generation asks "what does this function actually do, and have I tested all the paths through it?"

"""
Code-based test generation: analyze branches and generate
inputs that exercise each path.
"""
import ast
import textwrap

def extract_branches(source: str) -> list[dict]:
    """Extract conditional branches from Python source code.

    Returns a list of branch descriptors, each containing the
    condition expression and the line number.
    """
    tree = ast.parse(source)
    branches = []

    for node in ast.walk(tree):
        if isinstance(node, ast.If):
            # Extract the condition as source text
            condition = ast.unparse(node.test)
            branches.append({
                "type": "if",
                "condition": condition,
                "line": node.lineno,
                "has_else": len(node.orelse) > 0
            })
        elif isinstance(node, ast.For):
            branches.append({
                "type": "for",
                "iterable": ast.unparse(node.iter),
                "line": node.lineno
            })
        elif isinstance(node, ast.ExceptHandler):
            exc_type = ast.unparse(node.type) if node.type else "Exception"
            branches.append({
                "type": "except",
                "exception": exc_type,
                "line": node.lineno
            })

    return branches


def generate_branch_covering_tests(
    function_source: str,
    branches: list[dict],
    model: str = "claude-sonnet-4-20250514"
) -> str:
    """Generate tests that target specific uncovered branches."""
    client = Anthropic()

    branch_descriptions = "\n".join(
        f"  - Line {b['line']}: {b['type']} "
        f"({b.get('condition', b.get('exception', b.get('iterable', '')))})"
        for b in branches
    )

    prompt = f"""Given this Python function and its branch structure,
generate pytest tests that exercise EACH branch. For each if/else,
generate at least one test for the true branch and one for the false branch.
For each exception handler, generate a test that triggers it.

Function:
```python
{function_source}
```

Branches to cover:
{branch_descriptions}

Generate one clearly-named test function per branch. Include comments
stating which branch each test targets."""

    response = client.messages.create(
        model=model,
        max_tokens=4096,
        messages=[{"role": "user", "content": prompt}]
    )
    return response.content[0].text


# Example usage
source = textwrap.dedent('''
    def parse_measurement(raw: str) -> tuple[float, str]:
        """Parse a measurement string like '3.14 kg' into (value, unit)."""
        raw = raw.strip()
        if not raw:
            raise ValueError("Empty measurement string")

        parts = raw.split()
        if len(parts) == 1:
            # No unit specified, assume dimensionless
            try:
                return (float(parts[0]), "")
            except ValueError:
                raise ValueError(f"Cannot parse: {raw}")
        elif len(parts) == 2:
            try:
                value = float(parts[0])
            except ValueError:
                raise ValueError(f"Invalid number: {parts[0]}")
            return (value, parts[1].lower())
        else:
            raise ValueError(f"Too many parts in: {raw}")
''')

branches = extract_branches(source)
print(f"Found {len(branches)} branches to cover")
# Found 5 branches to cover

1.3 Failure-Based Test Generation

The third source of test cases is observed failures: bug reports, stack traces, error logs, and user complaints. When a bug is reported, the first step in a disciplined workflow is to write a test that reproduces it. This test should fail on the current code and pass after the fix, serving as both a verification and a regression guard. AI accelerates this process by reading a stack trace and generating a minimal reproducing test case.

"""
Failure-based test generation: convert a stack trace into a
minimal reproducing test case.
"""

def generate_regression_test(
    stack_trace: str,
    source_files: dict[str, str],
    model: str = "claude-sonnet-4-20250514"
) -> str:
    """Generate a pytest regression test from a stack trace.

    Args:
        stack_trace: The full Python stack trace from the failure.
        source_files: Dict mapping filenames to their source code
            for the files mentioned in the stack trace.

    Returns:
        A pytest test function that reproduces the failure.
    """
    client = Anthropic()

    files_context = "\n\n".join(
        f"# {name}\n```python\n{code}\n```"
        for name, code in source_files.items()
    )

    prompt = f"""A user reported this error. Generate a minimal pytest
test that reproduces the failure. The test should:
1. Set up the minimum state needed to trigger the bug
2. Call the failing function with the inputs that cause the crash
3. Use pytest.raises() to assert the specific exception
4. Include a comment explaining the root cause

Stack trace:
```
{stack_trace}
```

Relevant source files:
{files_context}

Generate ONLY the test function, with imports."""

    response = client.messages.create(
        model=model,
        max_tokens=2048,
        messages=[{"role": "user", "content": prompt}]
    )
    return response.content[0].text


# Example: a ZeroDivisionError in normalization
trace = """Traceback (most recent call last):
  File "pipeline.py", line 42, in process_batch
    normalized = normalize_spectrum(wl, intensities, method="min-max")
  File "spectrum.py", line 18, in normalize_spectrum
    return [(x - min_val) / (max_val - min_val) for x in intensities]
ZeroDivisionError: float division by zero"""

test_code = generate_regression_test(
    trace,
    {"spectrum.py": "... (source of normalize_spectrum)"}
)
print(test_code)
# Generates a test with all-equal intensities triggering max == min

2. Coverage Metrics: Measuring What Tests Actually Verify

Knowing where test cases come from, whether from specifications, code, or failures, raises an immediate follow-up question: once you have generated those tests, how do you know they are thorough enough?

Generating tests is necessary but insufficient; you must also measure what those tests verify. Coverage metrics quantify which code paths execute during testing, and the metric you choose directly affects which bugs you catch.

2.1 Statement Coverage

Statement coverage (also called line coverage) is the simplest metric: what fraction of the program's executable statements were executed by the test suite? If a function has 20 statements and the tests execute 18 of them, statement coverage is \(18/20 = 90\%\).

Statement coverage is easy to measure (the standard Python tool is coverage.py) and easy to understand, but it provides a false sense of security. A function with an if/else branch might have 100% statement coverage even though only one branch has been tested, if the other branch contains no unique statements.

2.2 Branch Coverage

Branch coverage measures whether every branch of every decision point has been exercised. For an if/else, branch coverage requires at least one test where the condition is true and one where it is false. For a loop, it requires at least one test that enters the loop and one that skips it (if the loop can be skipped).

"""
Measuring statement and branch coverage with coverage.py.
"""
# spectrum.py
def normalize_spectrum(wavelengths, intensities, method="min-max"):
    if len(wavelengths) != len(intensities):
        raise ValueError("Length mismatch")
    if not intensities:
        raise ValueError("Empty input")

    if method == "min-max":
        min_val = min(intensities)
        max_val = max(intensities)
        if max_val == min_val:
            return [0.5] * len(intensities)  # flat spectrum
        return [(x - min_val) / (max_val - min_val) for x in intensities]
    elif method == "z-score":
        mean_val = sum(intensities) / len(intensities)
        std_val = (
            sum((x - mean_val) ** 2 for x in intensities) / len(intensities)
        ) ** 0.5
        if std_val == 0:
            return [0.0] * len(intensities)
        return [(x - mean_val) / std_val for x in intensities]
    elif method == "peak":
        peak = max(abs(x) for x in intensities)
        if peak == 0:
            return [0.0] * len(intensities)
        return [x / peak for x in intensities]
    else:
        raise ValueError(f"Unknown method: {method}")

# Run tests with branch coverage measurement
pytest tests/test_spectrum.py --cov=spectrum --cov-branch --cov-report=term-missing

# Output:
# Name          Stmts   Miss Branch BrPart  Cover   Missing
# spectrum.py      28      0     14      0   100%

2.3 Condition and MC/DC Coverage

For safety-critical software (avionics, medical devices, autonomous vehicles), branch coverage is insufficient. Modified Condition/Decision Coverage (MC/DC), where each individual boolean condition must be shown to independently change the overall decision outcome, requires that every individual condition in a compound decision independently affects the outcome. Consider a decision with two conditions: if temperature > 100 and pressure > 50. Branch coverage requires only two tests (both true, at least one false). MC/DC requires that each condition's effect be demonstrated independently, typically needing at least \(n + 1\) test cases for \(n\) conditions. (For a safety interlock with 10 sensor checks, that means branch coverage needs just 2 tests while MC/DC demands 11, each isolating one sensor's influence on the outcome.)

"""
MC/DC test case generation for compound conditions.
Each condition must independently affect the decision.
"""

def generate_mcdc_tests(conditions: list[str], decision: str) -> list[dict]:
    """Generate MC/DC test vectors for a compound decision.

    For n conditions, produces n+1 test cases where each condition
    independently affects the decision outcome.

    Args:
        conditions: List of condition variable names, e.g. ["temp_high", "press_high"]
        decision: Boolean expression combining the conditions, e.g. "temp_high and press_high"

    Returns:
        List of test case dicts mapping condition names to bool values,
        plus a "decision" key with the expected outcome.
    """
    n = len(conditions)
    test_cases = []

    # Start with all-true case
    base = {c: True for c in conditions}
    base["decision"] = eval(decision, {}, base.copy())
    test_cases.append(base)

    # For each condition, flip it while keeping others at base values
    for i, cond in enumerate(conditions):
        flipped = base.copy()
        flipped[cond] = not base[cond]
        flipped["decision"] = eval(decision, {}, {
            c: flipped[c] for c in conditions
        })
        # Only include if the decision outcome actually changes
        if flipped["decision"] != base["decision"]:
            test_cases.append(flipped)
        else:
            # Need a different base for this condition; try all-false
            alt_base = {c: False for c in conditions}
            alt_base[cond] = True
            alt_base["decision"] = eval(decision, {}, {
                c: alt_base[c] for c in conditions
            })
            alt_flipped = alt_base.copy()
            alt_flipped[cond] = False
            alt_flipped["decision"] = eval(decision, {}, {
                c: alt_flipped[c] for c in conditions
            })
            if alt_base["decision"] != alt_flipped["decision"]:
                test_cases.extend([alt_base, alt_flipped])

    return test_cases


# Example: safety interlock for a robotic lab system
conditions = ["temp_safe", "pressure_safe", "door_closed"]
decision = "temp_safe and pressure_safe and door_closed"
vectors = generate_mcdc_tests(conditions, decision)

for v in vectors:
    vals = {k: v for k, v in v.items() if k != "decision"}
    print(f"  {vals} -> decision={v['decision']}")
# {temp_safe: True, pressure_safe: True, door_closed: True}  -> True
# {temp_safe: False, pressure_safe: True, door_closed: True}  -> False
# {temp_safe: True, pressure_safe: False, door_closed: True}  -> False
# {temp_safe: True, pressure_safe: True, door_closed: False}  -> False

3. End-to-End Test Generation with Playwright

Coverage metrics tell you whether your unit and integration tests exercise the code thoroughly, but they say nothing about whether the pieces work together from a user's perspective.

Unit and integration tests verify components in isolation. End-to-end (E2E) tests verify the complete system as a user experiences it: clicking buttons, filling forms, waiting for responses, and checking that the right content appears on screen. For Discovery Workbench components (introduced in Chapter 6), E2E tests ensure that the API, database, and UI work together correctly.

Recording and Refining E2E Tests

Playwright provides a codegen mode that records user interactions and generates test scripts automatically. Combined with an LLM, we can transform recorded interactions into robust, maintainable test suites with proper assertions and error handling.

"""
End-to-end test for the Discovery Workbench experiment dashboard.
Generated by Playwright codegen, then refined with LLM assistance
to add proper assertions and data-testid selectors.
"""
import pytest
from playwright.sync_api import Page, expect


@pytest.fixture
def dashboard_page(page: Page) -> Page:
    """Navigate to the experiment dashboard and wait for data load."""
    page.goto("http://localhost:8000/dashboard")
    # Wait for the experiment table to populate
    page.wait_for_selector("[data-testid='experiment-table'] tbody tr")
    return page


def test_experiment_list_loads(dashboard_page: Page):
    """Verify that the experiment list shows at least one experiment."""
    rows = dashboard_page.locator(
        "[data-testid='experiment-table'] tbody tr"
    )
    expect(rows).to_have_count(count=1, timeout=5000)  # at least 1


def test_experiment_filter_by_status(dashboard_page: Page):
    """Verify filtering experiments by status updates the table."""
    # Count initial rows
    initial_count = dashboard_page.locator(
        "[data-testid='experiment-table'] tbody tr"
    ).count()

    # Apply "completed" filter
    dashboard_page.select_option(
        "[data-testid='status-filter']", "completed"
    )

    # Wait for table to update
    dashboard_page.wait_for_timeout(500)

    # Filtered count should be <= initial count
    filtered_count = dashboard_page.locator(
        "[data-testid='experiment-table'] tbody tr"
    ).count()
    assert filtered_count <= initial_count

    # Every visible row should show "completed" status
    status_cells = dashboard_page.locator(
        "[data-testid='experiment-table'] tbody tr "
        "td[data-testid='status-cell']"
    )
    for i in range(status_cells.count()):
        expect(status_cells.nth(i)).to_have_text("completed")


def test_experiment_detail_navigation(dashboard_page: Page):
    """Verify clicking an experiment opens its detail view."""
    # Click the first experiment's name link
    first_name = dashboard_page.locator(
        "[data-testid='experiment-table'] tbody tr:first-child "
        "a[data-testid='experiment-link']"
    )
    experiment_name = first_name.text_content()
    first_name.click()

    # Should navigate to detail page
    expect(dashboard_page).to_have_url(
        pattern=r".*/experiments/\d+"
    )

    # Detail page should show the experiment name
    heading = dashboard_page.locator("h1[data-testid='experiment-title']")
    expect(heading).to_have_text(experiment_name)


def test_create_experiment_form_validation(dashboard_page: Page):
    """Verify that the create-experiment form validates required fields."""
    dashboard_page.click("[data-testid='create-experiment-btn']")

    # Submit without filling required fields
    dashboard_page.click("[data-testid='submit-experiment-btn']")

    # Should show validation error
    error = dashboard_page.locator("[data-testid='validation-error']")
    expect(error).to_be_visible()
    expect(error).to_contain_text("required")

4. The Coverage-Guided Generation Loop

E2E tests confirm that the full system behaves correctly for specific user scenarios, but they cannot systematically probe every code path the way unit tests can; the real power emerges when you combine both levels of testing inside an iterative feedback loop.

Teams that skip structured test generation pay the price in production: post-mortem analyses across scientific computing projects suggest that many first-day-of-production crashes trace back to input classes no developer thought to test. The coverage-guided loop described next exists because ad-hoc test writing, no matter how diligent, systematically misses the inputs that matter most.

The most effective AI test generation strategy combines all three sources in a coverage-guided loop. Start with specification-based tests for the contract. Measure coverage. Feed the uncovered lines and branches to the LLM for code-based generation. Measure again. If coverage plateaus, use failure-based generation from fuzzing or mutation testing (covered in Section 18.2) to find inputs that exercise stubborn paths. Figure 18.1 illustrates this iterative process. Figure 18.1.1 illustrates the coverage-guided test generation loop.

"""
Coverage-guided test generation loop.
Iteratively generates tests until a target branch coverage is reached.
"""
import subprocess
import json
from pathlib import Path


def get_coverage_gaps(
    test_file: str,
    source_file: str
) -> dict:
    """Run tests and return uncovered lines and branches."""
    result = subprocess.run(
        [
            "pytest", test_file,
            f"--cov={Path(source_file).stem}",
            "--cov-branch",
            "--cov-report=json:cov.json",
            "--quiet"
        ],
        capture_output=True, text=True
    )

    with open("cov.json") as f:
        cov_data = json.load(f)

    file_cov = cov_data["files"].get(source_file, {})
    return {
        "missing_lines": file_cov.get("missing_lines", []),
        "missing_branches": file_cov.get("missing_branches", []),
        "summary": {
            "line_rate": file_cov.get("summary", {}).get("percent_covered", 0),
            "branch_rate": file_cov.get("summary", {}).get(
                "percent_covered_branches", 0
            ),
        }
    }


def coverage_guided_generation(
    source_file: str,
    test_file: str,
    target_coverage: float = 95.0,
    max_iterations: int = 5
) -> dict:
    """Iteratively generate tests until target coverage is met.

    Returns:
        Final coverage statistics.
    """
    source_code = Path(source_file).read_text()

    for iteration in range(max_iterations):
        gaps = get_coverage_gaps(test_file, source_file)
        current = gaps["summary"]["branch_rate"]
        print(f"Iteration {iteration + 1}: branch coverage = {current:.1f}%")

        if current >= target_coverage:
            print("Target coverage reached!")
            return gaps["summary"]

        # Generate tests targeting uncovered branches
        new_tests = generate_branch_covering_tests(
            function_source=source_code,
            branches=[
                {"type": "branch", "line": line, "condition": "uncovered"}
                for line in gaps["missing_lines"][:10]  # limit context
            ]
        )

        # Append to test file
        with open(test_file, "a") as f:
            f.write(f"\n\n# Generated in iteration {iteration + 1}\n")
            f.write(new_tests)

    return gaps["summary"]

5. Test Quality Beyond Coverage

Coverage-guided generation can achieve impressive coverage numbers, but coverage is necessary, not sufficient, for test quality. A test suite with 100% branch coverage but weak assertions catches no bugs. Two complementary techniques measure test strength: property-based testing (where you declare invariants that must hold for all inputs, and a framework automatically generates hundreds of random inputs to try to violate them) generates inputs that explore the full input space, and mutation testing measures whether assertions detect injected faults. Section 18.2 covers both.

def merge_intervals(intervals: list[tuple[int, int]]) -> list[tuple[int, int]]:
    """Merge overlapping intervals and return sorted non-overlapping intervals.

    Raises ValueError if any interval has start > end.
    """