18.3 Building an Invariant Discovery System

1. System Architecture

Production bugs in scientific software frequently trace back to properties that no one thought to test: a simulation that silently violates conservation of energy, an interpolation routine that returns plausible but wrong values near domain boundaries. Manually anticipating every such property is impractical once a codebase grows beyond a few functions, which is exactly the gap an automated invariant discovery system fills.

Imagine handing a function to a system that reads its source code, guesses every behavioral rule the function should obey, writes hundreds of tests for those rules, then deliberately breaks the code in dozens of places to confirm the tests would notice. That is what the invariant discovery system does, in four modular stages that follow the same architecture pattern used throughout the Discovery Workbench (introduced in Chapter 6), each with a well-defined input and output, communicating through structured data rather than shared state. The diagram below (Figure 18.3.1) shows how these four stages connect, with each stage's primary input and output labeled.

A behavioral invariant is a property that a function must satisfy for every valid input, not just for a handful of hand-picked test cases. Behavioral invariants transform testing from "check these five examples" into "prove this universal rule." A single property statement catches entire categories of bugs. The system expresses each invariant as a property test using Hypothesis (a Python library for property-based testing that verifies properties by generating hundreds of random inputs automatically) and confirms the property holds for all of them. Use behavioral invariants when a function has a large or continuous input domain where manually chosen examples cannot cover boundary conditions. Fall back to example-based tests when correct output depends on specific business rules that no general property can capture (such as "customer #42 gets a 10% discount"). In short: let the machine propose the universal laws your code must obey, then let mutations prove those laws have teeth.

1. Invariant Inference: A large language model (LLM) reads the function's source code, docstring, and type annotations, then proposes a set of behavioral invariants from the five families (roundtrip, idempotence, preservation, metamorphic, oracle comparison).
2. Test Generation: Each inferred invariant is translated into a Hypothesis property test with an appropriate custom strategy (a rule that tells Hypothesis how to generate valid random inputs) for the function's input types.
3. Test Validation: The generated tests are compiled, executed, and filtered. Tests that fail on the current (presumably correct) code are discarded as false invariants. Tests that pass are retained as candidate properties.
4. Mutation Analysis: mutmut (a Python mutation testing tool that injects small faults into source code) runs against the function with the validated property tests. The resulting mutation score (the fraction of injected faults the tests detect), combined with branch coverage (the fraction of conditional branches exercised by the tests) and property count, produces the composite confidence score.

"""
Invariant Discovery System: core data structures.
"""
from dataclasses import dataclass, field
from enum import Enum


class InvariantFamily(Enum):
    """The five families of behavioral invariants."""
    ROUNDTRIP = "roundtrip"
    IDEMPOTENCE = "idempotence"
    PRESERVATION = "preservation"
    METAMORPHIC = "metamorphic"
    ORACLE = "oracle_comparison"


@dataclass
class InferredInvariant:
    """A behavioral invariant inferred by the LLM."""
    family: InvariantFamily
    description: str          # human-readable statement
    hypothesis_code: str      # executable Hypothesis test code
    confidence: float         # LLM's self-assessed confidence (0 to 1)
    validated: bool = False   # True after passing on current code
    killed_mutants: int = 0   # how many mutants this property catches


@dataclass
class DiscoveryReport:
    """Output of the invariant discovery pipeline."""
    function_name: str
    source_file: str
    invariants_proposed: int
    invariants_validated: int
    branch_coverage: float         # 0 to 1
    mutation_score: float          # 0 to 1
    property_ratio: float          # validated / target
    confidence_score: float        # composite metric
    surviving_mutants: list[str]   # descriptions of unkilled mutants
    invariants: list[InferredInvariant] = field(default_factory=list)

    def summary(self) -> str:
        return (
            f"Invariant Discovery Report: {self.function_name}\n"
            f"  Invariants: {self.invariants_validated}/{self.invariants_proposed} validated\n"
            f"  Branch coverage: {self.branch_coverage:.1%}\n"
            f"  Mutation score: {self.mutation_score:.1%}\n"
            f"  Confidence score: {self.confidence_score:.1%}\n"
            f"  Surviving mutants: {len(self.surviving_mutants)}"
        )

2. Stage 1: Invariant Inference

The inference stage uses an LLM to read a function's source code and propose behavioral invariants. The prompt is structured to elicit properties from each of the five families, with the LLM returning structured JSON that can be parsed directly into InferredInvariant objects. The key design decision is asking the LLM to generate executable Hypothesis code for each invariant, not just a natural-language description. This eliminates a translation step and ensures the invariant is testable.

"""
Stage 1: Invariant Inference using an LLM.
"""
import json
from anthropic import Anthropic


INVARIANT_INFERENCE_PROMPT = """You are a software testing expert. Analyze the following
Python function and infer behavioral invariants it should satisfy.

For each invariant, provide:
1. family: one of "roundtrip", "idempotence", "preservation", "metamorphic", "oracle_comparison"
2. description: a one-sentence natural-language statement of the property
3. hypothesis_code: a complete, runnable Hypothesis property test (with imports)
4. confidence: your confidence that this property holds (0.0 to 1.0)

Guidelines:
- Propose at least 2 invariants from each applicable family.
- Include Hypothesis strategies appropriate for the function's input types.
- For scientific functions, consider physical constraints (e.g., conservation laws,
  dimensional consistency, monotonicity, symmetry).
- For data transformations, consider roundtrip properties with inverse operations.
- Be specific: "output length equals input length" is better than "output is correct."

Return a JSON array of objects. No other text.

Function to analyze:
```python
{function_source}
```"""


def infer_invariants(
    function_source: str,
    model: str = "claude-sonnet-4-20250514"
) -> list[InferredInvariant]:
    """Infer behavioral invariants from function source code.

    Args:
        function_source: Complete source of the function under test.
        model: Claude model to use.

    Returns:
        List of inferred invariants with executable test code.
    """
    client = Anthropic()

    response = client.messages.create(
        model=model,
        max_tokens=8192,
        messages=[{
            "role": "user",
            "content": INVARIANT_INFERENCE_PROMPT.format(
                function_source=function_source
            )
        }]
    )

    raw_text = response.content[0].text

    # Strip markdown fences if present
    if raw_text.startswith("```"):
        raw_text = raw_text.split("\n", 1)[1]
        raw_text = raw_text.rsplit("```", 1)[0]

    invariants_data = json.loads(raw_text)

    return [
        InferredInvariant(
            family=InvariantFamily(item["family"]),
            description=item["description"],
            hypothesis_code=item["hypothesis_code"],
            confidence=item["confidence"]
        )
        for item in invariants_data
    ]


# Example: infer invariants for a data transformation function
source = '''
def interpolate_spectrum(
    wavelengths: list[float],
    intensities: list[float],
    target_wavelengths: list[float]
) -> list[float]:
    """Linearly interpolate a spectrum onto a new wavelength grid.

    Args:
        wavelengths: Original wavelength values (sorted ascending, at least 2).
        intensities: Original intensity values (same length as wavelengths).
        target_wavelengths: Desired output wavelength values (sorted ascending).

    Returns:
        Interpolated intensities at each target wavelength.
        Values outside the original range are extrapolated linearly.
    """
    result = []
    for target in target_wavelengths:
        # Find bracketing interval
        if target <= wavelengths[0]:
            # Extrapolate below
            slope = (intensities[1] - intensities[0]) / (wavelengths[1] - wavelengths[0])
            result.append(intensities[0] + slope * (target - wavelengths[0]))
        elif target >= wavelengths[-1]:
            # Extrapolate above
            slope = (intensities[-1] - intensities[-2]) / (wavelengths[-1] - wavelengths[-2])
            result.append(intensities[-1] + slope * (target - wavelengths[-1]))
        else:
            # Interpolate: find the interval
            for i in range(len(wavelengths) - 1):
                if wavelengths[i] <= target <= wavelengths[i + 1]:
                    t = (target - wavelengths[i]) / (wavelengths[i + 1] - wavelengths[i])
                    val = intensities[i] + t * (intensities[i + 1] - intensities[i])
                    result.append(val)
                    break
    return result
'''

invariants = infer_invariants(source)
for inv in invariants:
    print(f"[{inv.family.value}] {inv.description} (confidence: {inv.confidence})")

3. Stage 2: Test Generation and Validation

Domain-specific invariants are powerful precisely because they encode deep knowledge about a function's purpose, but that same depth means the LLM can also hallucinate plausible properties that the function does not actually satisfy.

The inferred invariants arrive as strings of Python code. The system compiles each one, runs it against the current implementation, and discards any that fail. A failing invariant either reflects a hallucinated property or exposes a genuine bug. Consistent failures (every run) signal real violations; intermittent failures signal flawed strategies or unstable numerical comparisons, and those invariants are discarded as flaky (a test is "flaky" when it passes on some runs and fails on others without any code change, typically due to nondeterminism or tight numerical tolerances) .

"""
Stage 2: Test Generation and Validation.
Compile, execute, and filter inferred invariants.
"""
import subprocess
import tempfile
import textwrap
from pathlib import Path


def validate_invariants(
    invariants: list[InferredInvariant],
    source_file: str,
    max_examples: int = 200,
    validation_runs: int = 3
) -> list[InferredInvariant]:
    """Validate inferred invariants against the current implementation.

    Runs each invariant's Hypothesis test multiple times. Invariants that
    pass consistently are marked as validated. Invariants that fail
    consistently are flagged as potential bugs. Flaky invariants are
    discarded.

    Args:
        invariants: List of inferred invariants with hypothesis_code.
        source_file: Path to the module containing the function under test.
        max_examples: Hypothesis max_examples setting for validation.
        validation_runs: Number of times to run each test for flakiness check.

    Returns:
        Filtered list of validated invariants.
    """
    validated = []
    module_name = Path(source_file).stem

    for inv in invariants:
        # Build a standalone test file
        test_code = textwrap.dedent(f"""
            import sys
            sys.path.insert(0, "{Path(source_file).parent}")
            from {module_name} import *
            from hypothesis import given, settings, assume
            from hypothesis import strategies as st
            from hypothesis.strategies import composite
            import math
            import numpy as np

            settings.register_profile("validate",
                max_examples={max_examples}, derandomize=True)
            settings.load_profile("validate")

            {inv.hypothesis_code}
        """)

        # Write to a temp file and run with pytest
        pass_count = 0
        fail_count = 0

        with tempfile.NamedTemporaryFile(
            mode="w", suffix=".py", prefix="test_inv_",
            delete=False, dir=tempfile.gettempdir()
        ) as f:
            f.write(test_code)
            test_path = f.name

        for run in range(validation_runs):
            result = subprocess.run(
                ["pytest", test_path, "-x", "--tb=short", "-q"],
                capture_output=True, text=True, timeout=120
            )
            if result.returncode == 0:
                pass_count += 1
            else:
                fail_count += 1

        # Classify the result
        if pass_count == validation_runs:
            inv.validated = True
            validated.append(inv)
        elif fail_count == validation_runs:
            # Consistent failure: possible real bug
            print(f"  POTENTIAL BUG: {inv.description}")
            print(f"    Family: {inv.family.value}")
            # Still include it so the report can flag it
            inv.validated = False
            validated.append(inv)
        else:
            # Flaky: discard
            print(f"  FLAKY (discarded): {inv.description}")

        # Clean up
        Path(test_path).unlink(missing_ok=True)

    return validated

4. Stage 3: Mutation Analysis

With validated property tests in hand, we run mutation testing to measure their strength. This stage combines all validated tests into a single test file, runs mutmut against the function under test, and collects the mutation score along with details of each surviving mutant.

"""
Stage 3: Mutation Analysis.
Run mutmut with the validated property tests and collect results.
"""
import subprocess
import json
import re
from pathlib import Path


def run_mutation_analysis(
    source_file: str,
    validated_invariants: list[InferredInvariant],
    timeout_per_mutant: int = 30
) -> dict:
    """Run mutation testing with validated property tests.

    Args:
        source_file: Path to the module under test.
        validated_invariants: List of validated invariants (only those
            where validated=True will be used).
        timeout_per_mutant: Timeout in seconds per mutant test run.

    Returns:
        Dict with mutation_score, killed, survived, total, and
        surviving_mutant_descriptions.
    """
    # Collect only validated invariants
    valid_tests = [
        inv for inv in validated_invariants if inv.validated
    ]

    if not valid_tests:
        return {
            "mutation_score": 0.0,
            "killed": 0,
            "survived": 0,
            "total": 0,
            "surviving_mutants": []
        }

    # Build combined test file
    module_name = Path(source_file).stem
    test_dir = Path(source_file).parent / "tests"
    test_dir.mkdir(exist_ok=True)
    test_file = test_dir / f"test_{module_name}_properties.py"

    header = f"""
import sys
sys.path.insert(0, "{Path(source_file).parent}")
from {module_name} import *
from hypothesis import given, settings, assume
from hypothesis import strategies as st
from hypothesis.strategies import composite
import math
import numpy as np

settings.register_profile("mutation",
    max_examples=50, derandomize=True, deadline=None)
settings.load_profile("mutation")
"""

    body = "\n\n".join(inv.hypothesis_code for inv in valid_tests)
    test_file.write_text(header + "\n" + body)

    # Run mutmut
    result = subprocess.run(
        [
            "mutmut", "run",
            f"--paths-to-mutate={source_file}",
            f"--tests-dir={test_dir}",
            f"--runner=pytest {test_file} -x --tb=no -q",
        ],
        capture_output=True, text=True,
        timeout=timeout_per_mutant * 200  # generous overall timeout
    )

    # Parse results
    result_output = subprocess.run(
        ["mutmut", "results"],
        capture_output=True, text=True
    )

    # Extract counts from mutmut output
    output = result_output.stdout
    killed = len(re.findall(r"Killed", output))
    survived = len(re.findall(r"Survived", output))
    timeout_count = len(re.findall(r"Timeout", output))

    # Timed-out mutants are excluded: they may be "equivalent mutants"
    # (mutations that change the source without changing observable behavior),
    # so counting them as survived would unfairly penalize the score.
    total_non_equivalent = killed + survived
    mutation_score = (
        killed / total_non_equivalent
        if total_non_equivalent > 0 else 1.0
    )

    # Collect surviving mutant descriptions
    surviving_descriptions = []
    if survived > 0:
        for i in range(1, killed + survived + timeout_count + 1):
            show_result = subprocess.run(
                ["mutmut", "show", str(i)],
                capture_output=True, text=True
            )
            if "Survived" in show_result.stdout or survived > 0:
                surviving_descriptions.append(
                    show_result.stdout.strip()[:200]
                )

    return {
        "mutation_score": mutation_score,
        "killed": killed,
        "survived": survived,
        "total": killed + survived + timeout_count,
        "surviving_mutants": surviving_descriptions[:10]  # limit
    }

5. Stage 4: Confidence Scoring and Reporting

The final stage computes the composite confidence score and generates the DiscoveryReport. This is where all the metrics converge: branch coverage from coverage.py (a Python tool that measures which lines and conditional branches the tests actually execute), mutation score from mutmut, and property count from the validation stage.

"""
Stage 4: Confidence Scoring and Reporting.
Assemble the complete invariant discovery report.
"""
import subprocess
import json
from pathlib import Path


def compute_branch_coverage(
    source_file: str,
    test_file: str
) -> float:
    """Measure branch coverage of the property tests."""
    module_name = Path(source_file).stem
    subprocess.run(
        [
            "pytest", test_file,
            f"--cov={module_name}",
            "--cov-branch",
            "--cov-report=json:cov_report.json",
            "-q"
        ],
        capture_output=True, text=True,
        timeout=120
    )

    try:
        with open("cov_report.json") as f:
            cov_data = json.load(f)
        file_cov = cov_data.get("files", {}).get(source_file, {})
        return file_cov.get("summary", {}).get(
            "percent_covered_branches", 0.0
        ) / 100.0
    except (FileNotFoundError, json.JSONDecodeError, KeyError):
        return 0.0
    finally:
        Path("cov_report.json").unlink(missing_ok=True)


def run_invariant_discovery(
    source_file: str,
    function_source: str,
    target_properties: int = 10,
    model: str = "claude-sonnet-4-20250514"
) -> DiscoveryReport:
    """Run the complete invariant discovery pipeline.

    This is the main entry point that orchestrates all four stages.

    Args:
        source_file: Path to the Python source file.
        function_source: Source code of the function to analyze.
        target_properties: Target number of validated properties.
        model: Claude model for invariant inference.

    Returns:
        A complete DiscoveryReport with confidence score.
    """
    import re

    # Extract function name
    match = re.search(r"def (\w+)\(", function_source)
    func_name = match.group(1) if match else "unknown"

    print(f"Analyzing: {func_name}")

    # Stage 1: Infer invariants
    print("  Stage 1: Inferring invariants...")
    invariants = infer_invariants(function_source, model=model)
    print(f"    Proposed {len(invariants)} invariants")

    # Stage 2: Validate invariants
    print("  Stage 2: Validating invariants...")
    validated = validate_invariants(invariants, source_file)
    valid_count = sum(1 for inv in validated if inv.validated)
    print(f"    Validated {valid_count} / {len(invariants)}")

    # Stage 3: Mutation analysis
    print("  Stage 3: Running mutation analysis...")
    mutation_results = run_mutation_analysis(
        source_file,
        validated
    )
    print(f"    Mutation score: {mutation_results['mutation_score']:.1%}")

    # Stage 4: Compute confidence score
    test_dir = Path(source_file).parent / "tests"
    test_file = str(
        test_dir / f"test_{Path(source_file).stem}_properties.py"
    )
    branch_cov = compute_branch_coverage(source_file, test_file)

    property_ratio = min(1.0, valid_count / target_properties)

    # Multiplication (rather than, say, a weighted average) is deliberate:
    # if ANY single factor is near zero the overall score collapses,
    # reflecting the principle that high coverage with weak properties
    # (or strong properties with low coverage) is not trustworthy.
    confidence = (
        branch_cov
        * mutation_results["mutation_score"]
        * property_ratio
    )

    report = DiscoveryReport(
        function_name=func_name,
        source_file=source_file,
        invariants_proposed=len(invariants),
        invariants_validated=valid_count,
        branch_coverage=branch_cov,
        mutation_score=mutation_results["mutation_score"],
        property_ratio=property_ratio,
        confidence_score=confidence,
        surviving_mutants=mutation_results["surviving_mutants"],
        invariants=validated
    )

    print(f"\n{report.summary()}")
    return report

The Re-Inference Loop

The dashed feedback arrow in Figure 18.3.1 represents the most important part of the pipeline: when surviving mutants indicate weak property coverage, the system re-runs Stage 1 with an augmented prompt that includes the surviving mutant diffs. This gives the LLM concrete evidence of what the current properties miss, so it can propose targeted invariants that address those gaps. The function below implements one iteration of this loop.

def reinfer_for_survivors(
    function_source: str,
    surviving_mutants: list[str],
    existing_invariants: list[InferredInvariant],
    model: str = "claude-sonnet-4-20250514"
) -> list[InferredInvariant]:
    """Re-run invariant inference targeting surviving mutants.

    Args:
        function_source: Source code of the function under test.
        surviving_mutants: Diffs of mutants that the current tests
            failed to detect.
        existing_invariants: Already-validated invariants (to avoid
            duplicates).
        model: Claude model for inference.

    Returns:
        New invariants specifically targeting the surviving mutants.
    """
    existing_descriptions = [
        inv.description for inv in existing_invariants
    ]
    survivors_text = "\n---\n".join(surviving_mutants[:5])

    prompt = f"""You are a software testing expert. The following function
has surviving mutants that the current property tests do not detect.

Function:
```python
{function_source}
```

Surviving mutant diffs (each is a small code change that goes undetected):
{survivors_text}

Existing properties (do NOT repeat these):
{chr(10).join(f'- {d}' for d in existing_descriptions)}

Propose new behavioral invariants that would detect these specific mutations.
Return a JSON array (same schema as before)."""

    client = Anthropic()
    response = client.messages.create(
        model=model, max_tokens=4096,
        messages=[{"role": "user", "content": prompt}]
    )
    raw = response.content[0].text
    if raw.startswith("```"):
        raw = raw.split("\n", 1)[1].rsplit("```", 1)[0]

    import json
    return [
        InferredInvariant(
            family=InvariantFamily(item["family"]),
            description=item["description"],
            hypothesis_code=item["hypothesis_code"],
            confidence=item["confidence"]
        )
        for item in json.loads(raw)
    ]

6. Discovery Workbench Integration

A pipeline that produces high confidence scores is useful only if developers and automated workflows can invoke it without manual setup at every step.

The invariant discovery system integrates into the Discovery Workbench as a testing service that can be triggered from the experiment dashboard, the command-line interface (CLI), or the continuous integration (CI) pipeline. The integration follows the same service pattern established in Chapter 6 and extended through Chapter 12: Building MCP Servers.

"""
Discovery Workbench integration: the invariant discovery service.
Exposes the pipeline as a FastAPI endpoint and an MCP tool.
"""
from fastapi import FastAPI, BackgroundTasks
from pydantic import BaseModel, Field
from pathlib import Path
import asyncio

app = FastAPI(title="Invariant Discovery Service")


class DiscoveryRequest(BaseModel):
    """Request to run invariant discovery on a function."""
    source_file: str = Field(
        description="Path to the Python source file"
    )
    function_name: str = Field(
        description="Name of the function to analyze"
    )
    target_properties: int = Field(
        default=10,
        description="Target number of validated properties"
    )
    model: str = Field(
        default="claude-sonnet-4-20250514",
        description="Claude model for invariant inference"
    )


class DiscoveryResponse(BaseModel):
    """Response from the invariant discovery pipeline."""
    function_name: str
    invariants_proposed: int
    invariants_validated: int
    branch_coverage: float
    mutation_score: float
    confidence_score: float
    surviving_mutant_count: int
    status: str  # "complete", "running", "failed"


# In-memory store for async results (use a database in production)
_results: dict[str, DiscoveryResponse] = {}


async def _run_discovery_async(
    request_id: str,
    request: DiscoveryRequest
):
    """Run the discovery pipeline in the background."""
    try:
        source_path = Path(request.source_file)
        source_code = source_path.read_text()

        # Extract the target function's source
        import ast
        tree = ast.parse(source_code)
        func_source = None
        for node in ast.walk(tree):
            if (isinstance(node, ast.FunctionDef)
                    and node.name == request.function_name):
                func_source = ast.get_source_segment(
                    source_code, node
                )
                break

        if func_source is None:
            _results[request_id] = DiscoveryResponse(
                function_name=request.function_name,
                invariants_proposed=0,
                invariants_validated=0,
                branch_coverage=0.0,
                mutation_score=0.0,
                confidence_score=0.0,
                surviving_mutant_count=0,
                status="failed"
            )
            return

        report = run_invariant_discovery(
            source_file=request.source_file,
            function_source=func_source,
            target_properties=request.target_properties,
            model=request.model
        )

        _results[request_id] = DiscoveryResponse(
            function_name=report.function_name,
            invariants_proposed=report.invariants_proposed,
            invariants_validated=report.invariants_validated,
            branch_coverage=report.branch_coverage,
            mutation_score=report.mutation_score,
            confidence_score=report.confidence_score,
            surviving_mutant_count=len(report.surviving_mutants),
            status="complete"
        )
    except Exception as e:
        _results[request_id] = DiscoveryResponse(
            function_name=request.function_name,
            invariants_proposed=0,
            invariants_validated=0,
            branch_coverage=0.0,
            mutation_score=0.0,
            confidence_score=0.0,
            surviving_mutant_count=0,
            status=f"failed: {str(e)[:200]}"
        )


@app.post("/discover", response_model=dict)
async def start_discovery(
    request: DiscoveryRequest,
    background_tasks: BackgroundTasks
):
    """Start an invariant discovery analysis (runs in background)."""
    import uuid
    request_id = str(uuid.uuid4())
    _results[request_id] = DiscoveryResponse(
        function_name=request.function_name,
        invariants_proposed=0,
        invariants_validated=0,
        branch_coverage=0.0,
        mutation_score=0.0,
        confidence_score=0.0,
        surviving_mutant_count=0,
        status="running"
    )
    background_tasks.add_task(
        _run_discovery_async, request_id, request
    )
    return {"request_id": request_id, "status": "running"}


@app.get("/discover/{request_id}", response_model=DiscoveryResponse)
async def get_discovery_result(request_id: str):
    """Check the status of a discovery analysis."""
    if request_id not in _results:
        from fastapi import HTTPException
        raise HTTPException(status_code=404, detail="Not found")
    return _results[request_id]

7. The Testing-as-Discovery Feedback Loop

Whether invoked through a web endpoint, an MCP tool, or the ghostwriter shortcut, every run of the pipeline generates the same structured report, and that report carries a deeper significance than its numeric scores suggest.

The invariant discovery system creates a feedback loop that mirrors the scientific method. The LLM proposes hypotheses (invariants). The test framework designs experiments (Hypothesis examples). The mutation engine simulates faulty implementations (mutants). The confidence score quantifies how well the hypotheses explain the observed behavior. When the confidence score is low, the system can be re-run with an expanded prompt asking the LLM to target the specific mutant survival patterns, just as a scientist designs follow-up experiments to address gaps in their theory.

This feedback loop connects directly to the broader discovery themes of the book. The search framework from Chapter 1 treats discovery as navigating a search space. Here the search space is the set of behavioral invariants, and the objective is to find properties that are both true (validated) and strong (high mutation score). The hypothesis generation patterns from Chapter 39 apply directly: the LLM generates candidate hypotheses, and testing either confirms or refutes them. The evaluation metrics from Chapter 56 will revisit this confidence score as a component of overall system trustworthiness.