Overview
Testing is the empirical science of software engineering. Just as a physicist designs experiments to falsify hypotheses about nature, a software engineer designs tests to falsify hypotheses about code. A passing test suite does not prove correctness; it proves that you have not yet found a bug with the inputs you tried. The challenge, then, is not writing tests (that is largely mechanical) but choosing what to test: which inputs, which properties, which interaction paths will most efficiently reveal the defects hiding in your code.
This chapter shows how AI transforms every layer of the testing stack. At the unit level, large language models generate test cases from function signatures, docstrings, and existing code. At the property level, AI infers behavioral invariants that hold across all inputs, then feeds those invariants to property-based testing frameworks like Hypothesis. At the mutation level, tools like mutmut inject faults into your code and measure whether your test suite catches them, producing a mutation score that is a far more honest measure of test quality than line coverage. At the end-to-end level, AI-driven tools like Playwright record and replay user workflows, generating regression suites from observed behavior.
The central idea unifying these techniques is invariant discovery: using AI to find properties your software should always satisfy, then verifying those properties exhaustively. This is testing as discovery, and it connects directly to the search framework of Chapter 1 and the verification techniques introduced in Chapter 9: Vibe Coding. The test generation strategies here also complement the debugging approaches in Chapter 19: AI-Assisted Debugging, where failing tests become the starting point for root-cause analysis.
Prerequisites
Readers should have completed Chapter 9: Vibe Coding (which introduced property-based testing with Hypothesis and executable contracts) and Chapter 17: Multi-Agent Software Teams (which established patterns for orchestrating multiple AI agents on a shared codebase). Familiarity with pytest, basic Python type annotations, and command-line test runners is assumed. The coverage metrics in this chapter use light probability and set theory notation; Chapter 1 provides sufficient mathematical background.
Learning Outcomes
- Generate unit, integration, and end-to-end tests from specifications, source code, and failure reports using LLM-based test generators.
- Apply property-based testing with Hypothesis to discover edge cases that example-based tests miss.
- Use mutation testing with mutmut to measure real test suite effectiveness beyond line coverage.
- Distinguish statement, branch, condition, and MC/DC coverage and choose the right metric for each risk level.
- Build an invariant discovery system that infers behavioral properties from code, generates property tests, and measures mutation scores.
- Integrate AI-assisted testing into CI/CD pipelines for continuous quality assurance.
Sections
18.1 Test Generation Strategies
Unit, integration, and end-to-end test generation from specifications, code, and failures. LLM-based test synthesis. Coverage metrics: statement, branch, condition, and MC/DC. Tools: pytest, Playwright. Recipe: generating a comprehensive test suite from a function signature and docstring.
18.2 Property-Based Testing and Mutation
Property-based testing with Hypothesis. Mutation testing with mutmut. Mutation score as a quality metric. Combining property tests with mutation analysis. The oracle problem and how properties solve it.
18.3 Building an Invariant Discovery System
Recipe: an AI system that reads source code, infers behavioral invariants, generates Hypothesis property tests, runs mutation testing, and reports a mutation-adjusted confidence score. Discovery Workbench integration.
Bibliography
Foundational Papers
Combines search-based test generation with LLM-generated tests to escape coverage plateaus, demonstrating that AI and traditional techniques are complementary.
Systematic evaluation of GPT-based unit test generation across open-source projects, measuring compilation rate, correctness, and coverage improvements.
The definitive survey of mutation testing theory and practice, establishing the competent programmer hypothesis and coupling effect as theoretical foundations.
Tools and Libraries
The leading property-based testing library for Python, generating test inputs from strategies to find minimal counterexamples. Central to Sections 18.2 and 18.3.
A mutation testing tool that introduces small faults (mutants) into Python source and checks whether the test suite kills them. Used extensively in Section 18.2.
The standard Python testing framework, providing fixtures, parametrization, and plugin architecture used throughout this chapter.
Cross-browser end-to-end testing framework with auto-waiting, network interception, and codegen for recording user interactions as test scripts.
The standard Python coverage tool measuring statement and branch coverage, with HTML reporting and CI integration.
Research Frontiers
Shows that feeding coverage information back to LLMs during test generation produces tests that reach uncovered branches more effectively than blind generation.
Demonstrates that LLMs can generate effective fuzz tests for complex APIs without examples, finding real bugs in TensorFlow and PyTorch.
Shows that conversational repair loops (generate test, observe failure, fix) achieve high bug-fix rates at minimal cost, motivating the test-driven repair approach in Section 18.3.
An iterative system that uses coverage gaps to guide LLM test generation, achieving near-complete branch coverage on real-world Python projects.