9.1 Natural Language as Specification

1. From Formal Specifications to Natural Language

Computer scientists have spent over fifty years building formal specification languages, where Z notation, VDM, TLA+, and Alloy are mathematical frameworks that describe a program's required behavior with logical precision, leaving no room for interpretation, that pin down every behavior of a program before a single line of code exists. Yet almost no working software uses them. These languages eliminate ambiguity but sacrifice accessibility: most working programmers never learn them. The practical reality is that requirements travel in natural language (English, Mandarin, Hebrew, whatever the team speaks), supplemented by examples, diagrams, and conversation.

The cost of ignoring that gap is measured in silent failures: a genomics pipeline drops 12% of its samples because the prompt never mentioned missing values, or a financial report double-counts transactions because "aggregate the data" left the deduplication rule unstated. Understanding specification completeness is what separates a productive vibe coding session from an expensive debugging session.

Vibe coding embraces this reality. Instead of fighting natural language's ambiguity, it treats natural-language prompts as the primary specification medium and uses other artifacts (tests, types, schemas) to compensate for what natural language cannot express precisely. This is not a retreat from rigor; it is a reallocation of rigor. The human focuses on what and why; the machine focuses on how; and executable contracts verify that the how actually satisfies the what. In short: a prompt is not a wish; it is a contract draft, and every clause you omit is a clause the model invents.

To see why this reallocation matters, consider the formal specification of a sorting function in Z notation versus a natural-language prompt:

# Z-notation-style formal spec (pseudocode rendering):
# sort: seq N -> seq N
# pre:  True
# post: output is a permutation of input
#       AND for all i in 1..len(output)-1: output[i-1] <= output[i]

# Natural-language prompt:
# "Write a function that sorts a list of numbers."

The formal specification captures two properties that the prompt leaves implicit: the output must be a permutation of the input (no elements added or lost), and the ordering must be non-decreasing. A code generator receiving only the prompt might produce a function that returns a sorted list of unique elements, discarding duplicates. That function "sorts a list of numbers" in one reading of the prompt, yet violates the permutation property. The gap between the prompt and the formal spec is where bugs are born. Figure 9.1 illustrates this relationship: the set of prompt-consistent programs is far larger than the set of intended programs, and the difference between them is the specification gap.

2. Anatomy of a Specification Gap

The specification gap has four distinct components, each requiring a different strategy to close.

The specification gap is the set of behaviors a program could exhibit that match what you asked for but contradict what you actually intended. Every unspecified behavior becomes a degree of freedom. The code generator fills that freedom with statistical patterns from its training data, not your domain knowledge. Natural language underdetermines program behavior. The generator picks one completion from many plausible ones, and the alternatives remain invisible until they surface as bugs. Use explicit specification (structured prompts, tests, schemas) whenever the cost of a wrong default exceeds the cost of writing the constraint; reserve terse prompts for throwaway scripts where any reasonable default is acceptable.

Ambiguity occurs when a prompt has multiple valid interpretations. "Parse the date" could mean parsing "2025-01-15", "January 15, 2025", "15/01/2025", or all three formats. The model must choose one interpretation, and it may choose differently from what you intended. Ambiguity is closed by providing examples or by specifying the exact format.

Underspecification occurs when the prompt says nothing about important behaviors. "Build a Representational State Transfer (REST) application programming interface (API) for users" says nothing about authentication, pagination, error responses, rate limiting, or database schema. The model fills these gaps with its training distribution, which may or may not match your requirements. Underspecification is closed by adding constraints, either in the prompt or in the test suite. Note that the model fills these gaps using its training distribution (the statistical patterns learned from its training corpus), which reflects common coding conventions but not your specific requirements.

Hidden Gaps: What You Do Not Know You Left Out

Implicit assumptions are requirements that the prompter considers obvious but never states. "Sort the list" implicitly assumes stability (equal elements maintain their original order) to many programmers, but not to all, and not to all models. Implicit assumptions are the hardest gap to close because the prompter does not know they are making them. Property-based testing (Section 9.3) excels at surfacing implicit assumptions by generating inputs the prompter never considered.

Contextual knowledge refers to domain-specific constraints that the prompt does not mention because the prompter assumes shared context. "Calculate the Body Mass Index (BMI)" assumes the model knows the formula (\(\text{BMI} = \text{weight} / \text{height}^2\) with weight in kilograms and height in meters). "Normalize the spectra" assumes the model knows which normalization method is standard in spectroscopy. Contextual knowledge gaps are closed by providing domain context in the prompt or in attached documentation, a technique we formalize in Chapter 11: Context Engineering.

We can model the specification gap mathematically. Let \(S_{\text{intended}}\) be the set of all programs that satisfy the programmer's full intent, and let \(S_{\text{prompt}}\) be the set of all programs consistent with the prompt. The specification gap is the difference:

Mental Model

Think of a specification gap like ordering furniture by phone. You say "a medium-sized wooden bookshelf," and the carpenter builds one. But you never mentioned the number of shelves, the wood species, the finish color, whether it should be wall-mounted or freestanding, or whether "medium" means waist-height or ceiling-height. The carpenter fills in every unspoken detail using their own defaults, which may differ wildly from what you pictured. Each detail you leave out is a dimension where the delivered product can diverge from your intent. Writing a structured prompt is like sending the carpenter a dimensioned sketch with material callouts: it does not guarantee a perfect result, but it eliminates the mismatches that stem from unshared assumptions.

A perfect prompt would make \(S_{\text{prompt}} = S_{\text{intended}}\), yielding \(G = \emptyset\). In practice, \(S_{\text{prompt}}\) is nearly always larger than \(S_{\text{intended}}\) because natural language rarely constrains program behavior completely. The goal of verification is to reject programs in \(G\) before they reach production.

import pytest
import tempfile
import os

def test_sample_std_not_population():
    """Standard deviation must use N-1 (sample), not N (population)."""
    data = "measurement\n2.0\n4.0\n6.0"
    with tempfile.NamedTemporaryFile(mode="w", suffix=".csv", delete=False) as f:
        f.write(data)
        f.flush()
        result = summarize_measurements(f.name)
    os.unlink(f.name)
    assert abs(result["std"] - 2.0) < 1e-10  # sample std of [2, 4, 6] is 2.0

def test_missing_values_are_dropped():
    """Rows with missing measurements are excluded, not imputed."""
    data = "measurement\n1.0\n\n3.0"
    with tempfile.NamedTemporaryFile(mode="w", suffix=".csv", delete=False) as f:
        f.write(data)
        f.flush()
        result = summarize_measurements(f.name)
    os.unlink(f.name)
    assert result["mean"] == 2.0  # mean of [1.0, 3.0], not [1.0, 0.0, 3.0]

def test_column_selection():
    """Function reads from the 'measurement' column specifically."""
    data = "id,measurement,noise\n1,10.0,999\n2,20.0,999"
    with tempfile.NamedTemporaryFile(mode="w", suffix=".csv", delete=False) as f:
        f.write(data)
        f.flush()
        result = summarize_measurements(f.name)
    os.unlink(f.name)
    assert result["mean"] == 15.0

3. Measuring Specification Completeness

If the specification gap is the problem, we need a way to measure how complete a specification is. Formal methods offer one approach: count the number of underdetermined behaviors relative to the total interface surface. But in vibe coding we can use a more practical proxy: behavioral coverage, where behavioral coverage is the fraction of a function's possible input/output/error scenarios that the specification explicitly addresses.

Define the behavioral surface \(B\) of a function as the set of distinct behaviors it can exhibit: input categories, output categories, error conditions, and state transitions. A specification (prompt plus tests) covers a behavior \(b \in B\) if it determines what the function should do in that case. The specification completeness is then:

In practice, we cannot enumerate \(B\) exactly, but we can approximate it. One useful heuristic is to list all input categories (valid inputs, boundary cases, invalid inputs, empty inputs, very large inputs) and check whether the specification (prompt or test) addresses each one. The following function automates a simple version of this analysis:

from dataclasses import dataclass, field

@dataclass
class SpecCompleteness:
    """Tracks which behavioral categories a specification covers."""
    categories: dict[str, bool] = field(default_factory=dict)

    def cover(self, category: str) -> None:
        self.categories[category] = True

    def miss(self, category: str) -> None:
        self.categories[category] = False

    @property
    def score(self) -> float:
        if not self.categories:
            return 0.0
        covered = sum(1 for v in self.categories.values() if v)
        return covered / len(self.categories)

    def report(self) -> str:
        lines = [f"Specification completeness: {self.score:.0%}"]
        for cat, covered in sorted(self.categories.items()):
            status = "COVERED" if covered else "MISSING"
            lines.append(f"  [{status}] {cat}")
        return "\n".join(lines)


# Example: analyzing the CSV summarizer specification
spec = SpecCompleteness()
spec.cover("valid input with single column")
spec.cover("missing values handling")
spec.cover("std deviation variant (sample vs population)")
spec.cover("column selection")
spec.miss("empty file")
spec.miss("non-numeric values")
spec.miss("file encoding")
spec.miss("very large files (memory)")
spec.miss("outlier handling")

print(spec.report())
# Output:
# Specification completeness: 44%
#   [COVERED] column selection
#   [MISSING] empty file
#   [MISSING] file encoding
#   [COVERED] missing values handling
#   [MISSING] non-numeric values
#   [MISSING] outlier handling
#   [COVERED] std deviation variant (sample vs population)
#   [COVERED] valid input with single column
#   [MISSING] very large files (memory)

In our experience, a completeness score below 60% typically signals that you should write more tests before asking the model to generate code. Above 80%, diminishing returns tend to set in for most applications, though critical systems may warrant higher coverage. The exact thresholds depend on the criticality of the code: a scientific data pipeline that feeds into a published paper demands higher completeness than a one-off data visualization script.

from pydantic import BaseModel, Field

class MeasurementSummary(BaseModel):
    """Executable contract for the summarize_measurements output."""
    mean: float = Field(description="Arithmetic mean of valid measurements")
    std: float = Field(ge=0, description="Sample standard deviation (N-1)")
    count: int = Field(gt=0, description="Number of valid measurements")
    missing_count: int = Field(ge=0, description="Number of dropped rows")

# Any function that returns a MeasurementSummary is guaranteed to satisfy
# these constraints. Invalid data raises ValidationError automatically.
result = MeasurementSummary(mean=15.0, std=2.0, count=8, missing_count=1)
print(result.model_dump())
# {'mean': 15.0, 'std': 2.0, 'count': 8, 'missing_count': 1}

4. Prompt Structure for Specification

Measuring incompleteness raises the practical question: how do you write prompts that start with higher completeness?

Not all prompts are equally effective as specifications. Research on prompt engineering for code generation (Jiang et al., 2024; Ridnik et al., 2024) consistently shows that structured prompts produce better code than unstructured ones. The following template captures the essential components of a specification-grade prompt:

SPEC_PROMPT_TEMPLATE = """
## Task
{one_sentence_description}

## Inputs
{input_schema_with_types_and_constraints}

## Outputs
{output_schema_with_types_and_constraints}

## Behavior
{numbered_list_of_behavioral_requirements}

## Edge Cases
{numbered_list_of_edge_cases_and_expected_behavior}

## Constraints
{performance_security_compatibility_constraints}

## Examples
{input_output_examples}
"""

Let us apply this template to our CSV summarizer:

SUMMARIZER_SPEC = """
## Task
Read a CSV file of experimental measurements and return summary statistics.

## Inputs
- file_path: str, path to a UTF-8 encoded CSV file
- The CSV must have a header row containing a 'measurement' column
- The 'measurement' column contains float values or empty strings (missing)

## Outputs
- A dictionary with keys: 'mean' (float), 'std' (float), 'count' (int),
  'missing_count' (int)
- 'std' is the sample standard deviation (ddof=1, where ddof is the
  delta degrees of freedom: N-1 corrects for bias when estimating
  population variance from a sample)

## Behavior
1. Read only the 'measurement' column; ignore all other columns.
2. Drop rows where 'measurement' is empty or non-numeric.
3. Compute mean and sample standard deviation of remaining values.
4. Return count of valid measurements and count of dropped rows.

## Edge Cases
1. Empty file (no data rows): raise ValueError("No measurements found")
2. All values missing: raise ValueError("No valid measurements")
3. Single valid value: std should be 0.0 (or NaN; we choose 0.0)

## Constraints
- Must handle files up to 1 GB without loading entire file into memory.
- No dependencies beyond the Python standard library and numpy.

## Examples
Input CSV:
  measurement
  2.0
  4.0

  6.0

Output: {'mean': 4.0, 'std': 2.0, 'count': 3, 'missing_count': 1}
"""

This prompt is roughly 200 words, compared to the original 15-word prompt. A 13x increase in prompt length, yet those additional 185 words take about five minutes to write and eliminate multiple rounds of generate-observe-repair that an underspecified prompt would require.

5. From Specification to Contract

A well-structured prompt narrows the specification gap, but even the best prompt remains a suggestion. The model may ignore clauses, misinterpret constraints, or satisfy the letter while violating the spirit. The fix is to translate specifications into executable contracts (type checks, unit tests, property tests) that run automatically and reject non-conforming code.

There are three levels of executable contract, each catching different classes of specification violations:

Type contracts enforce structural correctness. Python type annotations, checked by mypy or pyright (static analysis tools that verify type annotations without running the code) at development time, ensure that functions accept and return the right types. Pydantic models extend this to runtime validation with value constraints. Type contracts catch an estimated 20-40% of specification violations in practice, based on empirical studies of type-error frequency in Python codebases (Ore et al., 2018; Khan et al., 2021), though the exact figure varies by codebase and coding style.

Example contracts (unit tests) verify specific input-output pairs. Each test encodes one behavioral decision from the specification. Example contracts catch violations that match the tested scenarios but miss behaviors that no test covers. The coverage depends directly on the number and diversity of test cases.

Property contracts (property-based tests) verify invariants (properties that must remain true regardless of input) that hold across all valid inputs. "The output list is always a permutation of the input list" is a property contract. Property contracts catch violations that example contracts miss because they test randomly generated inputs, including edge cases the programmer never imagined. We build property contracts with Hypothesis, a Python library for property-based testing that automatically generates diverse test inputs, in Section 9.3.

from typing import Any

def specification_to_contracts(spec: dict[str, Any]) -> str:
    """
    Generate a pytest test file skeleton from a structured specification.

    This is a simplified version of what tools like Claude Code do internally
    when you ask them to "write tests for this spec."
    """
    lines = [
        "import pytest",
        f"from {spec['module']} import {spec['function']}",
        "",
    ]

    # Generate one test per behavioral requirement
    for i, req in enumerate(spec.get("behaviors", []), 1):
        lines.append(f"def test_behavior_{i}():")
        lines.append(f'    """Verify: {req["description"]}"""')
        lines.append(f"    result = {spec['function']}({req['input_repr']})")
        lines.append(f"    assert {req['assertion']}")
        lines.append("")

    # Generate one test per edge case
    for i, edge in enumerate(spec.get("edge_cases", []), 1):
        lines.append(f"def test_edge_case_{i}():")
        lines.append(f'    """Edge case: {edge["description"]}"""')
        if edge.get("raises"):
            lines.append(f"    with pytest.raises({edge['raises']}):")
            lines.append(f"        {spec['function']}({edge['input_repr']})")
        else:
            lines.append(f"    result = {spec['function']}({edge['input_repr']})")
            lines.append(f"    assert {edge['assertion']}")
        lines.append("")

    return "\n".join(lines)


# Example usage
spec = {
    "module": "analysis",
    "function": "summarize_measurements",
    "behaviors": [
        {
            "description": "Computes sample std (ddof=1)",
            "input_repr": "'test_data.csv'",
            "assertion": "abs(result['std'] - 2.0) < 1e-10",
        },
    ],
    "edge_cases": [
        {
            "description": "Empty file raises ValueError",
            "input_repr": "'empty.csv'",
            "raises": "ValueError",
        },
    ],
}

print(specification_to_contracts(spec))

Exercises

1. (Conceptual) Take a one-sentence prompt for a function in your domain (e.g., "normalize the spectra," "clean the dataset," "train a classifier"). List at least five specification gaps. Classify each as ambiguity, underspecification, implicit assumption, or contextual knowledge.
2. (Coding) Using the SpecCompleteness class, analyze the specification completeness of the following prompt: "Write a function that downloads a paper from arXiv given its ID and saves it as a PDF." List at least eight behavioral categories, mark each as covered or missing, and compute the completeness score. Then write the tests that would close the three most important gaps.
3. (Analysis) Generate the same function from two different prompts: the original one-sentence version and a specification-grade version using the template. Compare the generated code on five edge cases. How many edge cases does each version handle correctly? What is the relationship between prompt length and correctness?