10.1 Structured Outputs and Tool Use

"You gave me a JSON schema and suddenly I stopped hallucinating field names. Coincidence? I think not."

A Language Model That Finally Passed Validation

1. The Problem with Unstructured Outputs

You ask a language model to extract five fields from a paper abstract. It returns a bulleted list. You repeat the identical prompt and get a markdown table. A third time, you get a JSON blob whose keys bear no resemblance to the ones you requested. Each format requires different parsing logic. The format can shift between calls to the same model with the same prompt, because the model's output distribution includes many plausible continuations.

This instability is a fundamental obstacle to reliable systems. The Discovery Workbench from Chapter 6 routes typed artifacts through a provenance graph. Every node expects data in a specific shape; a response that violates the expected schema breaks the entire pipeline.

From Instability to Guarantees

When an automated pipeline silently drops records because the model returned a markdown table instead of JSON, no error is raised, no exception is logged; the data simply vanishes. That invisible data loss is why format reliability is not a convenience feature but a correctness requirement.

The solution is constrained generation: mechanisms that force the model to produce outputs conforming to a predefined schema. There are three levels of constraint, each progressively stronger:

Constrained generation restricts a language model's token sampling so that every possible output satisfies a formal specification such as a JSON Schema or a context-free grammar. Unconstrained models produce structurally unpredictable outputs that break downstream code. Even a 1% malformation rate means a pipeline processing thousands of requests encounters dozens of failures per day. (That 1% is not a rounding error; at 10,000 calls per day it produces 100 silent breakages, each one a potential data-loss event.) At each decoding step, the mechanism computes a mask over the model's vocabulary, zeroing out any token that would violate the specification. Only valid continuations survive. Use constrained generation whenever your application consumes model output programmatically. Reserve free-form text generation for conversational or creative tasks where rigid structure would be counterproductive.

1. Prompt-level: instruct the model in the system prompt to output JSON with specific fields. This is a request, not a guarantee. The model may comply 95% of the time, which means it fails at scale.
2. API-level: use the provider's structured output mode, which constrains the token sampling process itself. The model literally cannot produce tokens that would violate the schema.
3. Validation-level: parse and validate the response against a Pydantic model after generation, raising typed errors on violations. This catches semantic issues (wrong value ranges, missing cross-field constraints) that schema-level constraints cannot express.

Production systems combine all three layers. In short: a schema turns a language model from a conversational partner into a software component whose outputs your code can trust without parsing heroics.

2. JSON Schema as a Contract Language

JSON Schema is a vocabulary for annotating and validating JSON documents. Every major LLM API (Anthropic, OpenAI, Google) uses JSON Schema as the contract language for structured outputs. Understanding its core constructs is essential.

A schema defines the shape of valid data: which fields exist, what types they carry, which are required, and what constraints apply. Here is a schema for a paper metadata record:

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "type": "object",
  "properties": {
    "title": {
      "type": "string",
      "description": "Full title of the paper"
    },
    "authors": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "name": {"type": "string"},
          "affiliation": {"type": "string"}
        },
        "required": ["name"]
      },
      "minItems": 1
    },
    "year": {
      "type": "integer",
      "minimum": 1900,
      "maximum": 2030
    },
    "keywords": {
      "type": "array",
      "items": {"type": "string"},
      "maxItems": 10
    },
    "methodology": {
      "type": "string",
      "enum": ["experimental", "computational", "theoretical", "review", "meta-analysis"]
    }
  },
  "required": ["title", "authors", "year", "methodology"]
}

When you pass this schema to a structured output API, the provider's decoding engine uses it to mask invalid tokens at each generation step. If the model has just produced "year":, only integer tokens are allowed next. If the integer so far is 18, only digits that keep the value between 1900 and 2030 are permitted. This is not post-hoc validation; it is constrained decoding that makes invalid outputs impossible.

Mental Model

Think of constrained decoding like a custom keyboard that changes its keys after every keystroke. When you fill out a paper tax form, you can write anything in any box. But imagine a digital form where the "Year" field physically removes all letter keys from the keyboard and only shows digit keys, and after you type "19," it removes the keys for digits that would produce a number outside the valid range. The model still chooses which valid key to press (it retains its judgment about the most likely year), but the form makes it structurally impossible to type something invalid. That is exactly what a JSON Schema does to the model's token vocabulary at each generation step: it reshapes the set of available next tokens so that only schema-compliant continuations remain.

3. Pydantic: Schemas from Python Types

Writing JSON Schema by hand is tedious and error-prone. Pydantic (a Python library for data validation using type annotations) lets you define schemas as Python classes, with full IDE support, autocomplete, and type checking. The library generates the JSON Schema automatically, validates data at runtime, and provides clear error messages when validation fails.

from pydantic import BaseModel, Field
from enum import Enum
from typing import Optional


class Methodology(str, Enum):
    """Controlled vocabulary for research methodology types."""
    EXPERIMENTAL = "experimental"
    COMPUTATIONAL = "computational"
    THEORETICAL = "theoretical"
    REVIEW = "review"
    META_ANALYSIS = "meta-analysis"


class Author(BaseModel):
    """A paper author with name and optional affiliation."""
    name: str = Field(description="Full name of the author")
    affiliation: Optional[str] = Field(
        default=None,
        description="Primary institutional affiliation"
    )


class PaperMetadata(BaseModel):
    """Structured metadata extracted from a paper abstract."""
    title: str = Field(description="Full title of the paper")
    authors: list[Author] = Field(
        min_length=1,
        description="List of authors, at least one required"
    )
    year: int = Field(ge=1900, le=2030, description="Publication year")
    keywords: list[str] = Field(
        default_factory=list,
        max_length=10,
        description="Up to 10 topic keywords"
    )
    methodology: Methodology = Field(
        description="Primary research methodology"
    )


# Generate JSON Schema from the Pydantic model
schema = PaperMetadata.model_json_schema()
print(schema)
# Output matches the hand-written schema above, plus $defs for nested models

The key advantage is that the same class serves three roles: (1) schema definition for the LLM API, (2) runtime validation for the response, and (3) typed data object for downstream code. There is no impedance mismatch (where the schema says one thing and the code expects another) between the contract and the code.

A single Pydantic model defines, validates, and represents the data, so the remaining piece is connecting it to an LLM API that returns a guaranteed-valid instance.

4. Structured Outputs with the Anthropic SDK

The Anthropic software development kit (SDK) supports structured outputs through tool use. You define a tool whose input schema matches your desired output shape, then instruct the model to "use" that tool. The model's tool-use response is guaranteed to conform to the schema.

import anthropic
import json

client = anthropic.Anthropic()


def extract_metadata(abstract: str) -> PaperMetadata:
    """Extract structured metadata from a paper abstract.

    Uses Claude's tool-use mode to guarantee schema compliance.
    """
    # Define the extraction tool with our Pydantic schema
    tools = [
        {
            "name": "record_metadata",
            "description": (
                "Record the structured metadata extracted from a "
                "scientific paper abstract."
            ),
            "input_schema": PaperMetadata.model_json_schema(),
        }
    ]

    response = client.messages.create(
        model="claude-sonnet-4-20250514",
        max_tokens=1024,
        tools=tools,
        tool_choice={"type": "tool", "name": "record_metadata"},
        messages=[
            {
                "role": "user",
                "content": (
                    f"Extract metadata from this abstract:\n\n{abstract}"
                ),
            }
        ],
    )

    # The response is guaranteed to contain a tool_use block
    tool_block = next(
        block for block in response.content
        if block.type == "tool_use"
    )

    # Validate with Pydantic (catches semantic issues beyond schema)
    return PaperMetadata.model_validate(tool_block.input)


# Example usage
abstract = """
We present a graph neural network approach to predicting
protein-ligand binding affinity. Our model achieves state-of-the-art
performance on the PDBbind benchmark, with a Pearson correlation of
0.87 on the core set. We train on 16,000 complexes and validate on
1,000 held-out structures.
"""
metadata = extract_metadata(abstract)
print(f"Title: {metadata.title}")
print(f"Method: {metadata.methodology.value}")
print(f"Year: {metadata.year}")
# Title: Graph Neural Network Approach to Predicting ...
# Method: computational
# Year: 2024

Notice the two-layer validation strategy. The API's constrained decoding guarantees that the JSON conforms to the schema (correct types, required fields present, enum values valid). The Pydantic model_validate call then applies Python-level constraints (field ranges, cross-field logic, custom validators) that JSON Schema cannot express.

5. Function Calling and Tool Use

Structured outputs are one direction: model to code. Function calling (also called tool use) adds the reverse direction: the model can invoke your code and receive results. This creates a bidirectional interface where the model reasons about when to call tools, what arguments to pass, and how to interpret the results.

The pattern, introduced conceptually in Chapter 4 as part of ReAct-style reasoning (where the model alternates between reasoning about what to do and acting by calling a tool), works as follows. Figure 10.1 illustrates this cycle.

1. You define a set of tools, each with a name, description, and input schema.
2. The model receives a user query plus the tool definitions.
3. The model decides whether to answer directly or call a tool (or multiple tools).
4. If it calls a tool, your code executes the function and returns the result.
5. The model incorporates the result and either answers or calls another tool.

Figure 10.1.1 illustrates the tool-use agent loop. Here is a complete example that gives Claude access to a literature search tool:

import anthropic
from pydantic import BaseModel, Field


class SearchQuery(BaseModel):
    """Input schema for literature search."""
    query: str = Field(description="Search query for paper titles/abstracts")
    max_results: int = Field(default=5, ge=1, le=20)
    year_min: int = Field(default=2020, ge=1900)


class SearchResult(BaseModel):
    """A single search result."""
    title: str
    authors: list[str]
    year: int
    abstract: str
    doi: str


def search_literature(params: SearchQuery) -> list[dict]:
    """Search a paper database. In production, this calls
    Semantic Scholar, OpenAlex, or a local index."""
    # Simplified: in practice, call an API
    return [
        {
            "title": "Graph Neural Networks for Molecular Property Prediction",
            "authors": ["Alice Chen", "Bob Kumar"],
            "year": 2023,
            "abstract": "We propose a message-passing neural network...",
            "doi": "10.1234/example.2023.001",
        }
    ]


def run_agent_loop(user_query: str) -> str:
    """Run a tool-use agent loop until the model produces a final answer."""
    client = anthropic.Anthropic()

    tools = [
        {
            "name": "search_literature",
            "description": (
                "Search the scientific literature database. "
                "Returns papers matching the query with titles, "
                "authors, years, abstracts, and DOIs."
            ),
            "input_schema": SearchQuery.model_json_schema(),
        }
    ]

    messages = [{"role": "user", "content": user_query}]

    while True:
        response = client.messages.create(
            model="claude-sonnet-4-20250514",
            max_tokens=2048,
            tools=tools,
            messages=messages,
        )

        # Check if the model wants to use a tool
        if response.stop_reason == "tool_use":
            # Process each tool call
            tool_results = []
            for block in response.content:
                if block.type == "tool_use":
                    # Validate input, execute function
                    params = SearchQuery.model_validate(block.input)
                    results = search_literature(params)
                    tool_results.append({
                        "type": "tool_result",
                        "tool_use_id": block.id,
                        "content": str(results),
                    })

            # Feed results back to the model
            messages.append({"role": "assistant", "content": response.content})
            messages.append({"role": "user", "content": tool_results})
        else:
            # Model produced a final text response
            return response.content[0].text


answer = run_agent_loop(
    "Find recent papers on graph neural networks for drug discovery"
)
print(answer)

6. The Model Context Protocol (MCP)

Individual tool definitions work well for a single application, but they do not scale. If every team defines its own tool schemas, its own serialization, and its own execution protocol, interoperability is impossible. The Model Context Protocol (MCP), introduced by Anthropic in 2024, solves this by defining a standard protocol for connecting LLMs to external tools and data sources.

MCP defines three primitives:

• Tools: functions the model can call, with JSON Schema input/output contracts.
• Resources: data sources the model can read (files, database tables, API endpoints).
• Prompts: reusable prompt templates that tools can expose to the model.

An MCP server exposes these primitives over a standard transport (stdio or HTTP with server-sent events). Any MCP client (Claude Desktop, an IDE plugin, a custom agent) can discover and use the server's capabilities without custom integration code.

from mcp.server.fastmcp import FastMCP
from pydantic import BaseModel, Field

# Create an MCP server for literature search
mcp = FastMCP("literature-search")


class PaperSearchInput(BaseModel):
    """Schema for the search tool's input."""
    query: str = Field(description="Search terms")
    max_results: int = Field(default=10, ge=1, le=50)
    year_range: tuple[int, int] = Field(
        default=(2020, 2026),
        description="(start_year, end_year) inclusive"
    )


class PaperRecord(BaseModel):
    """Schema for a returned paper record."""
    title: str
    authors: list[str]
    year: int
    doi: str
    abstract: str
    citation_count: int


@mcp.tool()
def search_papers(input: PaperSearchInput) -> list[PaperRecord]:
    """Search the scientific literature and return matching papers.

    Queries Semantic Scholar and OpenAlex, deduplicates by DOI,
    and ranks by relevance score.
    """
    # Implementation calls external APIs
    # Returns validated PaperRecord objects
    ...


@mcp.resource("papers://{doi}")
def get_paper(doi: str) -> str:
    """Retrieve the full text of a paper by DOI."""
    ...


if __name__ == "__main__":
    mcp.run()

The critical insight is that MCP tool schemas are just JSON Schema, the same contract language used for structured outputs. This means the type system is consistent from end to end: Pydantic defines the schema, the LLM API enforces it during generation, MCP transmits it between processes, and Pydantic validates it again on receipt. One type definition flows through the entire stack.

Because every major provider has converged on JSON Schema as its contract language, switching between SDKs becomes a matter of surface syntax rather than conceptual redesign.

7. Cross-SDK Comparison: Anthropic, OpenAI, and Beyond

The structured output and tool-use patterns differ in surface syntax across providers, but the underlying model is the same: define a schema, pass it to the API, get back validated data. Here is a side-by-side comparison for the paper metadata extraction task:

# --- Anthropic SDK ---
import anthropic

def extract_anthropic(abstract: str) -> PaperMetadata:
    client = anthropic.Anthropic()
    response = client.messages.create(
        model="claude-sonnet-4-20250514",
        max_tokens=1024,
        tools=[{
            "name": "record_metadata",
            "description": "Record extracted paper metadata.",
            "input_schema": PaperMetadata.model_json_schema(),
        }],
        tool_choice={"type": "tool", "name": "record_metadata"},
        messages=[{"role": "user", "content": f"Extract: {abstract}"}],
    )
    tool_block = next(b for b in response.content if b.type == "tool_use")
    return PaperMetadata.model_validate(tool_block.input)


# --- OpenAI SDK (Responses API, 2025) ---
from openai import OpenAI

def extract_openai(abstract: str) -> PaperMetadata:
    client = OpenAI()
    response = client.responses.create(
        model="gpt-4o",
        input=[{"role": "user", "content": f"Extract: {abstract}"}],
        text={"format": {
            "type": "json_schema",
            "name": "paper_metadata",
            "schema": PaperMetadata.model_json_schema(),
            "strict": True,  # enables constrained decoding
        }},
    )
    return PaperMetadata.model_validate_json(response.output_text)

The Pydantic model is the common denominator. Write it once, and it works with any provider that accepts JSON Schema. This is why Pydantic has become the de facto standard for typed LLM interfaces in Python.

8. Designing Typed Agent Interfaces

With structured outputs and tool use in hand, you can design complete agent interfaces as type hierarchies. The pattern draws on the interface design principles from Chapter 3 (knowledge representation) and the architecture patterns from Chapter 6 (Discovery Workbench).

from pydantic import BaseModel, Field
from typing import Literal, Union
from enum import Enum
from datetime import datetime


# --- Domain types ---
class Confidence(str, Enum):
    HIGH = "high"
    MEDIUM = "medium"
    LOW = "low"


class Citation(BaseModel):
    """A reference to a specific paper."""
    doi: str
    title: str
    relevance: str = Field(description="Why this paper is relevant")


class Claim(BaseModel):
    """A scientific claim extracted from literature."""
    statement: str = Field(description="The claim in one sentence")
    evidence: list[Citation] = Field(min_length=1)
    confidence: Confidence
    methodology: str = Field(description="How the claim was established")


# --- Agent interface ---
class AnalysisRequest(BaseModel):
    """Input contract for the literature analysis agent."""
    research_question: str
    scope: Literal["narrow", "broad"] = "narrow"
    max_papers: int = Field(default=20, ge=5, le=100)
    year_range: tuple[int, int] = Field(default=(2020, 2026))


class AnalysisResult(BaseModel):
    """Output contract for the literature analysis agent."""
    question: str
    claims: list[Claim] = Field(min_length=1)
    gaps: list[str] = Field(
        description="Identified gaps in the literature"
    )
    suggested_experiments: list[str]
    papers_reviewed: int
    timestamp: datetime


# The agent function has a clear typed signature
def analyze_literature(request: AnalysisRequest) -> AnalysisResult:
    """Run the literature analysis agent.

    This function orchestrates multiple LLM calls and tool uses,
    but its interface is a simple typed function.
    """
    # Implementation uses the tool-use patterns from above
    ...

This design has three properties that matter for discovery systems:

• Testability: you can write unit tests that construct an AnalysisRequest, call the function, and assert properties of the AnalysisResult. No parsing, no regex, no "did the model say something that looks right."
• Composability: the output of one agent becomes the input of another. An AnalysisResult can feed directly into a hypothesis generation agent (see Chapter 39) because both share the Claim and Citation types.
• Observability: every request and result can be serialized to JSON and logged to the Discovery Workbench's provenance system, creating an auditable trace of what the agent did, what it found, and what it concluded.

class ReactionResult(BaseModel):
    reagents: list[str] = Field(min_length=1)
    product: str
    yield_percent: float = Field(ge=0, le=100)
    temperature_c: float
    catalyst: Optional[str] = None

Exercises

1. Conceptual: Explain why prompt-level instructions ("Please respond in JSON") are insufficient for production systems. What failure modes does constrained decoding eliminate that prompt instructions cannot?
2. Coding: Define a Pydantic model for a ExperimentRecord that captures: hypothesis (string), independent variables (list of named variables with types and ranges), dependent variables (list), methodology (enum: "in_vitro", "in_vivo", "in_silico", "observational"), sample size (positive integer), and statistical test used (string). Generate the JSON Schema and use it with the Anthropic SDK to extract experiment records from a paragraph describing a drug trial.
3. Analysis: Compare the structured output APIs of Anthropic (tool-use mode) and OpenAI (json_schema response format). What are the trade-offs in terms of (a) schema expressiveness, (b) streaming support, and (c) error recovery when the model struggles with a constraint? Write a wrapper function that provides a unified interface across both providers.