4.2 Reasoning in Language Models

"I can solve your math problem, but only if you ask me to think step by step. Otherwise I just guess. Please do not ask why."

A Foundation Model Pretending to Be a Scientist

1. The Surprising Emergence of Reasoning

A system whose sole training objective is "predict the next word" can prove olympiad geometry theorems, identify drug candidates for untreatable diseases, and derive physical laws from raw data. In the previous section, we defined five forms of reasoning and gave each a clean computational realization. Transformers take a radically different approach: they contain no resolution engine and no structure mapping algorithm. Yet when prompted appropriately, models like Claude, GPT-4, and Gemini can solve syllogisms, identify analogies, propose abductive explanations, and prove mathematical theorems. How?

Mechanistic interpretability, the subfield that reverse-engineers the internal computations of neural networks into human-readable algorithms, supports the working hypothesis that transformers learn to simulate reasoning processes during pretraining. When the training corpus contains mathematical proofs, the model learns patterns like "if the premises include \(P \rightarrow Q\) and \(P\), the text usually continues with \(Q\)." This does not constitute deduction in the logical sense; the model does not verify soundness. It is, however, a useful statistical approximation that produces correct results on a wide range of inputs. (As of 2025, mechanistic interpretability work by Anthropic and others has identified specific attention heads and circuit structures that implement reasoning-like operations, moving beyond the purely statistical approximation view, though a complete mechanistic account remains an open problem.)

The critical insight for discovery AI practitioners: LLM reasoning is a capability that must be elicited, tested, and bounded, not a guarantee that can be assumed. The techniques in this section are the elicitation tools; the failure modes at the end are the boundaries. In short: LLMs do not reason by default; they reason when the prompt turns thinking into text.

2. Chain-of-Thought Prompting

The most widely adopted of these elicitation techniques, and the natural starting point, is a deceptively simple idea: asking the model to show its work.

On GSM8K, a benchmark of 8,500 grade-school math word problems requiring two to eight reasoning steps, some large language models answered multi-step word problems correctly less than 18% of the time when prompted directly (exact figures vary by model size and family). A single change to the prompt format, with no retraining and no new data, raised accuracy above 55%. The difference was not a better model; it was a better way of asking.

Wei et al. (2022) introduced chain-of-thought (CoT) prompting: instruct the model to produce intermediate reasoning steps before the final answer, rather than jumping directly to the conclusion.

In chain-of-thought prompting, you structure a model's input (via instructions or few-shot examples) so that the output includes explicit intermediate reasoning steps before the final answer. This converts problems that exceed a transformer's fixed computational depth into individually tractable sub-problems, each benefiting from the full context of prior steps. The mechanism is autoregressive amplification, where each generated token is appended to the input context so that later tokens can attend to earlier reasoning steps, effectively giving the model a growing working memory. Use CoT when a task requires multiple dependent reasoning steps (multi-hop questions, arithmetic, logical deduction). Prefer direct prompting for single-step classification, sentiment analysis, or tasks where latency and token cost outweigh accuracy gains.

Why Chain-of-Thought Works

Consider the computation graph. Without CoT, the model must compress a multi-step reasoning chain into a single forward pass (a fixed number of transformer layers). With CoT, each intermediate step becomes part of the input context for subsequent steps, giving the model a variable-length "working memory" through its own output tokens. The model can attend to its previously generated reasoning steps when producing the next one.

The mathematical framing: for a question \(Q\) with answer \(A\) and reasoning chain \(R = (r_1, r_2, \ldots, r_k)\):

Direct prompting asks for \(P(A \mid Q)\) in one shot. CoT prompting samples \(R\) first, then computes \(P(A \mid R, Q)\), which is often a much easier conditional distribution because the reasoning chain \(R\) has already done the hard work.

import anthropic

client = anthropic.Anthropic()

def reason_with_cot(question: str, domain: str = "science") -> dict:
    """Elicit chain-of-thought reasoning from Claude.

    Returns both the reasoning chain and final answer,
    separated for downstream processing.
    """
    system_prompt = f"""You are a {domain} reasoning assistant.
When given a question, think through it step by step.
Structure your response as:

REASONING:
[numbered steps of your reasoning]

ANSWER:
[your final answer]

CONFIDENCE:
[high/medium/low with brief justification]"""

    response = client.messages.create(
        model="claude-sonnet-4-20250514",
        max_tokens=2048,
        system=system_prompt,
        messages=[{"role": "user", "content": question}]
    )

    text = response.content[0].text

    # Parse structured output
    parts = {}
    for section in ["REASONING", "ANSWER", "CONFIDENCE"]:
        start = text.find(f"{section}:")
        if start != -1:
            # Find next section or end
            next_sections = [
                text.find(f"{s}:", start + len(section))
                for s in ["REASONING", "ANSWER", "CONFIDENCE"]
                if text.find(f"{s}:", start + len(section)) != -1
            ]
            end = min(next_sections) if next_sections else len(text)
            parts[section.lower()] = text[start + len(section) + 1:end].strip()

    return parts

# Example: scientific reasoning question
result = reason_with_cot(
    "A sealed container holds gas at 300K and 1 atm. "
    "The temperature is doubled to 600K. "
    "What happens to the pressure, and why?"
)
print("Reasoning:", result.get("reasoning", "")[:200], "...")
print("Answer:", result.get("answer", ""))
print("Confidence:", result.get("confidence", ""))

3. Scratchpad and Extended Thinking

The scratchpad technique, introduced by Nye et al. (2021), takes chain-of-thought further. Instead of merely asking the model to "think step by step," we provide an explicit workspace where the model can perform intermediate computations, store partial results, and revise its approach.

Mental Model

Think of extended thinking like a chef's prep kitchen versus a live cooking show. In the live show (standard prompting), the chef must plate each ingredient the moment it comes out, with no chance to taste, adjust, or start over. In the prep kitchen (extended thinking), the chef can try a sauce, realize it needs more acid, discard a failed reduction, and rework the dish before it ever reaches the dining room. The thinking tokens are the prep kitchen: a private workspace where the model can explore dead ends, catch its own mistakes, and revise its approach before committing to a final answer. The diner (the user) only sees the polished plate, but the quality depends on the freedom to experiment behind the scenes.

Modern reasoning models formalize this through extended thinking: the model generates a potentially long internal reasoning trace (the "thinking" tokens) before producing the visible response. Claude's extended thinking, OpenAI's o1 and o3 models, and DeepSeek-R1 all implement variants of this approach. The key difference from basic CoT is that the model may backtrack, consider alternatives, and self-correct within the thinking trace.

def reason_with_extended_thinking(question: str) -> dict:
    """Use Claude's extended thinking for complex reasoning.

    Extended thinking allocates additional compute at inference
    time, allowing the model to explore and self-correct.
    """
    client = anthropic.Anthropic()

    response = client.messages.create(
        model="claude-sonnet-4-20250514",
        max_tokens=16000,
        temperature=1,  # Required for extended thinking
        thinking={
            "type": "enabled",
            "budget_tokens": 10000  # Up to 10k tokens for reasoning
        },
        messages=[{
            "role": "user",
            "content": question
        }]
    )

    result = {"thinking": "", "answer": ""}
    for block in response.content:
        if block.type == "thinking":
            result["thinking"] = block.thinking
        elif block.type == "text":
            result["answer"] = block.text

    return result

# Example: a multi-step scientific reasoning problem
result = reason_with_extended_thinking(
    "A researcher observes that a protein's enzymatic activity "
    "increases when the temperature rises from 25C to 37C, "
    "but drops sharply at 42C. Meanwhile, circular dichroism "
    "spectroscopy shows no change in secondary structure at 37C "
    "but significant changes at 42C. What can we conclude about "
    "the relationship between structure and function, and what "
    "experiment would confirm it?"  # circular dichroism (CD) spectroscopy measures protein secondary structure by detecting differential absorption of polarized light
)
print(f"Thinking tokens used: ~{len(result['thinking'].split())}")
print(f"Answer preview: {result['answer'][:300]}...")

4. Tool-Augmented Reasoning

Extended thinking and scratchpads expand the model's internal workspace, yet some reasoning steps require not more thinking but access to external ground truth that no amount of token generation can provide.

Pure language model reasoning hits hard limits: arithmetic errors on large numbers, inability to access current data, hallucinated facts. Tool-augmented reasoning addresses these limits by letting the model delegate specific subtasks to external tools: calculators, databases, code interpreters, search engines.

The architecture, which we explore in depth in Chapter 12: Building MCP Servers, follows a simple pattern: the model decides when to use a tool and what to ask; the tool provides a reliable answer; the model incorporates the answer into its reasoning chain. Figure 4.3 illustrates the general structure of this neural-symbolic loop as realized in AlphaGeometry.

import json

tools = [
    {
        "name": "calculate",
        "description": "Evaluate a mathematical expression. "
                       "Use for any arithmetic, algebra, or calculus.",
        "input_schema": {
            "type": "object",
            "properties": {
                "expression": {
                    "type": "string",
                    "description": "Math expression, e.g. '2**10 * 3.14'"
                }
            },
            "required": ["expression"]
        }
    },
    {
        "name": "lookup_constant",
        "description": "Look up a physical or chemical constant.",
        "input_schema": {
            "type": "object",
            "properties": {
                "name": {
                    "type": "string",
                    "description": "Constant name, e.g. 'speed_of_light'"
                }
            },
            "required": ["name"]
        }
    }
]

CONSTANTS = {
    "speed_of_light": {"value": 2.998e8, "unit": "m/s"},
    "boltzmann": {"value": 1.381e-23, "unit": "J/K"},
    "avogadro": {"value": 6.022e23, "unit": "mol^-1"},
    "planck": {"value": 6.626e-34, "unit": "J*s"},
    "gas_constant": {"value": 8.314, "unit": "J/(mol*K)"},
}

def handle_tool_call(tool_name: str, tool_input: dict) -> str:
    """Execute a tool call and return the result."""
    if tool_name == "calculate":
        try:
            # Safe eval for math expressions
            result = eval(tool_input["expression"], {"__builtins__": {}},
                         {"abs": abs, "round": round, "min": min, "max": max})
            return json.dumps({"result": result})
        except Exception as e:
            return json.dumps({"error": str(e)})
    elif tool_name == "lookup_constant":
        name = tool_input["name"].lower().replace(" ", "_")
        if name in CONSTANTS:
            return json.dumps(CONSTANTS[name])
        return json.dumps({"error": f"Unknown constant: {name}"})
    return json.dumps({"error": f"Unknown tool: {tool_name}"})

def reason_with_tools(question: str) -> str:
    """Reasoning loop with tool use for scientific calculations."""
    client = anthropic.Anthropic()
    messages = [{"role": "user", "content": question}]

    while True:
        response = client.messages.create(
            model="claude-sonnet-4-20250514",
            max_tokens=4096,
            tools=tools,
            messages=messages
        )

        # Collect all content blocks
        tool_results = []
        final_text = ""

        for block in response.content:
            if block.type == "text":
                final_text += block.text
            elif block.type == "tool_use":
                result = handle_tool_call(block.name, block.input)
                tool_results.append({
                    "type": "tool_result",
                    "tool_use_id": block.id,
                    "content": result
                })

        if response.stop_reason == "end_turn":
            return final_text

        # Feed tool results back for continued reasoning
        messages.append({"role": "assistant", "content": response.content})
        messages.append({"role": "user", "content": tool_results})

# Example: requires both constant lookup and calculation
answer = reason_with_tools(
    "What is the de Broglie wavelength of an electron "
    "moving at 1% of the speed of light? "
    "Show your work using the formula lambda = h / (m * v)."
)
print(answer)

Practical Example: AlphaGeometry as a Reasoning Architecture

AlphaGeometry (Trinh et al., 2024) demonstrates the most sophisticated integration of neural and symbolic reasoning to date. It solves olympiad-level geometry problems by combining two systems, as shown in Figure 4.3:

• A neural language model trained on synthetic geometry proofs. This model proposes auxiliary constructions ("add point \(E\) as the midpoint of \(AC\)") that unlock proof paths the symbolic engine cannot find on its own.
• A symbolic deduction engine (DD+AR: Deductive Database with Algebraic Reasoning) that applies geometry rules (angle chasing, congruence, similarity) deterministically and exhaustively.

The architecture loops: the symbolic engine attempts a proof, gets stuck, asks the neural model for a construction, adds it to the diagram, and retries. This loop solved 25 out of 30 olympiad geometry problems, approaching the performance of gold-medalist humans. (A successor system, AlphaGeometry 2, announced in 2024, raised this to 28 out of 30, surpassing gold-medalist average performance by combining a Gemini-based language model with a faster symbolic engine.)

For discovery AI, AlphaGeometry is a template: use neural models for the creative, abductive step (proposing constructions, hypotheses) and symbolic systems for the deductive, verification step (checking consequences, proving theorems). Neither alone is sufficient. The integration pattern recurs in Chapter 35: Symbolic Regression and Chapter 53: AI Scientists. Figure 4.2.1 illustrates neuro-symbolic reasoning loop.

5. Reasoning Models: Scaling Test-Time Compute

Tool augmentation addresses the reliability of individual reasoning steps, but a separate question remains: can we make models reason more deeply by giving them more time to think?

The 2024-2025 generation of "reasoning models" represents a paradigm shift: instead of making models larger (scaling training compute), these systems make models think longer (scaling test-time compute, the computational budget spent during inference rather than during training). The key idea is reinforcement learning (RL) on reasoning traces: the model learns not just to produce correct answers but to produce effective reasoning chains that lead to correct answers.

The landscape includes OpenAI's o1 and o3 series, DeepSeek-R1, and Claude's extended thinking mode. (As of mid-2025, this space continues to evolve rapidly; newer entries such as OpenAI's o4-mini and Google's Gemini 2.5 with "thinking mode" have joined the field, each offering different cost/performance tradeoffs for reasoning tasks.) These models share several architectural principles:

1. Internal chain-of-thought: the model generates a potentially long reasoning trace before the final answer
2. Backtracking and self-correction: the trace may explore dead ends, recognize errors, and try alternative approaches
3. Compute-accuracy tradeoff: more thinking tokens generally yield better answers, with diminishing returns
4. Verification loops: the model checks its own intermediate results, catching errors that a single-pass model would propagate

For discovery AI, reasoning models offer a new design dimension. Instead of choosing a larger model for harder problems, we can choose a longer thinking budget. This is particularly valuable for scientific reasoning, where problems vary enormously in difficulty but the types of reasoning steps are relatively consistent.

6. Systematic Reasoning Failures

LLM reasoning, however impressive, exhibits systematic failure modes that discovery AI practitioners must understand and design around. These are not random errors; they are predictable consequences of how language models process information.

6.1 Compositional Fragility

LLMs handle individual reasoning steps well but degrade as the number of required steps increases. A model that correctly solves 95% of single-step inferences will solve only \(0.95^{10} \approx 60\%\) of 10-step chains correctly (assuming independent errors). In practice, errors tend to compound worse than independently because early errors can corrupt the context for subsequent steps, though the degree of correlation varies by task and model.

6.2 Sensitivity to Irrelevant Information

Adding irrelevant details to a problem statement can dramatically change model performance. The classic demonstration: "John has 5 apples. He gives 2 to Mary. He then takes a bus to school. How many apples does John have?" Models sometimes incorporate the bus detail into their arithmetic. For discovery AI, this means that noisy or tangential data in the prompt can derail reasoning about the core scientific question.

6.3 Reversal Curse and Directional Failures

Models trained on "A is B" do not automatically know "B is A." Berglund et al. (2023) demonstrated that models trained on "The mother of X is Y" could not answer "Who is Y's child?" This asymmetry matters for scientific reasoning because many relationships are bidirectional (if gene A regulates gene B, then gene B is regulated by gene A), and a model's inability to reverse its training patterns creates blind spots.

6.4 Faithful vs. Unfaithful Chain-of-Thought

The previous failure modes concern what the model gets wrong; this one concerns what it gets right for the wrong reasons. Perhaps the most dangerous failure mode: the reasoning chain can be wrong while the answer is right (the model "knows" the answer from pattern matching and generates a post-hoc justification, a plausible-sounding explanation constructed after the conclusion rather than before it), or the chain can be right while the answer is wrong (the model follows correct logic but makes an arithmetic or factual error in the final step). Turpin et al. (2023) showed that CoT explanations are frequently unfaithful to the model's actual computation, meaning you cannot trust the reasoning trace as a reliable explanation of the model's conclusion.

def diagnose_reasoning(
    question: str,
    expected_answer: str,
    n_samples: int = 5
) -> dict:
    """Diagnose reasoning reliability by sampling multiple chains.

    If the model gives different answers across samples, the
    reasoning is fragile. If chains differ but answers agree,
    the model may be pattern-matching rather than reasoning.
    """
    client = anthropic.Anthropic()
    samples = []

    for i in range(n_samples):
        response = client.messages.create(
            model="claude-sonnet-4-20250514",
            max_tokens=2048,
            temperature=1.0,  # Increase diversity
            messages=[{
                "role": "user",
                "content": f"Think step by step:\n{question}"
            }]
        )
        samples.append(response.content[0].text)

    # Extract final answers (last line heuristic)
    answers = [s.strip().split("\n")[-1] for s in samples]
    unique_answers = set(answers)

    diagnosis = {
        "question": question,
        "n_samples": n_samples,
        "unique_answers": len(unique_answers),
        "answers": answers,
        "consensus": len(unique_answers) == 1,
        "agreement_rate": max(
            answers.count(a) for a in unique_answers
        ) / n_samples,
    }

    # Check if consensus matches expected
    if diagnosis["consensus"]:
        diagnosis["correct"] = expected_answer.lower() in answers[0].lower()
    else:
        diagnosis["correct"] = any(
            expected_answer.lower() in a.lower() for a in answers
        )

    return diagnosis

# Example: test a tricky reasoning problem
result = diagnose_reasoning(
    question=(
        "A researcher measures the boiling point of a liquid at "
        "sea level (100.0C) and at 2000m altitude. "
        "At altitude, will the boiling point be higher, lower, "
        "or the same? Why?"
    ),
    expected_answer="lower",
    n_samples=5
)
print(f"Consensus: {result['consensus']}")
print(f"Agreement rate: {result['agreement_rate']:.0%}")
print(f"Correct: {result['correct']}")