8.1 Code Foundation Models

The Big Picture

Code foundation models are transformer-based language models trained on massive corpora of source code and natural language. They learn statistical patterns over token sequences that capture syntax, semantics, idioms, API usage, and even aspects of program behavior. Three training phases determine what a code model can do: pre-training builds broad competence over code and language, fine-tuning specializes the model for specific tasks (instruction following, code repair, test generation), and in-context learning adapts behavior at inference time through prompts and examples. Understanding these three phases is essential for knowing what coding AI tools can and cannot do, and for using them effectively in your own work. Figure 8.1 illustrates how these phases build on each other. Figure 8.1.1 illustrates the three-phase training pipeline of code foundation models.

1. Code as a Sequence Modeling Problem

In 2021, a neural network that had never executed a single program reportedly scored higher on introductory programming exams than most human students, generating correct solutions from nothing but natural-language problem descriptions. The trick was deceptively simple: treat source code the same way a language model treats English prose, as a sequence of tokens to predict. Given a prefix of tokens \(x_1, x_2, \ldots, x_{t-1}\), the model estimates a probability distribution over the next token:

where \(\theta\) denotes the model's parameters. Training maximizes the log-likelihood over a corpus \(\mathcal{D}\) of code files:

This objective, called causal language modeling (CLM), where the model predicts each token using only the tokens that precede it, is the same one used for GPT-style text models. What makes code models distinctive is not the objective but the training data, the tokenizer, and the additional training objectives layered on top.

Code differs from natural language in ways that matter for modeling. Parsers and compilers enforce strict syntax; a single misplaced bracket makes a program fail. Identifiers carry semantic weight (a variable named user_count conveys intent that x does not). Python treats indentation as syntactic structure, and all languages rely on it for readability. Files within a repository depend on each other through imports, inheritance, and API contracts. These structural properties force a code model to learn far more rigid constraints than a prose model, but they also supply richer training signal. In short: code is a language strict enough to compile yet expressive enough to narrate its own intent, and that dual nature is exactly what makes it ideal training data for a neural model.

2. Tokenization for Programming Languages

Tokenization, the process of splitting raw text into the discrete units the model processes, is the very first step in any language model pipeline; the model never sees raw characters, only token IDs. This step presents unique challenges for code. Natural-language tokenizers like byte-pair encoding (BPE, explained below) are trained primarily on prose and may fragment code tokens in unhelpful ways. Consider the Python identifier get_user_by_email: a prose-trained BPE tokenizer might split it into ["get", "_user", "_by", "_email"], while a code-aware tokenizer preserves it as fewer, more meaningful tokens.

The tokenizer converts raw characters into integer IDs from a fixed vocabulary; the transformer then maps these IDs to embedding vectors for processing. The tokenizer choice directly controls how much source code fits within the model's finite context window, how much each API call costs (providers charge per token), and whether the model treats meaningful code constructs as atomic units rather than arbitrary character fragments.

Byte-pair encoding (BPE), the most common tokenization algorithm, starts with individual bytes or characters and iteratively merges the most frequently co-occurring pair into a new vocabulary entry. It repeats this process until the vocabulary reaches a target size. Use a code-specialized tokenizer (such as those shipped with StarCoder or DeepSeek-Coder) whenever your workload is predominantly code; fall back to a general-purpose tokenizer (like GPT-4's cl100k_base) only when the input mixes prose and code roughly equally.

Whitespace and indentation are critical in Python but are often collapsed by prose tokenizers. Code-specialized tokenizers preserve indentation tokens explicitly, sometimes encoding indentation depth as a single token (e.g., <INDENT_4>). This reduces sequence length (important because transformer cost scales quadratically with context in standard attention) while preserving syntactic structure.

The effect becomes concrete with a simple Python function:

import tiktoken

# Compare tokenization of the same code snippet
code = """def fibonacci(n: int) -> list[int]:
    sequence = [0, 1]
    for i in range(2, n):
        sequence.append(sequence[-1] + sequence[-2])
    return sequence
"""

# GPT-4 tokenizer (cl100k_base)
enc_gpt4 = tiktoken.get_encoding("cl100k_base")
tokens_gpt4 = enc_gpt4.encode(code)
print(f"cl100k_base tokens: {len(tokens_gpt4)}")

# Inspect individual tokens to see split points
for tok in tokens_gpt4[:15]:
    print(f"  {tok:6d} -> {repr(enc_gpt4.decode([tok]))}")

The token count directly impacts both inference cost and the amount of code that fits within the model's context window. A tokenizer that uses 40 tokens for a function signature leaves less room in the context for surrounding code. This is why models designed for code (StarCoder, Code Llama, DeepSeek-Coder) train their tokenizers on code-heavy corpora, yielding vocabularies where common programming constructs (def, return, import, self.) are single tokens.

3. Pre-training: Learning the Language of Code

Pre-training is the phase where the model absorbs the statistical structure of code from a large corpus. The scale of these corpora is staggering: The Stack v2, used to train StarCoder2, contains over 900 billion tokens across 600+ programming languages. Codex (circa 2021) was trained on 159 GB of Python code from GitHub. These datasets include not just code but also comments, docstrings, README files, commit messages, and issue discussions, providing the model with natural-language context that bridges intent and implementation.

The standard causal language modeling objective trains the model to predict code left-to-right. But real programming rarely proceeds linearly. A developer might write a function signature, skip to the return statement, then fill in the body. The fill-in-the-middle (FIM) objective addresses this by training the model to predict a missing span given surrounding context:

Pre-training and FIM give a model broad competence over code patterns, but that raw capability alone does not make the model useful as a programming assistant; it still needs to learn how to follow human instructions.

4. Fine-tuning: From Completion to Conversation

A pre-trained code model is a powerful completion engine, but it has no concept of following instructions. If you type "write a function that sorts a list," the model might complete your sentence with more English prose rather than producing code. Fine-tuning bridges this gap by training on curated datasets of (instruction, response) pairs.

Three fine-tuning approaches dominate code model development:

Supervised, Reward-Based, and Preference-Based Methods

Supervised fine-tuning (SFT) trains on human-written or human-verified instruction-response pairs. The dataset might contain entries like ("Write a Python function that computes the nth Fibonacci number using memoization", followed by a correct implementation). SFT teaches the model to map natural-language specifications to code.

Reinforcement learning from human feedback (RLHF) goes further by training a reward model on human preferences between pairs of outputs, then optimizing the code model to produce outputs that the reward model scores highly. For code, the reward signal can incorporate not just human preference but also execution results: does the code compile? Does it pass the test suite? This combination of human judgment and automated verification is particularly powerful for code because correctness is partially verifiable.

Direct preference optimization (DPO) simplifies RLHF by skipping the separate reward model. Instead, it directly optimizes the policy (the code model) using pairs of preferred and rejected outputs. DPO has gained traction for code models because it is simpler to implement while typically achieving comparable results.

The loss function for SFT is straightforward: given an instruction $I$ and a target response $R = (r_1, r_2, \ldots, r_m)$, we minimize:

Note that the loss is computed only over the response tokens $r_t$, not the instruction tokens. The instruction provides context but the model is not penalized for its representation of the instruction.

5. In-Context Learning: Adaptation at Inference Time

In-context learning (ICL) is the remarkable ability of large language models to adapt their behavior based on examples provided in the prompt, without updating any model parameters. For code models, ICL means you can teach the model a new API, a coding convention, or a transformation pattern simply by including examples in the context window.

ICL operates through what researchers call task vectors (directions in the model's internal representation space that encode a particular input-output mapping), where the model's internal activation patterns shift to represent the function implied by the in-context examples. When the model sees a pattern of (input, output) pairs in its context, its internal representations shift to implement the implied function. The number of examples needed depends on the complexity of the pattern: a simple name-refactoring rule might need one example, while a complex data transformation might need five or more.

ICL drives most practical code AI workflows. Pasting existing code into a chat with Claude or Cursor and requesting modifications provides in-context examples; a coding agent reading your repository before generating new code uses ICL to match your project's style. Context quality directly determines output quality, as detailed in Chapter 11: Context Engineering.

Whether the model adapts through fine-tuning or in-context examples, the end goal is the same: producing correct code from a specification. This goal has a long history that predates neural networks entirely.

6. Program Synthesis: From Specification to Code

Program synthesis, the automatic generation of programs from specifications, predates LLMs by decades. Classical approaches used deductive synthesis (deriving programs from logical specifications), inductive synthesis (generalizing from input-output examples), or constraint-based synthesis (encoding the specification as a satisfiability problem). LLM-based code generation represents a new paradigm: neural program synthesis, where the "specification" is a natural-language description and the "search" is implicit in the model's learned distribution over programs.

The connection to Chapter 1 is direct: program synthesis is search over the space of programs. The search space $S$ is the set of all syntactically valid programs; the objective $f$ is correctness with respect to the specification; and the model's learned distribution serves as a heuristic that guides search toward promising regions of program space. Temperature sampling (controlling randomness by scaling the logit distribution, where logits are the raw unnormalized scores the model assigns to each vocabulary token before converting them to probabilities), beam search (maintaining the top-$k$ partial sequences at each step), and nucleus sampling (sampling from the smallest set of tokens whose cumulative probability exceeds a threshold $p$) are all strategies for navigating this space, each with different exploration-exploitation trade-offs (recall Section 1.2).

The pass@k metric quantifies how effectively a model searches program space. Given a problem, we sample $n$ candidate programs and check how many of the first $k$ pass all test cases. The unbiased estimator for pass@k, given $n$ total samples and $c$ correct ones, is:

This metric reveals a key insight: generating multiple candidates and selecting among them is far more effective than generating a single "best" candidate. A model with 30% pass@1 can, under typical conditions, achieve 70% pass@10, because different samples explore different regions of program space. This motivates the generate-and-test approach that underlies most coding agents.

from math import comb

def pass_at_k(n: int, c: int, k: int) -> float:
    """Compute the unbiased pass@k estimator.

    Args:
        n: Total number of samples generated
        c: Number of correct samples (passing all tests)
        k: The k in pass@k (number of attempts allowed)

    Returns:
        Probability that at least one of k samples is correct
    """
    if n - c < k:
        return 1.0  # more correct than k, guaranteed pass
    return 1.0 - comb(n - c, k) / comb(n, k)


# Demonstrate the power of multiple samples
print("pass@k for a model with 30% base accuracy (n=100, c=30):")
for k in [1, 5, 10, 25, 50]:
    score = pass_at_k(n=100, c=30, k=k)
    print(f"  pass@{k:2d} = {score:.3f}")

# Output:
# pass@k for a model with 30% base accuracy (n=100, c=30):
#   pass@ 1 = 0.300
#   pass@ 5 = 0.836
#   pass@10 = 0.976
#   pass@25 = 1.000
#   pass@50 = 1.000

Library Shortcut: HumanEval with the Evaluate Library

The manual pass@k computation above illustrates the mathematics, but in practice you would use Hugging Face's evaluate library, which bundles the HumanEval metric (where HumanEval is a benchmark of 164 hand-written Python programming problems introduced by OpenAI in 2021 to measure functional correctness of code generation) and handles sampling, execution sandboxing, and the unbiased estimator in two lines (as of 2024, HumanEval has been largely supplemented by more challenging benchmarks such as SWE-bench, HumanEval+, MBPP+, and LiveCodeBench, which test repository-scale reasoning and guard against data contamination; HumanEval remains useful as a lightweight sanity check):

import evaluate

code_eval = evaluate.load("code_eval")
results = code_eval.compute(
    references=["assert add(2,3)==5"],       # test cases
    predictions=[["def add(a,b): return a+b",  # candidate 1
                  "def add(a,b): return a*b"]], # candidate 2
    k=[1, 2]
)
# results["pass@1"], results["pass@2"]

Listing 8.6: Evaluating code candidates against test cases using Hugging Face's evaluate library with sandboxed execution and automatic pass@k computation.

This reduces roughly 30 lines of sampling, sandboxed execution, and metric computation to 5 lines, while handling edge cases like infinite loops via timeouts.

With the core training techniques and evaluation metrics in hand, the natural next question is which models put them into practice, and how their design choices differ.

7. The Landscape of Code Foundation Models

The field has produced a rich ecosystem of code models, each with different trade-offs in size, training data, licensing, and capabilities. Understanding this landscape helps you choose the right model for your task.

Closed-source frontier models like Claude (Anthropic), GPT-4 (OpenAI), and Gemini (Google) are trained on diverse mixtures of code and text. They excel at complex multi-step reasoning, long-context understanding, and instruction following. Their code capabilities emerge from general intelligence rather than code-specific training alone, which makes them strong at tasks requiring both code and natural-language understanding: explaining code, translating between languages, and reasoning about software architecture.

Open-weight code-specialized models like StarCoder2, DeepSeek-Coder, and Code Llama are trained on code-heavy corpora with code-specific tokenizers and objectives (as of 2025, newer entrants such as DeepSeek-Coder-V2, Qwen2.5-Coder, and CodeGemma have joined this tier, often surpassing their predecessors on standard benchmarks). They are often smaller (1B to 33B parameters compared to frontier models at 100B+) but can match or exceed larger general models on pure coding benchmarks, particularly for languages well-represented in their training data. Their open weights enable fine-tuning for specific codebases or tasks, a capability we leverage in Chapter 16 on repository-scale implementation.

As a practical starting point for choosing between these tiers: use a closed-source frontier model when the task requires complex reasoning across code and natural language (architecture design, multi-step debugging, code review with explanations), and reach for an open-weight code-specialized model when you need low-latency completion, local deployment, or the ability to fine-tune on a proprietary codebase. Many production workflows combine both, routing simple completions to a smaller model and escalating complex queries to a frontier model.

import tiktoken, pathlib

code = pathlib.Path("your_file.py").read_text()
enc_100k = tiktoken.get_encoding("cl100k_base")   # GPT-4
enc_200k = tiktoken.get_encoding("o200k_base")     # GPT-4o
print(f"cl100k: {len(enc_100k.encode(code))} tokens")
print(f"o200k:  {len(enc_200k.encode(code))} tokens")

Exercises

1. (Conceptual) A code model trained only on Python is asked to generate Rust code. Explain why it might still produce partially correct Rust, and identify two specific failure modes you would expect compared to a model trained on both languages.
2. (Coding) Using the tiktoken library, compare the token count for the same 50-line Python function using the cl100k_base (GPT-4) and o200k_base (GPT-4o) tokenizers. Compute the percentage difference and explain which tokenizer is more efficient for code and why.
3. (Analysis) A model achieves pass@1 = 0.15 on HumanEval. You have a budget of 20 API calls per problem. Compute pass@20 using the formula from this section. Then argue whether it is more cost-effective to spend 20 calls on one model with 15% pass@1, or 10 calls each on two different models with 10% and 20% pass@1 respectively (assume independence).