Overview
Software engineering is entering a transformation as profound as the shift from assembly language to high-level programming. Large language models trained on billions of lines of code can now generate functions, explain algorithms, fix bugs, and reason about entire repositories. But these capabilities do not emerge from magic; they rest on concrete foundations in pre-training, fine-tuning, in-context learning, program synthesis, and structured representations of code.
This chapter builds those foundations. We start with how code LLMs learn to model programs as sequences of tokens, then move to the graph structures (dependency, call, symbol) that let AI reason about code at the repository level. We examine the collaboration patterns that emerge when humans and AI write software together, and we close with a hands-on recipe: using a coding agent to implement a feature with tests, reasoning traces, and structured verification.
The ideas here connect backward to Chapter 7's framing of software as a discovery process and forward to Chapter 9's treatment of vibe coding, Chapter 11's context engineering, and the multi-agent teams of Chapter 17. Every tool and technique we explore in Part II rests on the foundations laid here.
Prerequisites
Readers should have completed Chapter 1: Discovery as Search (for the search-over-programs perspective), Chapter 4: Reasoning for Discovery (for the inference and planning foundations), and Chapter 7: Software Development as a Discovery Process (for the framing of software development as exploration). We assume comfortable Python programming, familiarity with git, and a conceptual understanding of neural network training (loss functions, gradient descent, attention). No prior experience with code LLMs is required.
Learning Outcomes
- Explain how code foundation models are pre-trained, fine-tuned, and adapted through in-context learning.
- Distinguish fill-in-the-middle, instruction-tuned, and agentic code generation paradigms.
- Build and query repository-level graph structures: dependency graphs, call graphs, and symbol tables.
- Map the issue-to-patch pipeline and implement test-guided program repair.
- Compare human-AI collaboration patterns: autocomplete, chat, agent, and autonomous modes.
- Use a coding agent to implement a feature from a natural-language specification, with tests and a reasoning trace.
Sections
8.1 Code Foundation Models
Pre-training on code corpora, tokenization for programming languages, fill-in-the-middle objectives, instruction tuning, and in-context learning. The architecture and training pipeline behind models like Codex, StarCoder, and Claude.
8.2 Repository-Level Reasoning
Dependency graphs, call graphs, and symbol tables. Representing repositories as structured graphs. Issue-to-patch search, test-guided repair, and localizing changes across a codebase.
8.3 Human-AI Collaboration Patterns
Autocomplete, chat, agent, and autonomous collaboration modes. Tool-augmented code generation with Claude Code, Codex CLI, and Cursor. The spectrum from suggestion to delegation.
8.4 Building with a Coding Agent
Recipe: a coding agent implements a feature from specification to passing tests. Structured reasoning traces, tool use, iterative refinement, and verification. Extending the Discovery Workbench.
Bibliography
Foundational Papers
The Codex paper that introduced HumanEval and demonstrated the potential of large-scale code generation from natural language.
The StarCoder model, trained on The Stack, demonstrating open-source code LLMs with fill-in-the-middle capability and strong multi-language performance.
The FIM (fill-in-the-middle) training objective that enables code models to perform infilling, a key capability for IDE integration.
The benchmark that measures end-to-end issue resolution on real GitHub repositories, establishing the standard for evaluating coding agents.
Books
A practitioner-oriented guide to integrating AI tools into software development workflows, with case studies from GitHub Copilot's deployment.
Chapters on planning and search provide the formal foundations for understanding how coding agents explore solution spaces.
Tools & Libraries
Anthropic's agentic coding tool that operates directly in the terminal, reading and editing files, running commands, and managing git workflows.
OpenAI's open-source command-line coding agent for executing multi-step software engineering tasks with sandboxed code execution.
An AI-native code editor built on VS Code that integrates chat, autocomplete, and agentic editing with multi-model support.
An incremental parsing library that builds concrete syntax trees for source code, used as the backbone for repository-level code analysis.
Tutorials & Surveys
A comprehensive survey covering pre-training, fine-tuning, and evaluation of code LLMs across multiple tasks and languages.
A recent survey focusing specifically on code generation capabilities, benchmarks, and the trajectory of coding agent development.