11.2 Chunking and Summarization

1. Why Naive Chunking Fails

Imagine handing a colleague the top half of a function: the signature, the docstring, and seven lines of setup, but nothing after the for loop begins. Then you hand another colleague the bottom half: a loop body that references variables declared nowhere in the fragment, followed by a return statement whose meaning depends on the missing setup. Neither person can understand the code, and neither piece is useful in isolation. That is exactly what happens when a naive chunker splits source files into fixed-size token windows. The simplest such strategy uses windows of \(k\) tokens, optionally overlapping by \(k/2\) tokens. This works reasonably well for natural-language documents, where paragraph boundaries are somewhat arbitrary. For code, fixed-size chunking is catastrophic. A 512-token window might split a function signature from its body, separate a class definition from its methods, or cut a multi-line conditional in half. The resulting chunks are syntactically invalid and semantically incoherent.

Chunking, in retrieval-augmented generation, partitions a source document into discrete, self-contained segments. Each segment can be independently embedded, indexed, and retrieved. Chunk quality matters because every downstream step (embedding, ranking, context packing) depends on whether each chunk represents a coherent semantic unit. A chunk that mixes unrelated logic produces a noisy embedding that matches the wrong queries. The core mechanism: a splitter walks the document according to some boundary rule (character count, line breaks, or parse-tree nodes) and emits one segment per boundary. Use syntax-aware chunking (described below) whenever the source has formal grammar (code, markup, structured data). Reserve fixed-size or sentence-level splitting for unstructured prose where no grammar is available.

What Goes Wrong in Practice

Consider what happens when you chunk a 60-line class with a 30-line window. The first chunk contains the class declaration, __init__, and half of the second method. The second chunk contains the rest of the second method and the start of the third. Neither chunk is a self-contained unit. An embedding model (a neural network that maps text to fixed-length numerical vectors for similarity search) trained on these chunks learns a representation of code fragments, not code concepts. A retrieval system that returns these chunks gives the agent code it cannot understand without the missing halves.

The solution is to chunk along syntactic boundaries defined by the language's parse tree. Functions, classes, and top-level assignment blocks are natural chunks because they reflect the programmer's intent about what constitutes a logical unit. Figure 11.2 illustrates the complete pipeline that implements this idea, from raw source files through syntax-aware chunking and summarization to a searchable chunk store.

2. Syntactic Chunking with Tree-sitter

When retrieval feeds a coding agent broken fragments, the agent generates code that references undefined variables, calls functions with wrong signatures, or silently drops logic from incomplete loops. Getting chunk boundaries right is the difference between an agent that fixes bugs and one that creates them.

Turning the idea of "chunk along syntactic boundaries" into working code requires a parser that understands every language your repository contains. Tree-sitter is an incremental parsing library that builds a concrete syntax tree (CST) for source code in over 100 languages. Unlike an abstract syntax tree (AST) produced by Python's ast module, a CST preserves every character of the original source, including whitespace and comments. This makes tree-sitter ideal for chunking: we can extract syntactically complete units and reconstruct the original code exactly. In short: Let the parse tree, not a token counter, decide where one unit of meaning ends and the next begins.

import tree_sitter_python as tspython
from tree_sitter import Language, Parser

# Initialize the Python parser
PY_LANGUAGE = Language(tspython.language())
parser = Parser(PY_LANGUAGE)

def parse_source(source: str) -> "tree_sitter.Tree":
    """Parse Python source code into a concrete syntax tree."""
    return parser.parse(bytes(source, "utf-8"))

# Parse a simple example
source = """
class DataLoader:
    def __init__(self, path: str):
        self.path = path
        self.data = None

    def load(self) -> list[dict]:
        import json
        with open(self.path) as f:
            self.data = json.load(f)
        return self.data

def preprocess(records: list[dict]) -> list[dict]:
    return [r for r in records if r.get("valid")]
"""

tree = parse_source(source)
root = tree.root_node
print(f"Root type: {root.type}")
print(f"Children: {[child.type for child in root.children]}")

With the syntax tree in hand, we walk it to extract top-level definitions as chunks. Each chunk preserves the complete syntactic unit: the full class with all its methods, or the full standalone function.

from dataclasses import dataclass

@dataclass
class CodeChunk:
    """A syntactically complete unit of code extracted from a source file."""
    file_path: str
    chunk_type: str          # "class", "function", "assignment", "import_block"
    name: str                # symbol name (class/function name, or variable name)
    start_line: int
    end_line: int
    content: str             # the raw source code
    token_estimate: int      # estimated token count

    @property
    def id(self) -> str:
        return f"{self.file_path}:{self.name}"

# Node types that represent top-level definitions in Python
CHUNK_NODE_TYPES = {
    "class_definition",
    "function_definition",
    "decorated_definition",
    "expression_statement",  # top-level assignments
}

def extract_chunks(source: str, file_path: str) -> list[CodeChunk]:
    """Extract syntactically complete chunks from Python source."""
    tree = parse_source(source)
    chunks = []

    for node in tree.root_node.children:
        if node.type not in CHUNK_NODE_TYPES:
            continue

        content = source[node.start_byte:node.end_byte]

        # Determine chunk type and name
        if node.type == "class_definition":
            name_node = node.child_by_field_name("name")
            chunk_type = "class"
        elif node.type == "function_definition":
            name_node = node.child_by_field_name("name")
            chunk_type = "function"
        elif node.type == "decorated_definition":
            # The actual definition is a child of the decorator wrapper
            inner = [c for c in node.children
                     if c.type in ("class_definition", "function_definition")]
            if inner:
                name_node = inner[0].child_by_field_name("name")
                chunk_type = "class" if inner[0].type == "class_definition" else "function"
            else:
                name_node = None
                chunk_type = "decorated"
        else:
            name_node = None
            chunk_type = "assignment"

        name = (source[name_node.start_byte:name_node.end_byte]
                if name_node else f"block_{node.start_point[0]}")

        chunks.append(CodeChunk(
            file_path=file_path,
            chunk_type=chunk_type,
            name=name,
            start_line=node.start_point[0] + 1,
            end_line=node.end_point[0] + 1,
            content=content,
            token_estimate=len(content.split()) * 2,  # rough estimate
        ))

    return chunks

# Demo
chunks = extract_chunks(source, "data_loader.py")
for chunk in chunks:
    print(f"  [{chunk.chunk_type}] {chunk.name} "
          f"(lines {chunk.start_line}-{chunk.end_line}, ~{chunk.token_estimate} tokens)")

3. Hierarchical Chunking

Real code has nested structure: a module contains classes, classes contain methods, methods contain blocks. A hierarchical chunking strategy preserves this nesting by producing chunks at multiple granularity levels, linked by parent-child relationships. When the retriever selects a method-level chunk, the packer can optionally include the parent class signature for context, spending only the tokens needed for the signature rather than the entire class body. Figure 11.2.1 illustrates hierarchical chunking with parent-child chunk relationships.

@dataclass
class HierarchicalChunk:
    """A chunk with parent-child relationships for multi-granularity retrieval."""
    chunk: CodeChunk
    children: list["HierarchicalChunk"]
    parent_id: str | None = None

    @property
    def signature(self) -> str:
        """Extract just the signature (first line + docstring) for compact representation."""
        lines = self.chunk.content.split("\n")
        sig_lines = [lines[0]]  # def/class line

        # Include docstring if present
        in_docstring = False
        for line in lines[1:]:
            stripped = line.strip()
            if stripped.startswith('"""') or stripped.startswith("'''"):
                sig_lines.append(line)
                if in_docstring or stripped.count('"""') >= 2 or stripped.count("'''") >= 2:
                    break
                in_docstring = True
            elif in_docstring:
                sig_lines.append(line)
            else:
                break

        return "\n".join(sig_lines)

def build_hierarchy(source: str, file_path: str) -> list[HierarchicalChunk]:
    """Build a hierarchical chunk tree from Python source."""
    tree = parse_source(source)
    result = []

    for node in tree.root_node.children:
        if node.type == "class_definition" or (
            node.type == "decorated_definition" and
            any(c.type == "class_definition" for c in node.children)
        ):
            # Get the class node (might be wrapped in decorator)
            class_node = node if node.type == "class_definition" else next(
                c for c in node.children if c.type == "class_definition"
            )
            class_name = source[class_node.child_by_field_name("name").start_byte:
                                class_node.child_by_field_name("name").end_byte]

            class_content = source[node.start_byte:node.end_byte]
            class_chunk = CodeChunk(
                file_path=file_path, chunk_type="class", name=class_name,
                start_line=node.start_point[0] + 1, end_line=node.end_point[0] + 1,
                content=class_content, token_estimate=len(class_content.split()) * 2,
            )

            # Extract methods as children
            children = []
            body = class_node.child_by_field_name("body")
            if body:
                for child in body.children:
                    if child.type in ("function_definition", "decorated_definition"):
                        func_node = child if child.type == "function_definition" else next(
                            (c for c in child.children if c.type == "function_definition"), None
                        )
                        if func_node is None:
                            continue
                        method_name = source[func_node.child_by_field_name("name").start_byte:
                                             func_node.child_by_field_name("name").end_byte]
                        method_content = source[child.start_byte:child.end_byte]
                        method_chunk = CodeChunk(
                            file_path=file_path, chunk_type="method",
                            name=f"{class_name}.{method_name}",
                            start_line=child.start_point[0] + 1,
                            end_line=child.end_point[0] + 1,
                            content=method_content,
                            token_estimate=len(method_content.split()) * 2,
                        )
                        children.append(HierarchicalChunk(
                            chunk=method_chunk, children=[], parent_id=class_chunk.id
                        ))

            result.append(HierarchicalChunk(
                chunk=class_chunk, children=children, parent_id=None
            ))

        elif node.type == "function_definition":
            func_name = source[node.child_by_field_name("name").start_byte:
                               node.child_by_field_name("name").end_byte]
            content = source[node.start_byte:node.end_byte]
            chunk = CodeChunk(
                file_path=file_path, chunk_type="function", name=func_name,
                start_line=node.start_point[0] + 1, end_line=node.end_point[0] + 1,
                content=content, token_estimate=len(content.split()) * 2,
            )
            result.append(HierarchicalChunk(chunk=chunk, children=[], parent_id=None))

    return result

4. Summarization Strategies

Chunking breaks code into retrievable units. Summarization compresses those units so that more of them fit in the context window. There are three levels of summarization, each trading fidelity for compression (see the Summarizer stage in Figure 11.2):

4.1 Signature Extraction

The lightest summarization extracts only the function or class signature: the name, parameters, type annotations, and return type. A 50-line function compresses to 1-2 lines. This is the approach used by integrated development environment (IDE) "outline" views, and it is often sufficient for an agent to decide whether a function is relevant and how to call it.

def extract_signature(chunk: CodeChunk) -> str:
    """Extract the signature (declaration + docstring) from a code chunk."""
    lines = chunk.content.strip().split("\n")
    signature_lines = []

    # Collect the definition line(s), handling multi-line signatures
    paren_depth = 0
    for line in lines:
        signature_lines.append(line)
        paren_depth += line.count("(") - line.count(")")
        if paren_depth <= 0 and (":" in line or "->" in line):
            break

    # Collect the docstring if present
    remaining = lines[len(signature_lines):]
    in_docstring = False
    for line in remaining:
        stripped = line.strip()
        if not in_docstring and (stripped.startswith('"""') or stripped.startswith("'''")):
            signature_lines.append(line)
            if stripped.count('"""') >= 2 or stripped.count("'''") >= 2:
                break  # single-line docstring
            in_docstring = True
        elif in_docstring:
            signature_lines.append(line)
            if '"""' in stripped or "'''" in stripped:
                break
        else:
            break

    return "\n".join(signature_lines)

# Compression ratio example
full_code = '''def compute_attention(
    queries: torch.Tensor,
    keys: torch.Tensor,
    values: torch.Tensor,
    mask: torch.Tensor | None = None,
    dropout_p: float = 0.0,
) -> torch.Tensor:
    """Compute scaled dot-product attention.

    Args:
        queries: (batch, heads, seq_len, d_k)
        keys: (batch, heads, seq_len, d_k)
        values: (batch, heads, seq_len, d_v)
        mask: optional attention mask
        dropout_p: dropout probability

    Returns:
        Attention output tensor of shape (batch, heads, seq_len, d_v)
    """
    d_k = queries.size(-1)
    scores = torch.matmul(queries, keys.transpose(-2, -1)) / math.sqrt(d_k)
    if mask is not None:
        scores = scores.masked_fill(mask == 0, float("-inf"))
    weights = torch.softmax(scores, dim=-1)
    if dropout_p > 0.0:
        weights = torch.dropout(weights, p=dropout_p, train=True)
    return torch.matmul(weights, values)'''

chunk = CodeChunk("attention.py", "function", "compute_attention", 1, 25,
                  full_code, len(full_code.split()) * 2)
sig = extract_signature(chunk)
ratio = len(full_code) / len(sig)
print(f"Compression ratio: {ratio:.1f}x")

4.2 Docstring and Comment Extraction

When signatures are too terse, extracting docstrings and inline comments provides a middle ground. Comments encode the programmer's intent ("why"), which is often more useful for relevance scoring than the implementation details ("how"). A function with the comment # Apply Kruskal's algorithm for minimum spanning tree is immediately recognizable as graph-related, even if the variable names are opaque.

import ast

def extract_docstrings(source: str) -> dict[str, str]:
    """Extract all docstrings from a Python module, keyed by qualified name."""
    tree = ast.parse(source)
    docstrings = {}

    for node in ast.walk(tree):
        if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef)):
            docstring = ast.get_docstring(node)
            if docstring:
                docstrings[node.name] = docstring

    # Module-level docstring
    module_docstring = ast.get_docstring(tree)
    if module_docstring:
        docstrings["__module__"] = module_docstring

    return docstrings

def extract_comments(source: str) -> list[str]:
    """Extract all comments from Python source code."""
    comments = []
    for line in source.split("\n"):
        stripped = line.strip()
        if stripped.startswith("#"):
            comments.append(stripped[1:].strip())
        elif "#" in stripped:
            # Inline comment (simple heuristic; ignores # inside strings)
            idx = stripped.rfind("#")
            comment = stripped[idx + 1:].strip()
            if comment:
                comments.append(comment)
    return comments

4.3 LLM-Generated Summaries

Signatures and comments capture structure and intent, but they cannot explain complex interactions between components or describe undocumented edge cases. The most powerful (and most expensive) summarization strategy uses an LLM to generate natural-language descriptions of code chunks. The LLM reads the full implementation and produces a 2-3 sentence summary covering purpose, algorithm, key dependencies, and edge cases. These summaries excel at embedding-based retrieval because natural-language queries match natural-language summaries more easily than raw code. The trade-off: each chunk requires an LLM call during indexing.

SUMMARY_PROMPT = """Summarize this code in 2-3 sentences. State:
1. What the code does (purpose and algorithm)
2. Key inputs and outputs (types and shapes)
3. Important dependencies or side effects

Code:
```python
{code}
```

Summary:"""

def generate_summary(code: str, llm_client) -> str:
    """Generate an LLM summary for a code chunk.

    This is called during indexing (offline), not during retrieval.
    Cost: ~500 input tokens + ~100 output tokens per chunk.
    """
    prompt = SUMMARY_PROMPT.format(code=code)
    response = llm_client.complete(prompt, max_tokens=150, temperature=0.0)
    return response.text.strip()

# For a 1,500-chunk codebase at $0.003/1K input tokens (circa 2023):
# Cost estimate: 1500 chunks * 500 tokens * $0.003/1K = $2.25
# As of 2025, lightweight models such as GPT-4o-mini and Claude 3.5 Haiku
# have reduced per-token costs further, making this even cheaper.
# This is a one-time indexing cost; re-index only when code changes.

5. Building a Chunk Store

With chunking and summarization in place, we assemble them into a persistent chunk store: a structured database of all chunks in the repository, their summaries, and their relationships. The chunk store is the foundation that the retrieval system of Section 11.3 queries against.

import json
import sqlite3
from pathlib import Path

class ChunkStore:
    """Persistent storage for code chunks and their summaries."""

    def __init__(self, db_path: str):
        self.conn = sqlite3.connect(db_path)
        self._create_tables()

    def _create_tables(self):
        self.conn.executescript("""
            CREATE TABLE IF NOT EXISTS chunks (
                id TEXT PRIMARY KEY,
                file_path TEXT NOT NULL,
                chunk_type TEXT NOT NULL,
                name TEXT NOT NULL,
                start_line INTEGER,
                end_line INTEGER,
                content TEXT NOT NULL,
                token_estimate INTEGER,
                parent_id TEXT,
                FOREIGN KEY (parent_id) REFERENCES chunks(id)
            );
            CREATE TABLE IF NOT EXISTS summaries (
                chunk_id TEXT PRIMARY KEY,
                signature TEXT,
                docstring TEXT,
                llm_summary TEXT,
                FOREIGN KEY (chunk_id) REFERENCES chunks(id)
            );
            CREATE INDEX IF NOT EXISTS idx_chunks_file ON chunks(file_path);
            CREATE INDEX IF NOT EXISTS idx_chunks_type ON chunks(chunk_type);
            CREATE INDEX IF NOT EXISTS idx_chunks_parent ON chunks(parent_id);
        """)

    def upsert_chunk(self, chunk: CodeChunk, parent_id: str | None = None):
        """Insert or update a chunk in the store."""
        self.conn.execute(
            """INSERT OR REPLACE INTO chunks
               (id, file_path, chunk_type, name, start_line, end_line,
                content, token_estimate, parent_id)
               VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)""",
            (chunk.id, chunk.file_path, chunk.chunk_type, chunk.name,
             chunk.start_line, chunk.end_line, chunk.content,
             chunk.token_estimate, parent_id)
        )

    def upsert_summary(self, chunk_id: str, signature: str = "",
                       docstring: str = "", llm_summary: str = ""):
        """Insert or update summary data for a chunk."""
        self.conn.execute(
            """INSERT OR REPLACE INTO summaries
               (chunk_id, signature, docstring, llm_summary)
               VALUES (?, ?, ?, ?)""",
            (chunk_id, signature, docstring, llm_summary)
        )

    def get_chunk(self, chunk_id: str) -> dict | None:
        """Retrieve a chunk and its summary by ID."""
        row = self.conn.execute(
            """SELECT c.*, s.signature, s.docstring, s.llm_summary
               FROM chunks c
               LEFT JOIN summaries s ON c.id = s.chunk_id
               WHERE c.id = ?""",
            (chunk_id,)
        ).fetchone()
        if row is None:
            return None
        cols = ["id", "file_path", "chunk_type", "name", "start_line",
                "end_line", "content", "token_estimate", "parent_id",
                "signature", "docstring", "llm_summary"]
        return dict(zip(cols, row))

    def get_children(self, parent_id: str) -> list[dict]:
        """Get all child chunks of a parent (e.g., methods of a class)."""
        rows = self.conn.execute(
            "SELECT * FROM chunks WHERE parent_id = ?", (parent_id,)
        ).fetchall()
        cols = ["id", "file_path", "chunk_type", "name", "start_line",
                "end_line", "content", "token_estimate", "parent_id"]
        return [dict(zip(cols, row)) for row in rows]

    def commit(self):
        self.conn.commit()

    def close(self):
        self.conn.close()

The indexing pipeline ties everything together: walk the repository, parse each file, extract hierarchical chunks, generate summaries at each level, and persist (upsert) each record, meaning insert it if new or update it if it already exists.

def index_repository(repo_root: str, db_path: str,
                     extensions: set[str] = {".py"},
                     llm_client=None) -> dict:
    """Index an entire repository into the chunk store.

    Args:
        repo_root: path to the repository root
        db_path: path to the SQLite database
        extensions: file extensions to index
        llm_client: optional LLM client for generating summaries

    Returns:
        Statistics about the indexing run.
    """
    store = ChunkStore(db_path)
    stats = {"files": 0, "chunks": 0, "summaries": 0}

    for path in Path(repo_root).rglob("*"):
        if path.suffix not in extensions or not path.is_file():
            continue
        if any(part.startswith(".") for part in path.parts):
            continue  # skip hidden directories

        try:
            source = path.read_text(encoding="utf-8")
        except (UnicodeDecodeError, PermissionError):
            continue

        rel_path = str(path.relative_to(repo_root))
        hierarchy = build_hierarchy(source, rel_path)
        stats["files"] += 1

        for h_chunk in hierarchy:
            store.upsert_chunk(h_chunk.chunk, parent_id=h_chunk.parent_id)
            stats["chunks"] += 1

            # Generate summaries
            sig = extract_signature(h_chunk.chunk)
            docstrings = extract_docstrings(source)
            doc = docstrings.get(h_chunk.chunk.name, "")

            llm_summary = ""
            if llm_client and h_chunk.chunk.token_estimate > 50:
                llm_summary = generate_summary(h_chunk.chunk.content, llm_client)
                stats["summaries"] += 1

            store.upsert_summary(h_chunk.chunk.id, sig, doc, llm_summary)

            # Index children (methods)
            for child in h_chunk.children:
                store.upsert_chunk(child.chunk, parent_id=h_chunk.chunk.id)
                child_sig = extract_signature(child.chunk)
                child_doc = docstrings.get(
                    child.chunk.name.split(".")[-1], "")
                store.upsert_summary(child.chunk.id, child_sig, child_doc, "")
                stats["chunks"] += 1

    store.commit()
    store.close()
    return stats

from llama_index.core.node_parser import CodeSplitter

splitter = CodeSplitter(language="python", chunk_lines=60, chunk_lines_overlap=10)
nodes = splitter.get_nodes_from_documents(documents)

class Dog:
    def __init__(self, name):
        self.name = name

    def bark(self):
        return "Woof!"

def adopt(shelter, dog):
    shelter.append(dog)

Exercises

1. (Conceptual) Explain why fixed-size chunking with overlap is particularly problematic for Python (as opposed to, say, Java or Go). Consider how Python uses indentation for scoping and what happens when a chunk boundary falls mid-indent.
2. (Coding) Extend the extract_chunks function in Listing 11.7 to handle JavaScript/TypeScript files. You will need the tree-sitter-javascript or tree-sitter-typescript grammar packages. Identify the node types that correspond to top-level definitions in those languages (hint: function_declaration, class_declaration, lexical_declaration).
3. (Analysis) Index a medium-sized open-source Python project (e.g., Flask or FastAPI) using the pipeline from Listing 11.13 (without LLM summaries). Report the number of files, chunks, and the distribution of chunk sizes (in tokens). What percentage of chunks are under 200 tokens? Under 500? What does this distribution suggest about the optimal embedding model input length?