"I have read 47 million papers. I have understood none of them. But I can tell you which paragraph answers your question, and that turns out to be enough."
A Vector Database With Impostor Syndrome
Overview
Large language models know a great deal, but they do not know what they do not know. When a researcher asks about a specific protein interaction, a recent clinical trial, or a niche experimental protocol, the model must either retrieve the relevant evidence or confabulate an answer that sounds authoritative but is wrong. Retrieval Augmented Generation (RAG) solves this by grounding every generated response in retrieved documents, turning the language model from an unreliable oracle into a well-sourced research assistant.
This chapter builds RAG systems designed specifically for scientific discovery. We begin with the architecture (Section 37.1): the retriever that finds relevant passages, the reranker that sharpens relevance, and the generator that synthesizes a cited answer. We cover embedding models, vector databases (Qdrant, pgvector), cross-encoder rerankers, and the tradeoffs between long-context windows and retrieval-based approaches. Section 37.2 tackles the hardest problem in scientific RAG: ensuring that generated answers are faithful to their sources. We formalize hallucination taxonomies, build citation verification pipelines, and measure faithfulness with automated metrics. Finally, Section 37.3 assembles these components into a complete multi-hop research copilot that decomposes complex questions, retrieves evidence across a paper corpus, and produces cited, verifiable answers with faithfulness scores.
Prerequisites
This chapter assumes familiarity with Chapter 3 (Knowledge Representation), which introduced embeddings and similarity, and Chapter 36 (Literature Mining), which covered document parsing and information extraction from scientific text. You should be comfortable with Python, basic linear algebra (dot products, cosine similarity), and have an understanding of how language models generate text. Familiarity with the Anthropic API from Chapter 10 is helpful but not required; we introduce every API call from scratch.
What You Will Learn
- Design a RAG pipeline with retriever, reranker, and generator stages, choosing the right component at each level.
- Deploy vector search with Qdrant and pgvector, and understand when each backend is appropriate.
- Evaluate retrieval quality with Recall@k, Mean Reciprocal Rank, and Normalized Discounted Cumulative Gain.
- Implement cross-encoder reranking to improve precision without re-embedding your entire corpus.
- Detect and prevent hallucinations using citation verification and automated faithfulness scoring.
- Build a multi-hop research copilot that decomposes complex scientific questions, retrieves evidence, and produces cited answers.
Sections
37.1 RAG Architecture
Retriever, reranker, generator: the three-stage pipeline. Embedding models, vector databases, cross-encoder reranking, query planning, and the long-context versus retrieval tradeoff.
37.2 Evidence Grounding and Hallucination Control
Faithfulness as a first-class metric. Hallucination taxonomies, citation verification, PaperQA2, automated faithfulness scoring, and building trust in generated scientific answers.
37.3 Building a Research Copilot
A hands-on recipe: multi-hop query decomposition, parallel evidence retrieval, cited answer synthesis, and end-to-end faithfulness evaluation over a scientific paper corpus.
What's Next
RAG grounds language model answers in retrieved documents, but documents are flat: they contain facts without explicit relationships. Chapter 38: Knowledge Graph Discovery introduces structured knowledge representations where entities and relationships form a navigable graph. The retrieval techniques from this chapter combine powerfully with knowledge graphs: you can retrieve candidate entities via embedding search, then traverse graph edges to find multi-hop connections that no single document contains. The faithfulness metrics we developed here will reappear when we validate claims extracted from graph traversals.
Bibliography
Foundational Papers
The paper that introduced RAG as a paradigm, combining a pre-trained retriever (DPR) with a seq2seq generator and training them jointly.
Demonstrates a RAG agent that achieves superhuman performance on scientific question answering by combining retrieval, citation verification, and iterative refinement.
Introduced Dense Passage Retrieval (DPR), showing that learned dense representations outperform BM25 for open-domain QA retrieval.
Established cross-encoder reranking as the standard second stage in neural retrieval pipelines, improving precision dramatically over bi-encoder retrieval alone.
Retrieval and Evaluation
A comprehensive survey covering naive RAG, advanced RAG, and modular RAG architectures, with a taxonomy of retrieval, augmentation, and generation strategies.
Introduces the RAGAS framework for evaluating RAG pipelines along four dimensions: faithfulness, answer relevancy, context precision, and context recall.
Tools & Libraries
A high-performance vector database with filtering, payload indexing, and hybrid search. Supports both in-memory and on-disk modes for corpora of any size.
Adds vector similarity search to PostgreSQL, allowing teams to store embeddings alongside relational data without a separate vector database.
The standard library for computing dense text embeddings, with pre-trained models optimized for semantic search and clustering.
Reference for the Claude API used in our generator stage, including structured output, citations, and tool use for research copilots.
Open-source implementation of the PaperQA2 system, providing a complete pipeline for question answering over scientific papers with citation verification.
Hallucination and Faithfulness
Comprehensive taxonomy of hallucination types in LLMs, covering intrinsic vs. extrinsic hallucinations and their detection methods.
Introduces atomic fact decomposition and per-fact verification as a fine-grained alternative to coarse factuality scores.