"I cited a paper that does not exist, from a journal that was never published, by an author who would be very surprised to learn of their contribution. Otherwise, the answer was excellent."
A Citation Manager Who Has Seen Things
Retrieval reduces hallucination, but it does not eliminate it. A RAG system can retrieve the right passage and still generate an answer that misrepresents what the passage says, omits critical qualifications, or invents details that appear nowhere in the evidence. For scientific applications, where a fabricated statistic or a misattributed finding can derail an entire line of research, faithfulness is not a nice-to-have quality; it is the central engineering challenge. This section formalizes what faithfulness means, categorizes the ways RAG systems fail, introduces PaperQA2 as a state-of-the-art approach to verifiable scientific QA, and builds automated pipelines for detecting and measuring unfaithful generation.
1. A Taxonomy of RAG Hallucinations
In 2023, a biomedical research team spent two weeks chasing a promising lead from an AI-generated literature review, only to discover that the pivotal citation, complete with a plausible title, a real journal name, and a convincing DOI, pointed to a paper that had never been published.
What. A hallucination in RAG is any generated content that cannot be verified against the retrieved evidence. This definition is deliberately strict: even a true statement is a hallucination if it does not appear in the provided sources, because the system's contract is to answer from evidence, not from memory.
Evidence grounding requires every claim in a generated answer to trace back to a specific passage in the retrieved documents, so that no statement rests on the model's parametric memory (the knowledge baked into its weights during training) alone. Without it, a RAG system offers no meaningful advantage over a plain language model: the retrieved documents become decoration rather than evidence, and the user cannot distinguish verified facts from plausible fabrications. The mechanism works as follows: after generation, an entailment checker (a model or classifier that determines whether a source passage logically supports a given claim) aligns each sentence (or atomic claim) to a source passage and flags or removes any claim that lacks support. Use evidence grounding whenever the downstream consumer needs to audit or reproduce the reasoning behind an answer; for exploratory brainstorming where creativity matters more than traceability, lighter constraints may suffice.
Evidence grounding is not a retrieval problem; it is a generation-verification problem that must be engineered separately on top of retrieval.
Why. Different hallucination types require different mitigation strategies. A system that fabricates citations needs a different fix than one that subtly distorts reported statistics. Categorizing failures is the first step toward systematic prevention.
How. We distinguish four hallucination types, ordered from most to least obvious.
Type 1: Fabricated citations. The model invents a paper title, author list, or digital object identifier (DOI) that does not exist. This is the most dangerous hallucination type in scientific RAG because it creates a false evidence trail that a reader might follow. Example: citing "Smith et al. (2023)" when no such paper exists in the retrieved evidence or in the literature.
Type 2: Unsupported claims. The model generates a factual statement that does not appear in any retrieved passage. The statement may be true (drawn from the model's training data) or false, but either way it violates the evidence-grounding contract. Example: reporting a specific p-value that appears nowhere in the retrieved text.
Type 3: Distorted evidence. The model references a real passage but misrepresents its content. This includes flipping the direction of a finding ("increased" becomes "decreased"), omitting critical qualifiers ("in vitro" or "in a small cohort"), or combining results from different studies as if they came from one. This is the subtlest and most insidious type.
Type 4: Incomplete attribution. The model produces a correct, evidence-supported statement but fails to cite the source, or cites the wrong source. Less dangerous than the other types, but it undermines verifiability.
Common Misconception
A frequent misconception is that adding retrieval to a language model automatically makes its answers trustworthy, as if the presence of source documents guarantees the generated text faithfully reflects them. This is wrong: retrieval provides the model with evidence, but the generation step can still ignore, distort, or supplement that evidence with fabricated details. Faithfulness must be measured and enforced as a separate engineering concern on top of retrieval, not assumed as a byproduct of it.
A counterintuitive finding from the RAG evaluation literature: retrieval can sometimes increase hallucination rates compared to closed-book generation. This happens when the retriever surfaces irrelevant or contradictory passages. The model, instructed to answer from the provided context, tries to force an answer from unsuitable evidence and produces distorted or fabricated connections. The implication is that retrieval quality and generation faithfulness are coupled: improving one without the other can make the system worse. A high-precision retriever with a faithful generator outperforms a high-recall retriever with an unfaithful one.
Mental Model
Think of a RAG system like a student writing an open-book exam. Having the textbook on the desk (retrieval) does not prevent the student from writing an incorrect answer; the student can misread a paragraph, copy a number from the wrong table, or fill in a gap from a half-remembered lecture instead of flipping to the right page. A diligent student checks each sentence against the book before moving on, and an instructor who grades the exam verifies that every cited page actually says what the student claims. Evidence grounding is that verification step: not whether the book was on the desk, but whether every sentence in the essay points to a real passage that supports it.
2. Measuring Faithfulness
Recognizing the four hallucination types is a necessary first step. Prevention, however, requires detecting them automatically and at scale. That demands a formal metric for how well a generated answer stays faithful to its evidence.
Faithfulness quantifies the degree to which a generated answer is supported by the retrieved evidence. Unlike factual accuracy (which requires ground truth), faithfulness can be evaluated automatically by comparing the answer against the provided context.
Without a way to measure faithfulness automatically, teams discover hallucinations only when a downstream researcher follows a false lead, sometimes weeks after the answer was generated. Automated scoring turns a delayed, costly failure into an immediate, catchable signal.
The core idea is atomic claim decomposition. Break the generated answer into atomic claims (single, verifiable statements), then check each claim against the evidence passages. The faithfulness score is the fraction of claims that are supported. Figure 37.2.1 illustrates atomic claim decomposition and faithfulness verification pipeline.
Figure 37.3 illustrates this pipeline from end to end: a generated answer flows through claim decomposition, per-claim verification against evidence passages, and aggregation into a single faithfulness score.
FActScore (a benchmark that decomposes biographies into atomic facts and checks each against a knowledge source) pioneered this approach, and the Retrieval Augmented Generation Assessment (RAGAS) evaluation framework adopted it. In short: if you cannot decompose an answer into atomic claims and trace each one back to a source passage, you do not have evidence; you have a guess wearing a lab coat.
import anthropic
import json
client = anthropic.Anthropic()
def extract_atomic_claims(answer: str) -> list[str]:
"""Decompose a generated answer into atomic, verifiable claims."""
response = client.messages.create(
model="claude-sonnet-4-20250514",
max_tokens=2048,
messages=[{
"role": "user",
"content": f"""Decompose the following text into atomic claims.
Each atomic claim should be a single, self-contained factual statement
that can be independently verified. Do not include opinions, hedging
language, or meta-statements about the answer itself.
Text: {answer}
Return a JSON array of claim strings. Example:
["Claim 1", "Claim 2", "Claim 3"]"""
}],
)
return json.loads(response.content[0].text)
def verify_claim(claim: str, evidence_passages: list[str]) -> dict:
"""Check whether a single claim is supported by the evidence."""
evidence_text = "\n\n".join(
f"[Passage {i+1}]: {p}"
for i, p in enumerate(evidence_passages)
)
response = client.messages.create(
model="claude-sonnet-4-20250514",
max_tokens=512,
messages=[{
"role": "user",
"content": f"""Determine whether the following claim is
supported by the evidence passages.
Claim: {claim}
Evidence:
{evidence_text}
Respond with a JSON object:
{{
"verdict": "supported" | "not_supported" | "contradicted",
"supporting_passage": ,
"reasoning": ""
}}"""
}],
)
return json.loads(response.content[0].text)
def compute_faithfulness(
answer: str,
evidence_passages: list[str],
) -> dict:
"""Compute faithfulness score for a generated answer."""
claims = extract_atomic_claims(answer)
results = [verify_claim(c, evidence_passages) for c in claims]
supported = sum(1 for r in results if r["verdict"] == "supported")
contradicted = sum(1 for r in results if r["verdict"] == "contradicted")
return {
"faithfulness_score": supported / len(claims) if claims else 0.0,
"total_claims": len(claims),
"supported": supported,
"not_supported": len(claims) - supported - contradicted,
"contradicted": contradicted,
"details": [
{"claim": c, **r} for c, r in zip(claims, results)
],
}
# Example: evaluate faithfulness of a RAG-generated answer
answer = """EGFR mutations are found in approximately 15-20% of non-small
cell lung cancer (NSCLC) patients. These mutations confer sensitivity to
tyrosine kinase inhibitors (TKIs) such as erlotinib and gefitinib [Source 1].
The T790M resistance mutation emerges in about 60% of patients after
initial TKI treatment, leading to disease progression."""
evidence = [
"EGFR mutations in non-small cell lung cancer confer sensitivity "
"to tyrosine kinase inhibitors such as erlotinib and gefitinib.",
]
result = compute_faithfulness(answer, evidence)
print(f"Faithfulness: {result['faithfulness_score']:.2f}")
print(f"Claims: {result['total_claims']} total, "
f"{result['supported']} supported, "
f"{result['contradicted']} contradicted")
for detail in result["details"]:
print(f" [{detail['verdict']}] {detail['claim']}")
print(f" Reason: {detail['reasoning']}")
3. PaperQA2: Verifiable Scientific Question Answering
PaperQA2, developed by Future House, represents the current state of the art in scientific RAG. It achieves what its developers report as superhuman accuracy on scientific question answering benchmarks, matching or exceeding domain expert performance on the LitQA2 benchmark while producing fully cited, verifiable answers by combining several techniques that address the hallucination types cataloged above.
What. PaperQA2 is a RAG agent (not just a pipeline) that iteratively searches, reads, and synthesizes answers from scientific papers. Unlike a single-pass retrieve-then-generate pipeline, PaperQA2 can decide that its initial retrieval was insufficient, reformulate queries, fetch additional papers, and verify its own citations before producing a final answer.
Why. A single-pass pipeline locks in its retrieved passages before generation begins; if the retriever missed a critical paper, the generator cannot recover. PaperQA2's agentic loop recognizes these gaps and fills them by reformulating queries and fetching additional evidence.
How. The system operates through a sequence of tool calls:
- Paper search: queries a scientific search engine (Semantic Scholar, CrossRef) to find relevant papers.
- Gather evidence: retrieves and chunks the full text of found papers, embedding and storing them for similarity search.
- Generate answer: synthesizes a cited answer from the gathered evidence, with each claim linked to a specific passage.
- Citation verification: checks each citation against the source text, flagging any claim that is not directly supported.
from paperqa import Settings, ask
# Configure PaperQA2
settings = Settings(
llm="claude-sonnet-4-20250514",
summary_llm="claude-sonnet-4-20250514",
embedding="sentence-transformers/all-MiniLM-L6-v2",
answer=Settings.AnswerSettings(
evidence_k=10, # Number of evidence passages to gather
answer_max_sources=5, # Maximum sources in final answer
),
)
# Ask a scientific question
# PaperQA2 will search, retrieve papers, gather evidence, and generate
answer = await ask(
"What are the mechanisms of resistance to EGFR tyrosine "
"kinase inhibitors in non-small cell lung cancer?",
settings=settings,
)
print(f"Answer: {answer.answer}")
print(f"\nCitations ({len(answer.references)}):")
for ref in answer.references:
print(f" - {ref.citation}")
print(f"\nEvidence quality: {answer.quality_score:.2f}")
paperqa package (now distributed as paper-qa on PyPI) has evolved rapidly; as of 2025, its Settings API and async entry points may differ from the snippet shown here, so consult the current Future House documentation for exact parameter names.A computational chemist asks: "What is the binding affinity of compound X to the JAK2 kinase, and how does it compare to ruxolitinib?" A single-pass RAG system retrieves passages mentioning compound X and passages mentioning ruxolitinib, but the retrieved passages report binding affinities in different units (IC50 vs. Ki) measured under different assay conditions. The generator, lacking the context to recognize this incompatibility, produces a direct numerical comparison that is scientifically meaningless. PaperQA2's agentic loop, after gathering initial evidence, can recognize that the comparison requires matched assay conditions, reformulate its search to find head-to-head comparisons, and either report a valid comparison or explicitly state that the available evidence does not support a direct comparison. This ability to recognize and communicate the limits of the evidence is what makes PaperQA2 suitable for scientific use cases where a confidently wrong answer is worse than no answer.
4. Citation Verification
Citation verification is the process of checking whether each citation in a generated answer actually supports the claim it is attached to. This is distinct from faithfulness scoring: faithfulness asks "is the claim supported by any evidence?", while citation verification asks "is the claim supported by the specific source cited?"
The distinction matters because models frequently attach the wrong citation to a correct claim. A claim about protein folding might cite a paper about protein expression, and while the claim may be true and supported by another passage in the context, the specific citation is misleading.
def verify_citations(
answer_with_citations: str,
sources: dict[str, str], # {source_id: passage_text}
) -> list[dict]:
"""Verify that each citation supports its associated claim.
Args:
answer_with_citations: Answer text with [Source N] markers.
sources: Mapping from source identifiers to passage text.
Returns:
List of verification results per citation.
"""
import re
# Extract claims with their citations
# Pattern: sentence ending with [Source N] or [Source N, Source M]
citation_pattern = r'([^.]+\[Source \d+(?:,\s*Source \d+)*\]\.?)'
cited_segments = re.findall(citation_pattern, answer_with_citations)
results = []
for segment in cited_segments:
# Extract cited source IDs
source_ids = re.findall(r'Source (\d+)', segment)
# Remove citation markers to get the claim text
claim = re.sub(r'\[Source \d+(?:,\s*Source \d+)*\]', '', segment).strip()
for source_id in source_ids:
source_key = f"Source {source_id}"
if source_key not in sources:
results.append({
"claim": claim,
"cited_source": source_key,
"verdict": "source_not_found",
"reasoning": f"{source_key} does not exist in the "
f"provided evidence.",
})
continue
# Verify this specific citation supports this specific claim
verification = verify_claim(claim, [sources[source_key]])
results.append({
"claim": claim,
"cited_source": source_key,
"verdict": verification["verdict"],
"reasoning": verification["reasoning"],
})
return results
# Example usage
answer_text = (
"EGFR mutations confer sensitivity to TKIs such as erlotinib "
"and gefitinib [Source 1]. The T790M gatekeeper mutation is the "
"most common mechanism of acquired resistance [Source 2]. "
"Osimertinib overcomes T790M resistance with a median "
"progression-free survival of 18.9 months [Source 3]."
)
sources = {
"Source 1": "EGFR mutations in non-small cell lung cancer confer "
"sensitivity to tyrosine kinase inhibitors such as "
"erlotinib and gefitinib.",
"Source 2": "TP53 is the most frequently mutated gene in human "
"cancers, with loss-of-function mutations found in "
"over 50% of tumors.",
# Source 3 intentionally missing
}
verifications = verify_citations(answer_text, sources)
for v in verifications:
status = "PASS" if v["verdict"] == "supported" else "FAIL"
print(f" [{status}] {v['cited_source']}: {v['claim'][:60]}...")
print(f" {v['reasoning']}")
5. The RAGAS Evaluation Framework
Citation verification checks individual claims against their cited sources, but a production RAG system also needs to evaluate whether the retriever found the right passages in the first place and whether the answer actually addressed the user's question.
Evaluating a RAG system requires measuring multiple dimensions simultaneously. A system might have excellent retrieval but poor generation, or faithful generation but low recall. The RAGAS framework provides four complementary metrics that together give a complete picture of system quality.
Faithfulness (already covered): the fraction of generated claims supported by the retrieved context. Measures generation quality independent of retrieval.
Answer Relevancy: the degree to which the generated answer addresses the original question. A faithful answer that discusses irrelevant aspects of the retrieved evidence scores low on relevancy. Measured by generating questions from the answer and computing their semantic similarity to the original question:
$$\text{Answer Relevancy} = \frac{1}{N} \sum_{i=1}^{N} \text{sim}(\mathbf{e}_q, \mathbf{e}_{q_i^{\text{gen}}})$$where \(q_i^{\text{gen}}\) are questions generated from the answer and \(\mathbf{e}\) denotes embeddings.
Context Precision: the fraction of retrieved passages that are actually relevant to answering the question. High precision means the retriever is not wasting context window space on irrelevant passages.
Checkpoint
So far: faithfulness measures whether the generator's claims are backed by retrieved passages, answer relevancy measures whether the answer addresses the original question, and context precision measures whether the retriever avoided surfacing irrelevant passages.
Context Recall: the fraction of the ground-truth answer's claims that are attributable to the retrieved context. High recall means the retriever found all the evidence needed to answer the question fully.
from dataclasses import dataclass
@dataclass
class RAGEvaluation:
"""Complete RAGAS-style evaluation of a RAG response."""
faithfulness: float # Claims supported / total claims
answer_relevancy: float # Answer addresses the question
context_precision: float # Retrieved passages are relevant
context_recall: float # Needed evidence was retrieved
@property
def harmonic_mean(self) -> float:
"""Overall score as harmonic mean of all four metrics."""
scores = [self.faithfulness, self.answer_relevancy,
self.context_precision, self.context_recall]
if any(s == 0 for s in scores):
return 0.0
return len(scores) / sum(1.0 / s for s in scores)
def summary(self) -> str:
lines = [
f"Faithfulness: {self.faithfulness:.3f}",
f"Answer Relevancy: {self.answer_relevancy:.3f}",
f"Context Precision: {self.context_precision:.3f}",
f"Context Recall: {self.context_recall:.3f}",
f"Overall (H-mean): {self.harmonic_mean:.3f}",
]
return "\n".join(lines)
def evaluate_context_precision(
question: str,
retrieved_passages: list[str],
client: anthropic.Anthropic,
) -> float:
"""Evaluate what fraction of retrieved passages are relevant."""
relevant_count = 0
for passage in retrieved_passages:
response = client.messages.create(
model="claude-sonnet-4-20250514",
max_tokens=64,
messages=[{
"role": "user",
"content": f"""Is this passage relevant to answering
the following question? Respond with only "yes" or "no".
Question: {question}
Passage: {passage}""",
}],
)
if response.content[0].text.strip().lower() == "yes":
relevant_count += 1
return relevant_count / len(retrieved_passages) if retrieved_passages else 0.0
def evaluate_rag_response(
question: str,
answer: str,
retrieved_passages: list[str],
ground_truth: str | None = None,
) -> RAGEvaluation:
"""Run full RAGAS evaluation on a RAG response."""
client = anthropic.Anthropic()
# Faithfulness
faith_result = compute_faithfulness(answer, retrieved_passages)
# Context precision
ctx_precision = evaluate_context_precision(
question, retrieved_passages, client
)
# Answer relevancy (simplified: use embedding similarity)
from sentence_transformers import SentenceTransformer
embed_model = SentenceTransformer("BAAI/bge-large-en-v1.5")
q_emb = embed_model.encode([question], normalize_embeddings=True)
a_emb = embed_model.encode([answer], normalize_embeddings=True)
relevancy = float((q_emb @ a_emb.T)[0, 0])
# Context recall (requires ground truth)
ctx_recall = 0.0
if ground_truth:
gt_claims = extract_atomic_claims(ground_truth)
supported = sum(
1 for c in gt_claims
if verify_claim(c, retrieved_passages)["verdict"] == "supported"
)
ctx_recall = supported / len(gt_claims) if gt_claims else 0.0
return RAGEvaluation(
faithfulness=faith_result["faithfulness_score"],
answer_relevancy=max(0.0, relevancy), # Clamp negatives
context_precision=ctx_precision,
context_recall=ctx_recall,
)
The ragas library provides all four metrics with a single function call, using LLM-based evaluation internally. Our from-scratch implementation spans ~80 lines and gives you full control over the evaluation prompts, but for benchmarking a pipeline quickly, the library reduces this to five lines (plus data preparation). It also provides additional metrics like answer correctness and aspect critique that we did not implement.
from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy
from ragas.metrics import context_precision, context_recall
from datasets import Dataset
# Prepare evaluation data
eval_data = Dataset.from_dict({
"question": [question],
"answer": [answer],
"contexts": [[p for p in retrieved_passages]],
"ground_truth": [ground_truth],
})
result = evaluate(eval_data, metrics=[
faithfulness, answer_relevancy,
context_precision, context_recall,
])
print(result) # DataFrame with all scores
Note: the import paths shown above reflect the RAGAS v0.1 API. As of 2024, RAGAS v0.2 restructured its metric classes and evaluation interface; consult the current documentation for updated usage, though the underlying four-metric framework remains the same.
6. Strategies for Reducing Hallucination
Five concrete strategies reduce hallucination in scientific RAG systems, ordered by implementation effort.
6.1 Constrained Generation Prompting
The simplest intervention is prompt engineering. The generator prompt from Section 37.1 (Listing 37.8) already includes instructions to cite sources and acknowledge gaps. Additional constraints that improve faithfulness include:
- Enumerate before synthesize: ask the model to first list the key facts from each source, then synthesize them into a coherent answer. This forces attention to the evidence before generation begins.
- Self-check instruction: append "Before finalizing, verify each citation by re-reading the cited passage and confirming it supports the claim" to the prompt.
- Confidence calibration: ask the model to flag claims it is uncertain about with a confidence level, making potential hallucinations visible to the reader.
6.2 Post-Generation Verification
Run the citation verification pipeline (Listing 37.12) on every generated answer. Flag or remove claims that fail verification. For production systems, this adds one large language model (LLM) call per citation but catches the most egregious errors before they reach the user.
6.3 Retrieval Quality Gates
Do not generate answers from low-quality evidence. If the top reranker score (where a reranker is a cross-encoder model that rescores retrieved passages for relevance to the query, as introduced in Section 37.1) falls below a threshold, or if the retriever returns fewer than a minimum number of relevant passages, the system should respond with "I could not find sufficient evidence to answer this question" rather than attempting an answer from weak evidence.
def gated_generate(
question: str,
evidence: list[dict],
min_evidence: int = 2,
min_rerank_score: float = 0.5,
) -> str:
"""Generate an answer only if evidence quality meets thresholds."""
# Filter to high-quality evidence
strong_evidence = [
e for e in evidence
if e["rerank_score"] >= min_rerank_score
]
if len(strong_evidence) < min_evidence:
return (
f"Insufficient evidence to answer this question. "
f"Found {len(strong_evidence)} passages above the "
f"relevance threshold (minimum {min_evidence} required). "
f"Consider refining the question or expanding the corpus."
)
return generate_cited_answer(question, strong_evidence, client)
6.4 Structured Output with Source Mapping
Instead of generating free-text answers with inline citations, use structured output to enforce a strict mapping between claims and sources. Each claim is a separate object linked to specific source passages, making verification trivial and hallucination structurally harder.
from pydantic import BaseModel, Field
class CitedClaim(BaseModel):
"""A single factual claim with its supporting evidence."""
claim: str = Field(description="A single factual statement")
source_ids: list[str] = Field(
description="Source identifiers supporting this claim"
)
verbatim_support: str = Field(
description="Direct quote from the source that supports "
"this claim, copied verbatim"
)
confidence: float = Field(
ge=0.0, le=1.0,
description="Confidence that the source supports this claim"
)
class StructuredAnswer(BaseModel):
"""A research answer decomposed into individually cited claims."""
summary: str = Field(
description="One-sentence summary of the answer"
)
claims: list[CitedClaim] = Field(
description="Individual claims with source mappings"
)
limitations: list[str] = Field(
description="Aspects of the question not covered by "
"the available evidence"
)
def generate_structured_answer(
question: str,
evidence: list[dict],
) -> StructuredAnswer:
"""Generate a structured answer with per-claim source mapping."""
context = "\n\n".join(
f"[Source {i+1}]: {ev['passage']}"
for i, ev in enumerate(evidence)
)
response = client.messages.create(
model="claude-sonnet-4-20250514",
max_tokens=4096,
messages=[{
"role": "user",
"content": f"""Answer this research question using ONLY
the provided sources. Structure your response as a JSON object
matching this schema:
- summary: one-sentence answer
- claims: array of objects, each with:
- claim: a single factual statement
- source_ids: which sources support it (e.g., ["Source 1"])
- verbatim_support: exact quote from the source
- confidence: 0.0 to 1.0
- limitations: aspects not covered by the evidence
Question: {question}
Sources:
{context}""",
}],
)
return StructuredAnswer.model_validate_json(
response.content[0].text
)
CitedClaim schema forces the model to ground every statement in specific text, and the confidence field makes uncertainty visible to downstream consumers.6.5 Iterative Refinement with Self-Critique
The most effective (and most expensive) strategy: generate an answer, evaluate it for faithfulness, identify unsupported claims, and regenerate with explicit instructions to fix or remove them. This mirrors PaperQA2's agentic loop at the claim level.
def iterative_faithful_generation(
question: str,
evidence: list[dict],
max_iterations: int = 3,
target_faithfulness: float = 0.95,
) -> dict:
"""Generate and iteratively refine until faithfulness target is met."""
evidence_passages = [e["passage"] for e in evidence]
answer = generate_cited_answer(question, evidence, client)
for iteration in range(max_iterations):
eval_result = compute_faithfulness(answer, evidence_passages)
if eval_result["faithfulness_score"] >= target_faithfulness:
return {
"answer": answer,
"faithfulness": eval_result["faithfulness_score"],
"iterations": iteration + 1,
"details": eval_result["details"],
}
# Identify unsupported claims
unsupported = [
d for d in eval_result["details"]
if d["verdict"] != "supported"
]
unsupported_text = "\n".join(
f"- {d['claim']} (verdict: {d['verdict']})"
for d in unsupported
)
# Regenerate with feedback
response = client.messages.create(
model="claude-sonnet-4-20250514",
max_tokens=2048,
messages=[{
"role": "user",
"content": f"""Your previous answer to the question
"{question}" contained claims not supported by the evidence.
Unsupported claims:
{unsupported_text}
Please regenerate the answer, removing or correcting these claims.
Only include information that is directly stated in the sources.
If removing these claims leaves gaps in the answer, acknowledge
those gaps explicitly.
Sources:
{chr(10).join(f'[Source {i+1}]: {e["passage"]}' for i, e in enumerate(evidence))}""",
}],
)
answer = response.content[0].text
# Return best effort after max iterations
final_eval = compute_faithfulness(answer, evidence_passages)
return {
"answer": answer,
"faithfulness": final_eval["faithfulness_score"],
"iterations": max_iterations,
"details": final_eval["details"],
}
The strategies above add computational cost: citation verification requires one LLM call per citation, structured output increases prompt length, and iterative refinement doubles or triples the generation cost. For general-purpose chatbots, this overhead may not be justified. For scientific research copilots, it is a bargain. A single hallucinated drug interaction, fabricated experimental result, or misattributed finding can waste weeks of research effort. The cost of three LLM calls per answer is negligible compared to the cost of acting on false information. Build faithfulness checking into the pipeline from the start, not as an afterthought.
Research Frontier
The FACTS Grounding benchmark, released by Google DeepMind in late 2024, provides a standardized leaderboard for measuring how well language models ground their responses in supplied documents. Unlike earlier faithfulness benchmarks that relied on synthetic or small-scale evaluation sets, FACTS Grounding uses 1,719 examples spanning long documents (average ~32k tokens) with human-annotated eligibility and grounding labels. Models are scored on two axes: whether they correctly refuse unanswerable requests and whether answerable responses are fully grounded in the provided text. As of early 2025 (circa 2025), no model exceeded 90% on the combined metric, revealing that even frontier systems still hallucinate at meaningful rates when handling long, complex source documents. The benchmark is openly available and designed as a drop-in evaluation for any RAG pipeline, making it a practical tool for measuring progress beyond what the RAGAS metrics and FActScore capture.
Try It: Build a Faithfulness Scorer from Scratch
Build a minimal faithfulness evaluation pipeline using only Python, an LLM API, and a few paragraphs of text. This project takes about 30 minutes and requires no special libraries beyond an API client.
Step 1. Choose a Wikipedia article on a scientific topic (for example, CRISPR or mRNA vaccines). Copy three paragraphs as your "evidence corpus" and save them in a Python list of strings.
Step 2. Write a short (3-4 sentence) answer to a question about that topic. Intentionally include one claim that is supported by your evidence, one claim that is true but not in the evidence, and one claim that contradicts the evidence.
Step 3. Implement the extract_atomic_claims function from Listing 37.9, using your preferred LLM API. Run it on your answer and inspect the decomposed claims. Verify that each claim is truly atomic (a single checkable fact).
Step 4. Implement the verify_claim function from the same listing. Run it on each atomic claim against your evidence paragraphs. Confirm that it correctly identifies the supported, unsupported, and contradicted claims you planted.
Step 5. Compute the faithfulness score (supported / total) and print a per-claim report. Experiment with modifying your answer to raise the score to 1.0 by removing unsupported claims, then observe how the score drops when you add plausible but unsourced statistics.
Exercise 37.2.1
A RAG system generates the following answer about aspirin: "Aspirin inhibits cyclooxygenase enzymes [Source 1]. It reduces the risk of heart attack by 44% [Source 2]. Aspirin was first synthesized by Felix Hoffmann at Bayer in 1897 [Source 1]." The retrieved evidence contains two passages. Source 1: "Aspirin (acetylsalicylic acid) irreversibly inhibits cyclooxygenase-1 and cyclooxygenase-2 enzymes, reducing prostaglandin synthesis." Source 2: "A meta-analysis of six trials found that daily low-dose aspirin reduced the risk of myocardial infarction by 32% (95% CI: 21-41%)." Classify each of the three claims by hallucination type (Type 1 through 4, as defined in this section) and compute the faithfulness score.
Hint
Check each claim against its cited source. The first claim is a match. The second claim cites Source 2 but changes the reported statistic. The third claim cites Source 1, which says nothing about Hoffmann or 1897. Consider whether a distorted number and a fully unsupported historical fact fall into different categories.Step-Through: Atomic Claim Decomposition and Faithfulness Scoring
Trace through the faithfulness pipeline with a concrete example. Start with a two-sentence answer: "Metformin reduces HbA1c by 1.0-1.5% in type 2 diabetes patients. It works by inhibiting hepatic glucose production and improving insulin sensitivity."
Step 1: Decompose into atomic claims. The pipeline extracts three claims: (C1) "Metformin reduces HbA1c by 1.0-1.5% in type 2 diabetes patients." (C2) "Metformin inhibits hepatic glucose production." (C3) "Metformin improves insulin sensitivity."
Step 2: Verify each claim against evidence. Suppose the evidence contains one passage: "Metformin lowers HbA1c by approximately 1.0-2.0% and acts primarily by suppressing hepatic glucose output." C1: the passage says 1.0-2.0%, but the answer says 1.0-1.5%. The range overlaps but is narrower than stated, so the verdict depends on strictness; a strict evaluator marks this "not_supported." C2: "suppressing hepatic glucose output" entails "inhibits hepatic glucose production," so verdict = "supported," supporting_passage = 1. C3: nothing in the passage mentions insulin sensitivity, so verdict = "not_supported," supporting_passage = null.
Step 3: Compute the score. Supported claims: 1 out of 3. Faithfulness = 1/3 = 0.33. The system flags C1 and C3 for removal or revision before returning the answer to the user.
Real-World Application: Elicit (Ought)
Elicit, the research assistant built by Ought (now operating independently), applies faithfulness verification to every claim it extracts from scientific papers. When a user asks Elicit to summarize findings across multiple studies, each extracted data point (effect size, sample size, population) is linked to a specific passage in the source PDF, and claims that cannot be traced to verbatim text are flagged with reduced confidence or omitted from the summary table. This design directly implements the atomic claim decomposition and verification pattern described in this section, at a scale of millions of papers.
The Confident Confabulist
In a 2023 evaluation, researchers asked GPT-4 to answer biomedical questions with citations. The model produced references so convincingly formatted that reviewers initially assumed they were real: correct journal abbreviations, plausible author names, reasonable publication years, and coherent DOIs. When checked, the researchers reported that roughly 30% of the cited papers did not exist at all. The most striking detail was that the fabricated papers often had higher apparent relevance to the question than real papers would have, because the model was optimizing for a citation that perfectly supported its claim rather than retrieving one that actually existed. The model was, in effect, writing the ideal evidence for its own argument.
Lab: Faithfulness Fragility Under Retrieval Noise
Goal: Measure how faithfulness degrades as irrelevant passages are injected into the retrieved context alongside relevant ones.
Tools needed: Python, the anthropic SDK (or any LLM API client), and three Wikipedia paragraphs on a scientific topic of your choice.
Setup (5 min): Select one paragraph as your "gold" evidence and prepare a factual question it can answer. Write two distractor paragraphs from unrelated topics (e.g., one about Renaissance painting, one about tectonic plates).
Experiment (15 min): Implement the compute_faithfulness function from Listing 37.9. Run it in four conditions: (1) gold passage only, (2) gold + 1 distractor, (3) gold + 2 distractors, (4) 2 distractors only (no gold passage). For each condition, generate an answer with a standard RAG prompt ("Answer using only the provided sources"), then compute the faithfulness score.
What to vary: The ratio of distractors to gold passages, the topical similarity between distractors and the question, and the prompt strictness (add or remove the instruction "If the sources do not contain the answer, say so").
What to observe: Track the faithfulness score, the number of unsupported claims, and whether the model invents connections between the distractors and the question. You should see faithfulness drop as distractors increase, with a sharp decline when the gold passage is absent entirely. Note whether the strict prompt reduces hallucination in the distractor-only condition.
Section Summary
RAG reduces hallucination but does not eliminate it. Scientific RAG systems face four hallucination types (fabricated citations, unsupported claims, distorted evidence, incomplete attribution), each requiring targeted mitigation. Faithfulness, measured through atomic claim decomposition and verification (as shown in Figure 37.3), is the central metric for scientific RAG quality. PaperQA2 demonstrates that agentic architectures with iterative search and citation verification can, according to published benchmarks, match or exceed expert-level performance on scientific QA. Five practical strategies (constrained prompting, post-generation verification, retrieval quality gates, structured output, and iterative refinement) reduce hallucination rates progressively, with diminishing returns above 95% faithfulness. The RAGAS framework provides a standard evaluation toolkit covering faithfulness, answer relevancy, context precision, and context recall. The next section assembles all of these components into a complete research copilot.