28.3 Scientific Document AI

1. The Document Understanding Pipeline

Somewhere in a stack of 500 clinical trial PDFs sits a dose-response curve on page 14 that contradicts the headline claim of a competing study, but no text-only search will ever surface it. Extracting that kind of evidence requires a pipeline with several stages: layout analysis (identifying text blocks, figures, tables, equations), text extraction (optical character recognition (OCR) for scanned documents, direct extraction for digital PDFs), figure captioning (understanding what a chart or microscopy image shows), table parsing (converting visual tables to structured data), and equation recognition (converting rendered math to LaTeX or MathML).

A document understanding pipeline chains machine learning and rule-based components to transform a raw PDF (a stream of characters, vector graphics, and embedded images with no semantic markup) into structured, queryable content. Every text paragraph, figure, table, and equation becomes individually typed, located, and ready for downstream embedding or generation. Scientific PDFs frequently encode a substantial portion of their key findings in non-text elements (figures, tables, equations); without explicit extraction, those findings stay invisible to language models. A layout detector (typically a vision transformer trained on document page images) classifies rectangular regions by type. Modality-specific extractors (OCR for text, image cropping for figures, cell parsers for tables) then convert each region into a machine-readable representation. Figure 28.3 illustrates this flow from raw PDF through layout detection, modality-specific extraction, embedding, and unified retrieval. Use a full document understanding pipeline when your corpus contains scanned or figure-heavy documents; for born-digital, text-only papers, simpler PDF text extraction (such as PyMuPDF or pdfplumber) is sufficient and much faster.

Modern document AI models handle many of these stages jointly. LayoutLMv3 (Huang et al., 2022) is a multimodal transformer that jointly processes text tokens and image patches from document pages, learning to classify regions as text, table, figure, or equation. Nougat (Blecher et al., 2023) takes a different approach: it treats the entire PDF page as an image and uses a vision encoder-decoder to output Markdown text, preserving equations and structure without explicit layout analysis. As of 2025, Marker v2 and Docling provide faster, more accurate PDF-to-structured-text conversion and have largely replaced Nougat in production pipelines. In short: every figure, table, and equation in a PDF is invisible to a language model until the document understanding pipeline converts it to searchable text.

import fitz  # PyMuPDF
from dataclasses import dataclass, field
from pathlib import Path


@dataclass
class DocumentChunk:
    """A chunk of content extracted from a scientific document.

    Chunks can be text paragraphs, figure descriptions,
    table contents, or equations.
    """
    content: str
    chunk_type: str       # "text", "figure", "table", "equation"
    page_number: int
    source_file: str
    section_title: str = ""
    caption: str = ""     # for figures and tables
    metadata: dict = field(default_factory=dict)


def parse_scientific_pdf(pdf_path: str,
                         chunk_size: int = 500) -> list[DocumentChunk]:
    """Extract structured chunks from a scientific PDF.

    Uses PyMuPDF for text extraction with page-level
    layout awareness. Figures are identified by image blocks.

    Parameters
    ----------
    pdf_path : str
        Path to the PDF file.
    chunk_size : int
        Target number of characters per text chunk.
    """
    doc = fitz.open(pdf_path)
    chunks = []

    for page_num, page in enumerate(doc):
        # Extract text blocks with position info
        blocks = page.get_text("dict")["blocks"]

        current_text = ""
        for block in blocks:
            if block["type"] == 0:  # text block
                for line in block["lines"]:
                    line_text = " ".join(
                        span["text"] for span in line["spans"]
                    )
                    current_text += line_text + " "

                    # Chunk when we reach target size
                    if len(current_text) >= chunk_size:
                        chunks.append(DocumentChunk(
                            content=current_text.strip(),
                            chunk_type="text",
                            page_number=page_num + 1,
                            source_file=str(pdf_path),
                        ))
                        current_text = ""

            elif block["type"] == 1:  # image block
                chunks.append(DocumentChunk(
                    content=f"[Figure on page {page_num + 1}]",
                    chunk_type="figure",
                    page_number=page_num + 1,
                    source_file=str(pdf_path),
                    metadata={
                        "width": block.get("width", 0),
                        "height": block.get("height", 0),
                    },
                ))

        # Flush remaining text
        if current_text.strip():
            chunks.append(DocumentChunk(
                content=current_text.strip(),
                chunk_type="text",
                page_number=page_num + 1,
                source_file=str(pdf_path),
            ))

    doc.close()
    return chunks

2. Vision-Language Models for Scientific Figures

Scientific figures encode critical information that text alone cannot capture: dose-response curves, phylogenetic trees, crystal structure visualizations, and experimental setup photographs. Understanding these figures requires vision-language models (VLMs) trained on scientific imagery.

General-purpose VLMs like GPT-4V and Claude can interpret many scientific figure types, but specialized models typically perform better on domain-specific tasks. MatSci-NLP (a language model fine-tuned on materials science literature) processes materials science charts. ChartQA answers questions about data visualizations. The practical strategy uses a VLM to generate text descriptions of figures, which are then indexed alongside the paper's text for retrieval.

import torch
from transformers import (
    AutoProcessor, AutoModelForVision2Seq,
)
from PIL import Image


class ScientificFigureCaptioner:
    """Generate text descriptions of scientific figures.

    Uses a vision-language model to produce detailed captions
    that describe the figure's content, axes, trends, and
    key findings for indexing and retrieval.
    """

    def __init__(self, model_name: str = "Salesforce/blip2-opt-2.7b"):
        self.processor = AutoProcessor.from_pretrained(model_name)
        self.model = AutoModelForVision2Seq.from_pretrained(
            model_name, torch_dtype=torch.float16,
        )
        self.model.eval()

    def caption_figure(self, image_path: str,
                       context: str = "") -> str:
        """Generate a detailed caption for a scientific figure.

        Parameters
        ----------
        image_path : str
            Path to the figure image file.
        context : str
            Surrounding text from the paper (section title,
            nearby paragraphs) to guide the captioning.
        """
        image = Image.open(image_path).convert("RGB")

        prompt = (
            "This is a figure from a scientific paper. "
            "Describe what the figure shows, including axes, "
            "data trends, and key findings."
        )
        if context:
            prompt += f" Context from the paper: {context}"

        inputs = self.processor(
            images=image, text=prompt, return_tensors="pt"
        ).to(self.model.device)

        with torch.no_grad():
            output_ids = self.model.generate(
                **inputs, max_new_tokens=256,
            )

        caption = self.processor.decode(
            output_ids[0], skip_special_tokens=True
        )
        return caption


def enrich_chunks_with_figure_captions(
    chunks: list[DocumentChunk],
    captioner: ScientificFigureCaptioner,
    figure_dir: str,
) -> list[DocumentChunk]:
    """Add AI-generated captions to figure chunks.

    Replaces placeholder figure content with detailed
    VLM-generated descriptions for better retrieval.
    """
    enriched = []
    for chunk in chunks:
        if chunk.chunk_type == "figure":
            # Find the corresponding image file
            img_path = f"{figure_dir}/page{chunk.page_number}_fig.jpg"
            try:
                caption = captioner.caption_figure(img_path)
                chunk.content = caption
                chunk.caption = caption
            except FileNotFoundError:
                pass  # keep placeholder
        enriched.append(chunk)
    return enriched

Once figures have been converted to searchable text descriptions, the next challenge is orchestrating retrieval and reasoning across an entire corpus of papers, not just individual figures or chunks.

3. PaperQA2: Agentic Question Answering Over Literature

When a researcher asks "What is the recommended dose adjustment for patients with renal impairment?" and the answer lives in a table on page 47 of one paper, contradicted by a figure in another, no single retrieval pass will reconcile the evidence; the system must search, read, re-search, and reason across sources. PaperQA2 (Skarlinski et al., 2024) is an agentic RAG system (one that autonomously decides which tools to invoke in a loop, rather than making a single retrieval pass) designed specifically for scientific literature. Unlike single-pass RAG pipelines, PaperQA2 operates as a multi-step agent that iteratively gathers evidence. It searches for papers, reads relevant sections, gathers supporting passages, scores their relevance, and synthesizes a cited answer. On the LitQA2 benchmark of biomedical questions, it produced answers approaching the quality of human subject-matter experts.

The PaperQA2 architecture consists of several agent actions. Figure 28.3.1 illustrates PaperQA2 agentic document QA pipeline.

• paper_search: Query a corpus (local PDFs, Semantic Scholar, PubMed) to find relevant papers.
• gather_evidence: Read retrieved papers and extract passages relevant to the question.
• gen_answer: Synthesize an answer from gathered evidence with inline citations.

# PaperQA2 usage via the paper-qa package
# pip install paper-qa

from paperqa import Settings, ask
from paperqa.settings import AgentSettings
from pathlib import Path


async def answer_research_question(
    question: str,
    paper_directory: str,
    max_sources: int = 10,
) -> dict:
    """Answer a scientific question using PaperQA2.

    Searches a local directory of PDFs, extracts evidence,
    and generates a cited answer.

    Parameters
    ----------
    question : str
        The scientific question to answer.
    paper_directory : str
        Path to directory containing PDF papers.
    max_sources : int
        Maximum number of papers to consult.
    """
    settings = Settings(
        agent=AgentSettings(
            agent_type="fake",  # use "OpenAIFunctionsAgent" in prod
            max_search_queries=3,
        ),
        answer=Settings.AnswerSettings(
            evidence_k=max_sources,
            answer_max_sources=5,
        ),
        paper_directory=paper_directory,
    )

    result = await ask(
        question,
        settings=settings,
    )

    return {
        "answer": result.answer,
        "references": [
            {
                "title": ref.title,
                "authors": ref.authors,
                "year": ref.year,
                "key_passage": ref.text,
                "relevance_score": ref.score,
            }
            for ref in result.references
        ],
        "confidence": result.confidence,
    }


# Usage example (requires asyncio event loop)
# import asyncio
# result = asyncio.run(answer_research_question(
#     "What is the binding affinity of nirmatrelvir to SARS-CoV-2 Mpro?",
#     paper_directory="./papers/covid_antivirals/",
# ))
# print(result["answer"])
# for ref in result["references"]:
#     print(f"  [{ref['year']}] {ref['title']}: {ref['key_passage'][:100]}...")

4. Dense Retrieval for Heterogeneous Scientific Content

Scientific documents mix text, figures, tables, and equations, so a retrieval system must handle all four modalities. The strategy embeds every chunk into a single dense vector space (a high-dimensional numeric representation where semantically similar content maps to nearby points) for nearest-neighbor search: text chunks go directly to the encoder, figure chunks use VLM-generated captions as a proxy, table chunks are serialized to Markdown or CSV, and equation chunks use their LaTeX source.

import numpy as np
from sentence_transformers import SentenceTransformer
from typing import Optional


class ScientificRetriever:
    """Dense retrieval over heterogeneous scientific content.

    Embeds text, figures (via captions), tables (via serialization),
    and equations (via LaTeX) into a single vector space for
    unified cross-modal search.
    """

    def __init__(self, model_name: str = "BAAI/bge-base-en-v1.5"):
        self.encoder = SentenceTransformer(model_name)
        self.chunks: list[DocumentChunk] = []
        self.embeddings: Optional[np.ndarray] = None

    def index_documents(self, chunks: list[DocumentChunk]) -> None:
        """Build a dense index from document chunks.

        Each chunk (text, figure, table, equation) is embedded
        as text for unified retrieval.
        """
        self.chunks = chunks

        # Convert all chunk types to text for embedding
        texts = []
        for chunk in chunks:
            if chunk.chunk_type == "text":
                texts.append(chunk.content)
            elif chunk.chunk_type == "figure":
                # Use VLM caption or original caption
                texts.append(
                    f"Figure: {chunk.caption or chunk.content}"
                )
            elif chunk.chunk_type == "table":
                texts.append(f"Table: {chunk.content}")
            elif chunk.chunk_type == "equation":
                texts.append(f"Equation: {chunk.content}")
            else:
                texts.append(chunk.content)

        self.embeddings = self.encoder.encode(
            texts, show_progress_bar=True,
            normalize_embeddings=True,
        )

    def search(self, query: str, top_k: int = 10,
               chunk_type: Optional[str] = None) -> list[dict]:
        """Retrieve chunks most relevant to a query.

        Parameters
        ----------
        query : str
            Natural language query.
        top_k : int
            Number of results to return.
        chunk_type : str, optional
            Filter results by type ("text", "figure", "table").
        """
        q_emb = self.encoder.encode(
            [query], normalize_embeddings=True
        )
        scores = (q_emb @ self.embeddings.T).squeeze()

        # Apply type filter if specified
        if chunk_type:
            mask = np.array([
                c.chunk_type == chunk_type for c in self.chunks
            ])
            scores = np.where(mask, scores, -np.inf)

        top_indices = np.argsort(-scores)[:top_k]

        return [
            {
                "rank": rank + 1,
                "score": float(scores[idx]),
                "content": self.chunks[idx].content[:300],
                "type": self.chunks[idx].chunk_type,
                "page": self.chunks[idx].page_number,
                "source": self.chunks[idx].source_file,
            }
            for rank, idx in enumerate(top_indices)
        ]

Retrieving the right chunks is only half the problem; the system must also ensure that the answers it generates from those chunks are faithful to the sources rather than fabricated.

5. Hallucination Detection and Source Attribution

Scientific question answering has a unique challenge: hallucinated answers can propagate false claims into the research record. A model that confidently states "compound X has IC50 (half-maximal inhibitory concentration, the drug concentration needed to inhibit a biological target by 50%) = 3.2 nM against target Y" when no source supports this claim is dangerous. PaperQA2 addresses this with source attribution: every statement in the generated answer must be traceable to a specific passage in a specific paper, and the system assigns a confidence score reflecting how well the evidence supports the claim.

Common Misconception

A frequent misunderstanding is that retrieval-augmented generation cannot hallucinate because it grounds answers in real documents. In practice, RAG systems hallucinate in at least three ways: the model may fabricate details not present in any retrieved passage, it may incorrectly combine facts from different passages (conflation), or it may cite a real passage that does not actually support the stated claim (misattribution). Retrieval reduces hallucination relative to pure generation, but it does not eliminate it; explicit claim verification (as shown below) remains necessary.

from dataclasses import dataclass
import re


@dataclass
class CitedClaim:
    """A claim extracted from a generated answer with its source."""
    claim_text: str
    source_passage: str
    source_paper: str
    source_page: int
    support_score: float  # 0-1, how well the passage supports the claim


def extract_and_verify_claims(
    answer: str,
    evidence_passages: list[dict],
    verifier_model,
) -> list[CitedClaim]:
    """Extract claims from an answer and verify source support.

    Uses a natural language inference (NLI) model, where NLI is
    the task of determining whether a premise sentence entails
    (logically implies), contradicts, or is neutral toward a
    hypothesis sentence.

    Parameters
    ----------
    answer : str
        Generated answer with inline citations like [1], [2].
    evidence_passages : list[dict]
        Retrieved passages with paper metadata.
    verifier_model : callable
        NLI model that scores (premise, hypothesis) pairs.
        Returns probability that premise entails hypothesis.
    """
    # Split answer into individual claims (sentences)
    sentences = re.split(r'(?<=[.!?])\s+', answer)

    verified_claims = []
    for sentence in sentences:
        # Extract citation markers [N]
        citations = re.findall(r'\[(\d+)\]', sentence)
        clean_claim = re.sub(r'\[\d+\]', '', sentence).strip()

        if not clean_claim or not citations:
            continue

        # Verify each cited source supports the claim
        best_score = 0.0
        best_source = None
        for cite_num in citations:
            idx = int(cite_num) - 1
            if idx < len(evidence_passages):
                passage = evidence_passages[idx]
                # NLI: does the passage entail the claim?
                score = verifier_model(
                    premise=passage["text"],
                    hypothesis=clean_claim,
                )
                if score > best_score:
                    best_score = score
                    best_source = passage

        if best_source:
            verified_claims.append(CitedClaim(
                claim_text=clean_claim,
                source_passage=best_source["text"][:200],
                source_paper=best_source.get("title", "Unknown"),
                source_page=best_source.get("page", 0),
                support_score=best_score,
            ))

    return verified_claims


def flag_unsupported_claims(
    claims: list[CitedClaim],
    threshold: float = 0.5,
) -> list[CitedClaim]:
    """Flag claims with insufficient source support.

    Returns claims where the best supporting evidence
    falls below the entailment threshold, where entailment
    means the evidence logically implies the claim.
    """
    return [c for c in claims if c.support_score < threshold]

6. Building a Document Retrieval Pipeline

Combining the components above, we can build a complete pipeline that ingests scientific PDFs, extracts and indexes all modalities, and answers questions with cited sources. This pipeline forms the document understanding layer of the multimodal research assistant built in Section 28.4.

from pathlib import Path
import json


class ScientificDocumentPipeline:
    """End-to-end pipeline for scientific document understanding.

    Combines PDF parsing, figure captioning, dense retrieval,
    and answer generation with source attribution.
    """

    def __init__(self, paper_dir: str,
                 embedding_model: str = "BAAI/bge-base-en-v1.5"):
        self.paper_dir = Path(paper_dir)
        self.retriever = ScientificRetriever(embedding_model)
        self.all_chunks: list[DocumentChunk] = []
        self._indexed = False

    def ingest_papers(self, chunk_size: int = 500) -> int:
        """Parse all PDFs in the paper directory.

        Returns the total number of chunks extracted.
        """
        pdf_files = list(self.paper_dir.glob("*.pdf"))
        print(f"Found {len(pdf_files)} PDF files")

        for pdf_path in pdf_files:
            chunks = parse_scientific_pdf(
                str(pdf_path), chunk_size=chunk_size
            )
            self.all_chunks.extend(chunks)

        print(f"Extracted {len(self.all_chunks)} chunks "
              f"({sum(1 for c in self.all_chunks if c.chunk_type == 'text')} text, "
              f"{sum(1 for c in self.all_chunks if c.chunk_type == 'figure')} figures)")
        return len(self.all_chunks)

    def build_index(self) -> None:
        """Build the dense retrieval index over all chunks."""
        self.retriever.index_documents(self.all_chunks)
        self._indexed = True
        print(f"Indexed {len(self.all_chunks)} chunks")

    def query(self, question: str, top_k: int = 10,
              chunk_type: str = None) -> list[dict]:
        """Search the indexed documents.

        Parameters
        ----------
        question : str
            Natural language query.
        top_k : int
            Number of results.
        chunk_type : str, optional
            Filter by "text", "figure", or "table".
        """
        if not self._indexed:
            raise RuntimeError("Call build_index() before querying")

        return self.retriever.search(question, top_k, chunk_type)

    def answer(self, question: str,
               max_sources: int = 5) -> dict:
        """Generate a cited answer from the document corpus.

        Retrieves evidence, synthesizes answer, and provides
        source attribution for each claim.
        """
        evidence = self.query(question, top_k=max_sources * 2)

        # Format evidence for answer generation
        context = "\n\n".join(
            f"[{i+1}] (p.{e['page']}, {Path(e['source']).stem}): "
            f"{e['content']}"
            for i, e in enumerate(evidence[:max_sources])
        )

        return {
            "question": question,
            "evidence": evidence[:max_sources],
            "context_for_llm": context,
            "num_sources": len(evidence[:max_sources]),
        }


# Usage
# pipeline = ScientificDocumentPipeline("./papers/kinase_inhibitors/")
# pipeline.ingest_papers()
# pipeline.build_index()
#
# results = pipeline.query(
#     "What is the selectivity profile of lapatinib?",
#     top_k=5,
# )
# for r in results:
#     print(f"  [{r['type']}] p.{r['page']} score={r['score']:.3f}")
#     print(f"    {r['content'][:100]}...")

# Full PaperQA2 pipeline (replaces ~250 lines with ~10)
from paperqa import Settings, ask

result = await ask(
    "What is the binding affinity of erlotinib to EGFR?",
    settings=Settings(paper_directory="./papers/"),
)

print(result.answer)          # cited answer
print(result.confidence)      # 0-1 confidence score
for ref in result.references:
    print(f"  {ref.title}: {ref.text[:100]}...")

7. Evaluation: Answer Quality and Attribution Fidelity

Evaluating a scientific document QA system requires two orthogonal measurements: answer correctness (does the answer match the ground truth?) and attribution fidelity (do the cited sources actually support the claims?). A system can produce a correct answer with wrong citations (lucky guess), or a well-cited answer that is factually wrong (the source was misinterpreted).

from dataclasses import dataclass


@dataclass
class DocumentQAMetrics:
    """Evaluation metrics for scientific document QA systems.

    Measures both answer quality and source attribution.
    """

    @staticmethod
    def answer_accuracy(predictions: list[str],
                        ground_truths: list[str],
                        exact_match: bool = False) -> float:
        """Fraction of answers that match the ground truth.

        Uses exact string match or fuzzy substring match.
        """
        correct = 0
        for pred, truth in zip(predictions, ground_truths):
            if exact_match:
                if pred.strip().lower() == truth.strip().lower():
                    correct += 1
            else:
                # Fuzzy: truth appears as substring in prediction
                if truth.strip().lower() in pred.strip().lower():
                    correct += 1
        return correct / len(predictions) if predictions else 0.0

    @staticmethod
    def citation_precision(cited_passages: list[list[str]],
                           relevant_passages: list[list[str]]) -> float:
        """Fraction of cited passages that are actually relevant.

        High precision means the system does not cite irrelevant sources.
        """
        total_cited = 0
        relevant_cited = 0
        for cited, relevant in zip(cited_passages, relevant_passages):
            for c in cited:
                total_cited += 1
                if any(r in c or c in r for r in relevant):
                    relevant_cited += 1
        return relevant_cited / total_cited if total_cited else 0.0

    @staticmethod
    def citation_recall(cited_passages: list[list[str]],
                        relevant_passages: list[list[str]]) -> float:
        """Fraction of relevant passages that are actually cited.

        High recall means the system does not miss important sources.
        """
        total_relevant = 0
        recalled = 0
        for cited, relevant in zip(cited_passages, relevant_passages):
            for r in relevant:
                total_relevant += 1
                if any(r in c or c in r for c in cited):
                    recalled += 1
        return recalled / total_relevant if total_relevant else 0.0