Chapter 36: Literature Mining | Building Discovery AI

Overview

Science advances through the accumulation, synthesis, and recombination of published knowledge. But the scale of modern scientific literature has outgrown any individual's capacity to survey it. PubMed alone indexes over 37 million biomedical articles. The arXiv preprint server receives more than 20,000 new papers per month. No researcher, regardless of dedication, can read even a fraction of what is relevant to their field.

Literature mining transforms this problem from a human bottleneck into a computational opportunity. By treating the scientific literature as a structured dataset (complete with citation networks, metadata graphs, and free-text knowledge), we can build systems that map entire research landscapes, identify emerging trends, extract structured claims from unstructured prose, and reveal gaps that no single researcher could spot. This chapter builds those systems from the ground up.

We begin with citation analysis and bibliometrics (Section 36.1): the mathematics of citation networks, influence metrics like PageRank and h-index, and community detection algorithms that reveal the topical structure of a research field. We then move to knowledge extraction from papers (Section 36.2): parsing PDFs, extracting entities and relations, identifying claim-evidence structures, and building semantic search over paper collections. Finally, we assemble a complete literature mining pipeline (Section 36.3) that ingests 500+ papers from OpenAlex, clusters them by topic, maps the citation landscape, and identifies research gaps through structural holes in the knowledge network.

The tools and techniques here feed directly into the retrieval-augmented discovery systems of Chapter 37, the knowledge graphs of Chapter 38, and the hypothesis generation systems of Chapter 39. Literature mining also connects back to the knowledge representation foundations of Chapter 3 and provides the raw material for the claim validation pipelines of Chapter 41.

Prerequisites

Readers should be comfortable with Chapter 3: Knowledge Representation for graph and ontology concepts, and Chapter 1: Discovery as Search for search algorithms over structured spaces. Familiarity with basic graph theory (nodes, edges, connected components) from Appendix A is assumed. Python proficiency with pandas and basic NLP concepts (tokenization, embeddings) is helpful. No prior experience with bibliometric APIs or PDF parsing is required.

Learning Outcomes

Query OpenAlex, Semantic Scholar, and Crossref APIs to harvest metadata, abstracts, and citation links for large paper collections.
Compute bibliometric indicators (h-index, impact factor, citation half-life) and apply PageRank to identify influential papers in a citation network.
Apply Louvain community detection to partition a citation graph into topical clusters and interpret the resulting structure.
Extract entities, relations, and claim-evidence pairs from scientific PDFs using PyMuPDF, Docling, and transformer-based NLP models.
Build semantic search over a paper corpus using sentence embeddings and approximate nearest neighbors.
Construct a complete literature mining pipeline that ingests 500+ papers, clusters them by topic, and identifies research gaps through citation network analysis.

Sections

36.1 Citation Analysis and Bibliometrics

Citation networks as directed graphs. Bibliometric indicators: h-index, impact factor, citation half-life. PageRank for paper influence. Louvain community detection for topic clustering. Querying OpenAlex, Semantic Scholar, and Crossref at scale.

36.2 Knowledge Extraction from Papers

PDF parsing with PyMuPDF and Docling. Named entity recognition for scientific text. Relation extraction and claim-evidence linking. Semantic search with sentence embeddings. Building structured knowledge from unstructured papers.

36.3 Building a Literature Miner

Recipe: a 500+ paper domain miner with automated harvesting, topic clustering via citation community detection, gap analysis through structural holes, and trend visualization. Integration with the Discovery Workbench.

What's Next

Literature mining extracts knowledge from papers, but using that knowledge at query time requires a different architecture. Chapter 37: Retrieval-Augmented Discovery Systems builds on the extraction and embedding techniques from this chapter to create systems that can answer complex scientific questions by retrieving and synthesizing evidence from large paper collections. Where this chapter mines the literature into structured form, the next chapter teaches you to query it conversationally.

Bibliography

Foundational Papers

Garfield, E. (1972). Citation analysis as a tool in journal evaluation. Science, 178(4060), 471-479.

The paper that established citation analysis as a quantitative discipline, introducing the impact factor and laying the groundwork for modern bibliometrics.

Newman, M. E. J. (2004). Fast algorithm for detecting community structure in networks. Physical Review E, 69(6), 066133.

Introduced modularity-based community detection, the foundation for the Louvain algorithm used throughout this chapter for topic clustering in citation networks.

Blondel, V. D., Guillaume, J.-L., Lambiotte, R., & Lefebvre, E. (2008). Fast unfolding of communities in large networks. J. Stat. Mech., P10008.

The Louvain algorithm for community detection, which scales to millions of nodes and is the workhorse method for partitioning citation networks in this chapter.

Hirsch, J. E. (2005). An index to quantify an individual's scientific research output. PNAS, 102(46), 16569-16572.

Introduced the h-index, the most widely used single-number measure of a researcher's publication impact, analyzed in detail in Section 36.1.

Tools & Platforms

Priem, J., Piwowar, H., & Orr, R. (2022). OpenAlex: A fully-open index of scholarly works, authors, venues, institutions, and concepts. arXiv:2205.01833.

OpenAlex provides free, comprehensive scholarly metadata for over 250 million works. The primary data source for the literature mining pipeline in Section 36.3.

Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.

Semantic Scholar's API provides citation contexts, paper embeddings (SPECTER2), and AI-generated TLDRs. Used for semantic search and embedding-based retrieval in Section 36.2.

Crossref. (2024). Crossref REST API documentation.

Crossref provides DOI resolution and bibliographic metadata for over 150 million records. Used for metadata enrichment and reference linking.

Artifex Software. (2024). PyMuPDF documentation.

PyMuPDF (fitz) provides high-performance PDF text extraction, layout analysis, and image extraction. The primary PDF parsing tool in Section 36.2.

Auer, C., et al. (2024). Docling: An easy, self-contained and permissive document parser. arXiv:2408.09869.

Docling provides AI-powered document understanding with table extraction, figure detection, and structured output. Used for deep PDF parsing in Section 36.2.

Libraries

Hagberg, A. A., Schult, D. A., & Swart, P. J. (2008). Exploring network structure, dynamics, and function using NetworkX. Proc. SciPy, 11-15.

NetworkX provides graph algorithms (PageRank, Louvain, betweenness centrality) used throughout this chapter for citation network analysis.

Reimers, N. & Gurevych, I. (2019). Sentence-BERT: Sentence embeddings using Siamese BERT-networks. Proc. EMNLP.

Sentence Transformers provides the embedding models used for semantic search over paper abstracts in Section 36.2.

Pedregosa, F., et al. (2011). Scikit-learn: Machine learning in Python. JMLR, 12, 2825-2830.

Scikit-learn provides TF-IDF vectorization, clustering algorithms, and evaluation metrics used for text-based topic modeling in Section 36.3.

Surveys

Waltman, L. (2016). A review of the literature on citation impact indicators. Journal of Informetrics, 10(2), 365-391.

Comprehensive review of citation-based indicators, covering their mathematical properties, strengths, limitations, and appropriate use cases.

Kononova, O., et al. (2021). Opportunities and challenges of text mining in materials science. iScience, 24(3), 102155.

Survey of NLP methods for extracting structured knowledge from scientific papers, covering entity recognition, relation extraction, and knowledge base construction.