Chapter 27: Scientific Foundation Models

"I was pretrained on three billion proteins. I still cannot fold my own laundry."
A Foundation Model Pretending to Be a Scientist

Overview

Foundation models have reshaped natural language processing and computer vision. The same idea, pretraining a massive model on unlabeled data and then adapting it to downstream tasks, is now transforming the natural sciences. Protein language models predict structure and function from amino acid sequences. Molecular transformers learn chemical intuition from billions of SMILES strings. Graph neural networks pretrained on millions of crystal structures accelerate materials discovery by orders of magnitude. Genomic transformers capture regulatory grammar from raw nucleotide sequences.

This chapter surveys the landscape of scientific foundation models, explains why they work, and teaches you to adapt them to your own research problems. We begin with the principles that separate a scientific foundation model from a generic deep network (Section 27.1). We then tour the major model families for proteins and molecules (Section 27.2) and for materials and genomes (Section 27.3). Section 27.4 introduces Low-Rank Adaptation (LoRA), the parameter-efficient technique that makes fine-tuning these billion-parameter models feasible on a single GPU. Finally, Section 27.5 walks through a complete recipe: fine-tuning ESM-2 with LoRA to predict protein thermostability.

Prerequisites

This chapter assumes you have read Chapter 26 (Representation Learning), where we covered embeddings, self-supervised learning, and the transformer architecture. Familiarity with PyTorch and Hugging Face Transformers is helpful; Chapter 5 provides the necessary context on data-driven versus theory-driven modeling. You should also be comfortable with the attention mechanism and token embeddings from Appendix A.

What You Will Learn

Define what makes a model a "scientific foundation model" and distinguish it from task-specific deep learning.
Navigate the zoo of protein, molecular, materials, and genomic foundation models: ESM-2, ESM-3, ProGen2, Uni-Mol2, MolFormer, GNoME, MACE-MP-0, CHGNet, Nucleotide Transformer, and DNABERT-2.
Explain scaling laws for scientific models and the conditions under which more data and parameters improve downstream performance.
Derive and implement Low-Rank Adaptation (LoRA): $W = W_0 + BA$ where $B \in \mathbb{R}^{d \times r}$, $A \in \mathbb{R}^{r \times d}$, and $r \ll d$.
Fine-tune ESM-2 with LoRA on a protein thermostability dataset using Hugging Face PEFT.
Integrate scientific foundation models into the Discovery Workbench as reusable embedding and prediction modules.

Sections

27.1 What Makes a Scientific Foundation Model

The four pillars: self-supervised pretraining on scientific corpora, emergent scientific reasoning, transferability across tasks, and scaling laws that predict performance. The 2024 Nobel Prize in Chemistry as a watershed.

27.2 Protein and Molecular Models

ESM-2 and ESM-3 for protein language modeling; ProGen2 for protein generation; Uni-Mol2 and MolFormer for molecular property prediction. Architecture choices and pretraining strategies.

27.3 Materials and Genomics Models

GNoME for crystal structure prediction; MACE-MP-0 and CHGNet for universal interatomic potentials; Nucleotide Transformer and DNABERT-2 for genomic sequence understanding.

27.4 Fine-Tuning with LoRA

The low-rank adaptation framework: $W = W_0 + BA$. Why it works, how to choose the rank, and practical considerations for scientific models. Comparison with full fine-tuning and other PEFT methods.

27.5 Building a Protein Property Predictor

A complete recipe: loading ESM-2 from Hugging Face, applying LoRA adapters with PEFT, training on the Meltome thermostability dataset, evaluating with Spearman correlation, and deploying the predictor.

What's Next

Foundation models produce powerful representations for single modalities: sequences, graphs, or structures. But scientific discovery increasingly demands models that reason across modalities simultaneously, combining images, spectra, text, and molecular graphs. Chapter 28: Multimodal Scientific AI explores how to fuse these representations and build systems that see, read, and reason about science the way human researchers do.

Bibliography

Foundational Papers

Lin, Z. et al. (2023). "Evolutionary-scale prediction of atomic-level protein structure with a language model." Science, 379(6637), 1123-1130.

The ESM-2/ESMFold paper: a protein language model with 15 billion parameters that predicts 3D structure directly from sequence, rivaling AlphaFold2 without multiple sequence alignments.

Hayes, T. et al. (2024). "Simulating 500 million years of evolution with a language model." Nature.

ESM-3: a multimodal protein language model that jointly reasons over sequence, structure, and function, generating functional proteins with novel folds.

Nijkamp, E. et al. (2023). "ProGen2: Exploring the Boundaries of Protein Language Models." Cell Systems, 14(11), 968-978.

ProGen2: autoregressive protein language models up to 6.4B parameters trained on diverse protein sequences, demonstrating scaling benefits for protein generation.

Merchant, A. et al. (2023). "Scaling deep learning for materials discovery." Nature, 624, 80-85.

GNoME: Google DeepMind's graph neural network that discovered 2.2 million stable crystal structures, expanding known inorganic materials by an order of magnitude.

Hu, E. J. et al. (2022). "LoRA: Low-Rank Adaptation of Large Language Models." ICLR 2022.

The LoRA paper: parameter-efficient fine-tuning by injecting trainable low-rank matrices, reducing trainable parameters by 10,000x while matching full fine-tuning performance.

Models and Architectures

Ji, Y. et al. (2024). "Uni-Mol2: Exploring Molecular Pretraining Model at Scale." ICLR 2025.

Uni-Mol2: a molecular pretraining framework at 1.1B parameters combining atomic, graph, and geometry features, demonstrating consistent scaling improvements.

Ross, J. et al. (2022). "Large-scale chemical language representations capture molecular structure and properties." Nature Machine Intelligence, 4, 1256-1264.

MolFormer: a transformer pretrained on 1.1 billion SMILES strings using linear attention, producing molecular embeddings competitive with graph neural networks.

Batatia, I. et al. (2024). "A foundation model for atomistic simulation." arXiv:2401.00096.

MACE-MP-0: a universal interatomic potential trained on the Materials Project database, providing accurate force predictions across the periodic table.

Deng, B. et al. (2023). "CHGNet as a pretrained universal neural network potential for charge-informed atomistic modelling." Nature Machine Intelligence, 5, 1031-1041.

CHGNet: a graph neural network potential that explicitly tracks magnetic moments and charges, enabling accurate simulation of electrochemistry and magnetism.

Dalla-Torre, H. et al. (2023). "The Nucleotide Transformer." Nature Methods.

Nucleotide Transformer: a family of genomic language models up to 2.5B parameters pretrained on 3,200 genomes, achieving state-of-the-art on 18 genomic prediction tasks.

Zhou, Z. et al. (2024). "DNABERT-2: Efficient Foundation Model and Benchmark for Multi-Species Genome." ICLR 2024.

DNABERT-2: a genome foundation model using byte pair encoding instead of k-mers, achieving strong performance with 117M parameters across 28 datasets.

Tools and Libraries

Meta AI. ESM: Evolutionary Scale Modeling. GitHub.

The official repository for ESM-1b, ESM-2, ESMFold, and ESM-3, with pretrained weights, inference scripts, and fine-tuning examples.

Hugging Face. PEFT: Parameter-Efficient Fine-Tuning. GitHub.

The Hugging Face library implementing LoRA, QLoRA, prefix tuning, and other PEFT methods, with seamless integration into the Transformers ecosystem.

ACEsuit. MACE: Higher Order Equivariant Message Passing Neural Networks. GitHub.

The MACE framework for building equivariant interatomic potentials, including pretrained universal models (MACE-MP-0) and fine-tuning utilities.