27.1 What Makes a Scientific Foundation Model

"I was pretrained on every amino acid sequence in UniProt. You could say I have a broad education but no specialization, which is exactly the opposite of a Ph.D."

A Foundation Model Pretending to Be a Scientist

1. From Language Models to Scientific Models

In 2020, a neural network that had never seen a single laboratory measurement predicted the 3D structure of a protein with near-atomic accuracy (within roughly 1 angstrom of experimental coordinates for most targets), addressing a problem that had resisted biophysical methods for roughly half a century. That network was a scientific foundation model, and its success raises a question every computational scientist now faces: what properties allow a model trained on raw, unlabeled data to reason about the physical world?

The foundation model paradigm originated in natural language processing (NLP). GPT, BERT, and their successors demonstrated that a model pretrained on unlabeled text could be adapted to translation, summarization, question answering, and dozens of other tasks by fine-tuning on small labeled datasets. The key insight, as covered in Chapter 26, is that self-supervised pretraining learns general-purpose representations that capture the statistical structure of the data.

Consider the alternative: without self-supervised pretraining, building a protein stability predictor requires tens of thousands of expensive wet-lab measurements, each costing days and thousands of dollars. Self-supervised pretraining sidesteps that bottleneck entirely, letting a model learn the rules of protein behavior from millions of unlabeled sequences already stored in public databases.

Self-supervised pretraining is a training strategy in which the model generates its own supervision signal from unlabeled data by hiding part of the input and learning to reconstruct it. This approach eliminates the need for expensive human annotations or laboratory measurements. Models can learn from millions or billions of examples already stored in scientific databases. The mechanism corrupts the input (masking a token, adding noise to coordinates, or shuffling a graph) and trains the model to recover the original, forcing it to learn the internal rules governing the data. Use self-supervised pretraining when labeled examples are scarce relative to the domain's complexity. If you already have large, high-quality labeled datasets for a narrow task, supervised training from scratch may be simpler and equally effective.

Scientific foundation models apply this same paradigm to scientific data. Instead of English sentences, the pretraining corpus consists of protein sequences, molecular graphs, crystal structures, or DNA sequences. Instead of predicting masked words, the model predicts masked amino acids, masked atoms, or masked nucleotides. The transfer hypothesis carries over directly: a model that predicts the next token in a protein sequence must internalize biophysical rules governing protein folding. Likewise, a language model that predicts the next word must internalize grammar and semantics. In short: teach a model to fill in blanks across millions of scientific records, and it discovers the laws of nature on its own.

Defining the Concept

What. A scientific foundation model is a pretrained model whose representations encode the physical, chemical, or biological laws governing its training domain.

Why. Labeled scientific data is expensive. Measuring protein stability requires wet-lab experiments; computing crystal energies requires density functional theory (DFT) calculations that take hours per structure. Self-supervised pretraining sidesteps this bottleneck by learning from the vast supply of unlabeled sequences, structures, and spectra that scientific databases have accumulated over decades.

How. Through self-supervised objectives tailored to scientific data types: masked language modeling for sequences, denoising for 3D structures, contrastive learning for molecular graphs. The model architecture (transformer, graph neural network (GNN), or equivariant network, a network whose outputs transform predictably when inputs are rotated or translated) is chosen to respect the symmetries and constraints of the data domain.

When. Use a scientific foundation model when your downstream task has limited labeled data but belongs to a domain with abundant unlabeled data. If you have millions of labeled examples and a narrow task, a task-specific model trained from scratch may perform equally well at lower cost. Figure 27.1.1 illustrates the four pillars of scientific foundation models and the pretrain-adapt pipeline.

2. The Four Pillars

Not every large pretrained model is a foundation model. A ResNet pretrained on ImageNet is large and pretrained, but it does not qualify because its representations do not transfer broadly to scientific tasks outside computer vision. Four properties distinguish genuine scientific foundation models.

2.1 Self-Supervised Pretraining on Scientific Corpora

The pretraining corpus defines the model's "world." ESM-2 was pretrained on 65 million protein sequences from UniRef (a clustered, non-redundant protein sequence database derived from UniProt); GNoME was pretrained on millions of crystal structures from the Materials Project and ICSD (Inorganic Crystal Structure Database). The corpus must be large enough to cover the diversity of the domain and clean enough that the model does not memorize artifacts.

The self-supervised objective must be meaningful. For protein sequences, masked language modeling works because predicting a masked amino acid requires understanding the evolutionary and biophysical constraints that determine which residues can occupy each position. For crystal structures, predicting formation energy from composition and geometry works because it requires learning the quantum-mechanical interactions between atoms.

2.2 Emergent Scientific Reasoning

A foundation model is more than a pattern matcher. The most compelling evidence for scientific foundation models is that they exhibit behaviors never explicitly trained. ESM-2 was trained only to predict masked amino acids, yet its internal representations encode 3D contact maps (matrices indicating which pairs of residues are close together in the folded 3D structure), secondary structure, and binding site locations. This emergence is analogous to how GPT models develop arithmetic capabilities from pure text prediction.

We can probe for emergent reasoning by extracting the model's internal representations and testing whether they predict scientific properties not present in the training signal:

import torch
from transformers import AutoTokenizer, AutoModel

# Load ESM-2 (650M parameter variant)
model_name = "facebook/esm2_t33_650M_UR50D"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModel.from_pretrained(model_name)
model.eval()

# Encode a protein sequence (human lysozyme)
sequence = "MKALIVLGLVLLSVTVQGKVFERCELARTLKRLGMDGYRGISLANWMCLAKWESGYNTRATNYNAGDRSTDYGIFQINSRYWCNDGKTPGAVNACHLSCSALLQDNIADAVACAKRVVRDPQGIRAWVAWRNRCQNRDVRQYVQGCGV"
inputs = tokenizer(sequence, return_tensors="pt", padding=True)

with torch.no_grad():
    outputs = model(**inputs)
    # Per-residue embeddings from the last hidden layer
    embeddings = outputs.last_hidden_state[0, 1:-1]  # strip BOS/EOS

print(f"Sequence length: {len(sequence)}")
print(f"Embedding shape: {embeddings.shape}")
print(f"Embedding dimension: {embeddings.shape[1]}")

# Compute pairwise cosine similarity as a proxy for contact prediction
similarity = torch.nn.functional.cosine_similarity(
    embeddings.unsqueeze(0),   # (1, L, d)
    embeddings.unsqueeze(1),   # (L, 1, d)
    dim=2
)
print(f"Contact map shape: {similarity.shape}")
print(f"Mean self-similarity: {similarity.diagonal().mean():.4f}")
print(f"Mean off-diagonal: {similarity[~torch.eye(len(sequence), dtype=bool)].mean():.4f}")

Sequence length: 148
Embedding shape: torch.Size([148, 1280])
Embedding dimension: 1280
Contact map shape: torch.Size([148, 148])
Mean self-similarity: 1.0000
Mean off-diagonal: 0.6823

2.3 Transferability Across Tasks

A foundation model is useful precisely because it transfers. The same ESM-2 checkpoint can be fine-tuned for protein stability prediction, subcellular localization, secondary structure classification, variant effect prediction, and enzyme function annotation. The same GNoME representations transfer to band gap prediction, formation energy estimation, and phonon spectrum calculation.

Transferability arises because pretrained representations capture domain fundamentals: amino acid properties, sequence motifs, co-evolutionary patterns, and implicit structural constraints. These features support nearly any downstream protein task. A model trained from scratch on thermostability data, by contrast, learns only what that narrow dataset can teach, missing the contextual knowledge encoded in 65 million evolutionary sequences.

2.4 Scaling Laws

Perhaps the most remarkable property of foundation models is that their performance improves predictably with scale. Kaplan et al. (2020) showed that language model loss decreases as a power law in model size, dataset size, and compute budget. Scientific foundation models exhibit similar scaling behavior.

A scaling law takes the form:

where \(L\) is the loss on a held-out evaluation set, \(N\) is the number of parameters (or dataset size, or floating-point operations, FLOPs), \(\alpha\) and \(\beta\) are domain-specific constants, and \(L_{\infty}\) is the irreducible error (the Bayes-optimal loss for the task). The exponent \(\beta\) determines how quickly performance improves with scale: larger \(\beta\) means faster improvement.

import numpy as np
from scipy.optimize import curve_fit

# Scaling law demonstration: ESM-2 performance vs. model size
# Data points from Lin et al. (2023), Table 1: contact prediction accuracy
esm2_params = np.array([8e6, 35e6, 150e6, 650e6, 3e9, 15e9])
esm2_contact_acc = np.array([0.324, 0.413, 0.495, 0.536, 0.568, 0.587])

# Fit power law: L(N) = alpha * N^(-beta) + L_inf
def scaling_law(n, alpha, beta, l_inf):
    return alpha * n ** (-beta) + l_inf

popt, pcov = curve_fit(
    scaling_law, esm2_params, 1 - esm2_contact_acc,  # fit error = 1 - acc
    p0=[10.0, 0.1, 0.3],
    bounds=([0, 0, 0], [100, 1, 1])
)
alpha, beta, l_inf = popt
print(f"Scaling law: L(N) = {alpha:.2f} * N^(-{beta:.3f}) + {l_inf:.3f}")
print(f"Scaling exponent beta = {beta:.3f}")

# Predict performance at hypothetical 100B parameters
n_100b = 100e9
predicted_error = scaling_law(n_100b, *popt)
print(f"Predicted error at 100B params: {predicted_error:.3f}")
print(f"Predicted accuracy at 100B params: {1 - predicted_error:.3f}")

3. The 2024 Nobel Prize: A Watershed Moment

Scaling laws tell us how much better a model can become with more parameters and data, but they do not tell us whether the scientific community will trust computational predictions enough to act on them. That question was answered decisively in October 2024.

On October 9, 2024, the Nobel Committee awarded the Chemistry Prize to David Baker, Demis Hassabis, and John Jumper "for computational protein design and protein structure prediction." Hassabis and Jumper received their half for AlphaFold2, the system that solved the 50-year-old protein folding problem. Baker received his half for developing Rosetta-based methods for de novo protein design, which subsequent foundation models like RFdiffusion (a diffusion-based generative model for protein backbone design built on RoseTTAFold) have dramatically accelerated.

This was arguably the first Nobel Prize awarded for work that relies centrally on foundation model techniques. AlphaFold2 is pretrained on the Protein Data Bank (PDB) and multiple sequence alignments (MSAs, tables of evolutionarily related sequences aligned position by position to reveal conserved and variable sites) from UniRef, and it transfers to any protein sequence for which no experimental structure exists. As of 2024, AlphaFold DB contains predicted structures for over 200 million proteins, covering essentially all known protein sequences. ESM-2 and ESMFold then demonstrated that a simpler, purely sequence-based approach (no MSAs) could achieve comparable accuracy for many proteins, opening the door to faster inference and easier integration into discovery pipelines.

The Nobel recognition signals that the scientific community now accepts computational prediction as a first-class method alongside X-ray crystallography, cryo-EM (cryogenic electron microscopy, a technique that images frozen biological samples at near-atomic resolution), and NMR spectroscopy. This recognition validates the foundation model paradigm: pretrain on the available scientific record, and the model can generalize to new scientific questions. The same paradigm now operates across chemistry (Section 27.2), materials science, and genomics (Section 27.3), with each domain adapting the architecture and training objective to its own data modalities.

4. Taxonomy of Scientific Foundation Models

Scientific foundation models can be organized along three axes: the data modality they consume, the architecture they use, and the pretraining objective they optimize. The following table summarizes the major models discussed in Sections 27.2 and 27.3.

Several patterns emerge from this taxonomy. Protein and genomic models favor transformer architectures because sequences are naturally tokenizable. Materials models favor GNNs and equivariant architectures because crystal structures are 3D and must respect rotational and translational symmetries (as discussed in Chapter 3). Molecular models sit at the intersection: SMILES-based models like MolFormer use transformers, while 3D-aware models like Uni-Mol2 combine transformers with geometric features.

5. The Foundation Model Pipeline

Knowing what models exist and how they differ is necessary but not sufficient; a practitioner also needs a concrete workflow for selecting, adapting, and deploying one on a real scientific problem.

Figure 27.1 illustrates the three-stage pipeline that mirrors the pretrain-then-adapt paradigm from NLP:

Stage 1: Select. Choose a pretrained model whose training domain matches your scientific question. If you study proteins, start with ESM-2 or ESM-3. If you study small molecules, consider MolFormer or Uni-Mol2. If you study crystal structures, look at GNoME or MACE-MP-0.

Stage 2: Extract or Adapt. You can either extract fixed embeddings (using the model as a feature extractor) or fine-tune the model on your task-specific data. Extraction is faster and requires no GPU for training; fine-tuning yields better performance when you have enough labeled data. Section 27.4 introduces Low-Rank Adaptation (LoRA), which makes fine-tuning feasible on a single GPU by updating only a small fraction of the parameters.

Stage 3: Evaluate and Deploy. Assess the adapted model on held-out data using domain-appropriate metrics. For regression tasks (stability, binding affinity), use Spearman correlation and root mean square error (RMSE). For classification (function, localization), use area under the receiver operating characteristic curve (AUROC) and precision-recall curves. Deploy the model as a component in your Discovery Workbench (see Chapter 6), connecting it to upstream data pipelines and downstream decision systems.

from transformers import AutoTokenizer, AutoModel
import torch

def extract_protein_embedding(sequence: str,
                               model_name: str = "facebook/esm2_t33_650M_UR50D") -> torch.Tensor:
    """Extract a fixed-length protein embedding from ESM-2.

    Returns the mean-pooled representation across all residues,
    producing a single vector per protein regardless of length.
    """
    tokenizer = AutoTokenizer.from_pretrained(model_name)
    model = AutoModel.from_pretrained(model_name)
    model.eval()

    inputs = tokenizer(sequence, return_tensors="pt", padding=True)
    with torch.no_grad():
        outputs = model(**inputs)

    # Mean pool over residue positions (excluding BOS/EOS tokens)
    residue_embeddings = outputs.last_hidden_state[0, 1:-1]
    protein_embedding = residue_embeddings.mean(dim=0)

    return protein_embedding

# Example: embed two proteins and compare
lysozyme = "MKALIVLGLVLLSVTVQGKVFERCELARTLKRLGMDGYRGISLANWMCLAKWESGYNTRATNYNAGDRSTDYGIFQINSRYWCNDGKTPGAVNACHLSCSALLQDNIADAVACAKRVVRDPQGIRAWVAWRNRCQNRDVRQYVQGCGV"
insulin_a = "GIVEQCCTSICSLYQLENYCN"

emb_lys = extract_protein_embedding(lysozyme)
emb_ins = extract_protein_embedding(insulin_a)

cosine_sim = torch.nn.functional.cosine_similarity(
    emb_lys.unsqueeze(0), emb_ins.unsqueeze(0)
)
print(f"Lysozyme embedding: {emb_lys.shape}")
print(f"Insulin A chain embedding: {emb_ins.shape}")
print(f"Cosine similarity: {cosine_sim.item():.4f}")