Chapter 34: Generative Models for Discovery

Overview

Generative models invert the usual machine learning question. Instead of asking "what label does this data point have?", they ask "what new data points could exist?" In scientific discovery, this inversion is transformative: a generative model trained on known molecules can propose novel drug candidates; one trained on protein backbones can design structures with specific binding properties; one trained on crystal structures can suggest new materials with target band gaps. The model becomes a hypothesis generator, proposing candidates that downstream experiments can validate.

This chapter traces the evolution of generative architectures from variational autoencoders through adversarial training to score-based diffusion models and flow matching, each generation bringing better sample quality, more stable training, and richer conditioning mechanisms. We then confront the central challenge of scientific generation: the data lives in 3D space and obeys physical symmetries. A molecule rotated by 90 degrees is the same molecule, so the generative model must respect SE(3) equivariance. This leads us to equivariant diffusion models (EDM), which have produced breakthroughs in protein design (RFDiffusion, FrameDiff, FoldFlow) and structure prediction (Boltz-1, Chai-1).

The chapter builds on Chapter 26: Representation Learning (latent spaces, encoders, decoders), Chapter 33: Scientific Machine Learning (physics-informed constraints), and Chapter 32: Bayesian Discovery (probabilistic modeling and uncertainty). It connects forward to Chapter 45: Optimization for Discovery (where generative models become components of optimization loops) and Chapter 49: Discovery AI for Chemistry and Materials (domain-specific deployment). The capstone recipe in Section 34.5 integrates diffusion-based structure generation with multi-property scoring and REINFORCE optimization, producing a working molecular generator you can extend to your own discovery problems.

Prerequisites

Readers should be comfortable with neural network basics (backpropagation, loss functions) and PyTorch fundamentals from Chapter 26: Representation Learning. The probability background from Appendix A: Mathematical Foundations (distributions, KL divergence, expectations) is essential. Some familiarity with graph neural networks from Section 26.3 will help with the molecular generation material, but we review the necessary concepts inline. No prior exposure to generative models or computational chemistry is assumed.

Learning Outcomes

Explain the mathematical foundations of VAEs, GANs, diffusion models, and flow matching, and identify when each architecture is appropriate for a given discovery task.
Derive the score function and its role in score-based generative modeling; implement a simple denoising score matching network in PyTorch.
Define SE(3) equivariance and explain why it is necessary for generating physically valid 3D molecular and protein structures.
Use Hugging Face Diffusers and domain-specific tools (RDKit, PyG, torchdrug) to build conditional molecular generators.
Implement a complete molecular generation pipeline with diffusion-based proposal, multi-property scoring, and REINFORCE-based optimization.
Evaluate generative models for scientific discovery using validity, novelty, uniqueness, and task-specific property metrics.

Sections

34.1 Generative Models as Hypothesis Generators

The generative modeling landscape for discovery: VAEs, GANs, autoregressive models, and normalizing flows. ELBO derivation and the reparameterization trick. Mode collapse and training instabilities. Why diffusion models superseded earlier architectures for scientific applications.

34.2 Score-Based Diffusion and Flow Matching

The score function and denoising score matching. DDPM forward and reverse processes. Noise schedules and the continuous-time SDE formulation. Flow matching as an alternative: conditional optimal transport paths. EDM (Karras et al.) preconditioning. Implementation in PyTorch and Diffusers.

34.3 Equivariant Generation for 3D Structures

SE(3) symmetry in molecular and protein generation. Equivariant graph neural networks (EGNN, PaiNN). Equivariant diffusion models for point clouds: EDM framework. Protein backbone generation: RFDiffusion, FrameDiff, FoldFlow. Structure prediction: AlphaFold, Boltz-1, Chai-1.

34.4 Conditional Generation and Evaluation

Classifier-free guidance for property-conditioned generation. Inpainting and motif scaffolding. REINFORCE for non-differentiable reward optimization. Evaluation metrics: validity, novelty, uniqueness, FCD, property distributions. Benchmarks: MOSES, GuacaMol, Therapeutic Data Commons.

34.5 Building a Molecular Generator

Recipe: end-to-end molecular generation pipeline with DiffSBDD-style structure-based drug design. Multi-property scoring (QED, SA, docking proxy). REINFORCE fine-tuning for target optimization. Discovery Workbench integration and deployment patterns.

What's Next

Generative models propose candidate structures; the next step is discovering the mathematical equations that govern them. Chapter 35: Symbolic Regression and Equation Discovery tackles the complementary problem of learning interpretable symbolic expressions from data, using genetic programming, neural-guided search, and large language models. Where this chapter generates molecules and proteins, Chapter 35 generates the equations that describe their behavior, closing the loop between structure generation and physical understanding.

Bibliography

Foundational Papers

Kingma, D. P. & Welling, M. (2014). Auto-encoding variational Bayes. ICLR 2014.

The VAE paper that introduced the reparameterization trick, enabling gradient-based training of latent variable models. Foundation for Section 34.1.

Ho, J., Jain, A., & Abbeel, P. (2020). Denoising diffusion probabilistic models. NeurIPS 2020.

The DDPM paper that made diffusion models practical, connecting denoising score matching to a tractable variational bound. Core of Section 34.2.

Lipman, Y., Chen, R. T. Q., Ben-Hamu, H., Nickel, M., & Le, M. (2023). Flow matching for generative modeling. ICLR 2023.

Flow matching as a simulation-free alternative to diffusion, using conditional optimal transport paths. Covered in Section 34.2.

Wohlwend, J., Corso, G., Passaro, S., Reveiz, M., Leidal, K., Swanson, K., et al. (2024). Boltz-1: Democratizing biomolecular interaction modeling. bioRxiv.

Open-weight biomolecular structure prediction rivaling AlphaFold3, using diffusion on molecular assemblies. Key to Section 34.3.

Protein and Molecular Design

Watson, J. L., Juergens, D., Bennett, N. R., Trippe, B. L., Yim, J., Eisenach, H. E., et al. (2023). De novo design of protein structure and function with RFDiffusion. Nature, 620, 1089-1100.

RFDiffusion applies diffusion to protein backbone generation conditioned on functional motifs, producing experimentally validated designs. Central to Section 34.3.

Yim, J., Trippe, B. L., De Bortoli, V., Mathieu, E., Doucet, A., Barzilay, R., & Jaakkola, T. (2023). SE(3) diffusion model with application to protein backbone generation. ICML 2023.

FrameDiff formulates protein backbone diffusion on the SE(3) manifold using frames, providing the mathematical foundation for equivariant protein generation.

Bose, A. J., Akhound-Sadegh, T., Huguet, G., Fatras, K., Rector-Brooks, J., Liu, C.-H., et al. (2024). SE(3)-Stochastic Flow Matching for protein backbone generation. ICLR 2024.

FoldFlow extends flow matching to SE(3) manifolds for protein generation, achieving competitive diversity and designability.

Tools & Libraries

von Platen, P., Patil, S., et al. (2022). Diffusers: State-of-the-art diffusion models. Hugging Face.

The standard library for diffusion model training and inference, with schedulers, pipelines, and pretrained models. Used throughout Sections 34.2 and 34.5.

Landrum, G. et al. (2023). RDKit: Open-source cheminformatics.

The foundational cheminformatics toolkit for molecular manipulation, property calculation, and SMILES/3D conversion. Used in all molecular generation code.

Fey, M. & Lenssen, J. E. (2019). Fast graph representation learning with PyTorch Geometric. ICLR Workshop 2019.

Graph neural network library providing the GNN layers and molecular graph data structures used in equivariant generation.

Zhu, Z., Shi, C., Zhang, Z., Liu, S., Xu, M., Yuan, X., et al. (2022). TorchDrug: A powerful and flexible machine learning platform for drug discovery. arXiv:2202.08320.

Drug discovery platform with pretrained models, molecular property predictors, and generation benchmarks. Used in the Section 34.5 recipe.

Tutorials & Surveys

Guo, M., Thost, V., Li, B., Das, P., Chen, J., & Matusik, W. (2024). Diffusion models in bioinformatics and computational biology. Nature Reviews Bioengineering, 2, 136-154.

Comprehensive survey of diffusion models for biological applications, covering proteins, molecules, and genomics.

Karras, T., Aittala, M., Aila, T., & Laine, S. (2022). Elucidating the design space of diffusion-based generative models. NeurIPS 2022.

The EDM paper that unified and simplified diffusion model design through principled preconditioning and noise schedule analysis.