Overview
Neural networks are powerful function approximators, but they are opaque. A neural network that predicts planetary orbits with nine decimal places of accuracy tells you nothing about the underlying law of gravitation. Symbolic regression inverts this trade-off: it searches the space of mathematical expressions for compact, interpretable formulas that fit the data. When it works, the result is not a black-box model but a human-readable equation that a scientist can analyze, generalize, and physically interpret.
The idea is simple to state and extraordinarily difficult to execute. The space of possible mathematical expressions is combinatorially vast: even restricting to binary operators $\{+, -, \times, \div\}$ and unary functions $\{\sin, \cos, \exp, \log\}$ over three variables, the number of syntactically valid expression trees of depth 5 exceeds $10^{10}$. Most of these expressions are garbage. A tiny fraction fit the data by coincidence. An even tinier fraction represent genuine physical laws. Finding the needle in this haystack requires both intelligent search and principled model selection.
This chapter covers three complementary approaches to the search. Genetic programming (Section 35.1) evolves populations of expression trees through mutation, crossover, and selection, using biological metaphors to explore the space. Neural-guided symbolic regression (Section 35.2) uses neural networks to propose candidate expressions or guide the search, combining the flexibility of deep learning with the interpretability of symbolic formulas. The equation discovery pipeline (Section 35.3) puts everything together with PySR, dimensional constraints, and SymPy-based validation to recover hidden scientific laws from experimental data.
These techniques connect directly to the scientific machine learning methods of Chapter 33, where neural networks learn solutions to differential equations. Where SciML produces fast but opaque surrogates, symbolic regression aims for the equation itself. They also connect to the discovery as search framework of Chapter 1 (expression trees as search spaces), the reasoning for discovery methods of Chapter 4 (symbolic manipulation), and the optimization techniques of Chapter 45 (fitness landscape navigation). Looking ahead, the interpretable equations produced here feed directly into the hypothesis generation pipelines of Chapter 39 and the AI scientist systems of Chapter 53.
Prerequisites
Readers should be familiar with Chapter 1: Discovery as Search for the general framework of search spaces and heuristics, and with Chapter 25: Exploratory Discovery for data preprocessing and feature selection. Basic familiarity with tree data structures and recursion is assumed; Appendix B covers Python prerequisites. The dimensional analysis in Section 35.3 uses concepts from Chapter 5: Discovery Through Data, Models, and Simulation.
Learning Outcomes
- Represent mathematical expressions as trees and define crossover, mutation, and selection operators for genetic programming.
- Explain how Kolmogorov complexity, minimum description length (MDL), and Pareto-front selection balance accuracy against expression complexity.
- Compare genetic programming, neural-guided search, and reinforcement-learning approaches to symbolic regression.
- Use PySR with dimensional constraints to recover physically consistent equations from noisy data.
- Validate and simplify discovered equations using SymPy's symbolic algebra engine.
- Integrate symbolic regression into a Discovery Workbench pipeline for automated equation discovery.
Sections
35.1 Equation Discovery from Data
Expression trees as search spaces. Fitness, bloat, and the Pareto front of accuracy vs. complexity. Kolmogorov complexity, minimum description length, and information-theoretic model selection. Dimensional analysis as hard constraints. The gplearn library for classical genetic programming.
35.2 Genetic Programming and Neural Guidance
Tournament selection, subtree crossover, and point mutation. Neural-guided symbolic regression with transformers (SymbolicGPT, NeSymReS). Reinforcement learning for expression construction. PySR's multi-population evolutionary algorithm. Scaling to high-dimensional problems.
35.3 Building an Equation Discovery Pipeline
Recipe: recover Kepler's third law and a damped oscillator equation from synthetic data using PySR with dimensional constraints. Validate with SymPy simplification. Integrate with scipy curve fitting. Deploy as a Discovery Workbench component for automated law discovery.
Bibliography
Foundational Papers
The foundational work on genetic programming that established expression-tree evolution as a method for automatic program synthesis.
The PySR paper describing the multi-population evolutionary algorithm with adaptive complexity penalties that is the primary tool in this chapter.
Eureqa: the landmark demonstration that symbolic regression can rediscover known physics laws (Hamiltonians, Lagrangians) from raw experimental data without prior knowledge.
AI Feynman combines neural networks with physics-inspired decomposition (dimensional analysis, symmetry, separability) to solve symbolic regression problems from the Feynman Lectures.
Neural-Guided Approaches
NeSymReS: a set-transformer architecture that generates symbolic expressions directly from datasets, scaling to hundreds of input variables.
Transformer-based symbolic regression that generates expression tokens autoregressively, trained on millions of synthetic equation-data pairs.
Uses reinforcement learning to construct symbolic expressions token-by-token, with a risk-seeking policy gradient that prioritizes finding exact solutions.
Theory and Complexity
The MDL principle provides the information-theoretic foundation for balancing model fit against complexity in symbolic regression.
The definitive reference on algorithmic information theory and Kolmogorov complexity, which provides the theoretical motivation for preferring shorter expressions.
Tools & Libraries
The primary symbolic regression tool used in this chapter: a Python wrapper for SymbolicRegression.jl with multi-population evolution and dimensional constraints.
SymPy provides the symbolic algebra engine used for simplifying, validating, and analyzing discovered equations.
A scikit-learn compatible genetic programming library for symbolic regression, used in Section 35.1 for pedagogical examples.
SciPy's optimization module (curve_fit, minimize) is used for refining constants in discovered symbolic expressions.
Benchmarks & Datasets
SRBench: a comprehensive benchmark of 14 symbolic regression methods on 122 real-world datasets, establishing performance baselines for the field.