Overview
Scientific discovery rests on reproducibility. If you cannot trace how a result was produced, you cannot verify it, extend it, or trust it. Yet the modern ML research workflow generates a torrent of artifacts: datasets, preprocessing scripts, model checkpoints, hyperparameter configurations, training logs, evaluation metrics, and deployment packages. Without a systematic registry that tracks what was produced, by whom, from what inputs, and under what conditions, even the researcher who ran the experiment may struggle to reproduce it a month later.
This chapter builds the infrastructure layer that makes every other chapter in this book reproducible. We start with the design of experiment registries: systems that assign stable identifiers to every artifact and record the relationships between them. We then formalize provenance using established standards (W3C PROV and OpenLineage) that make lineage information portable across tools and organizations. Finally, we construct a complete three-stage ML pipeline with full artifact tracing, hash verification, and reproducible regeneration from any point in the artifact graph.
The ideas here connect backward to the MLOps infrastructure of Chapter 22, where we introduced model lifecycle management, and to the automated experiment design of Chapter 46, which generates the experiments this chapter tracks. They connect forward to self-driving laboratories (Chapter 55), where provenance tracking becomes the audit trail that allows autonomous systems to operate safely, and to responsible discovery AI (Chapter 57), where reproducibility is an ethical obligation.
Prerequisites
Readers should be comfortable with Chapter 22: MLOps, LLMOps, and AgentOps for model lifecycle and deployment concepts, and with basic Git workflows. Familiarity with Chapter 6: Discovery System Architecture helps with understanding how the experiment registry integrates into the Discovery Workbench. No specialized database or systems engineering background is required.
Learning Outcomes
- Design an experiment registry that tracks code, data, models, configurations, and their relationships through content-addressable hashing.
- Model data lineage using W3C PROV entities, activities, and agents, and serialize provenance graphs for exchange.
- Implement artifact versioning with DVC and Git that separates large binary assets from source code while maintaining a single commit history.
- Build run graphs and artifact DAGs that enable point-in-time reconstruction of any experiment result.
- Integrate MLflow, Weights & Biases, and DVC into a unified provenance pipeline with hash verification at every stage.
- Construct a three-stage ML pipeline (preprocessing, training, evaluation) with full artifact tracing and reproducible regeneration.
Sections
47.1 Registry Design and Artifact Versioning
Content-addressable storage, artifact taxonomy, and versioning strategies. Building a registry data model that tracks code, data, models, and configs with cryptographic hashes. Tools: Git, DVC, MLflow Model Registry, Docker.
47.2 Data Lineage and Provenance Standards
W3C PROV data model, OpenLineage specification, and run graphs. Formalizing the relationships between artifacts, processes, and agents. Building queryable provenance graphs that answer "how was this result produced?"
47.3 Building an Experiment Registry
Recipe: a three-stage ML pipeline with full artifact tracing, hash verification, and reproducible regeneration. Integrating MLflow tracking, DVC pipelines, and the Discovery Workbench into a provenance-complete system.
Bibliography
Standards and Specifications
The W3C standard for provenance interchange, defining entities, activities, and agents as the core primitives for describing how things were produced. The foundation for all provenance work in this chapter.
An open standard for runtime lineage collection across data pipelines. Provides the event-based model for capturing job runs, dataset inputs/outputs, and facets used in Section 47.2.
Tools and Platforms
Introduced MLflow's tracking, projects, models, and registry components. The primary experiment tracking platform used throughout this chapter.
The data versioning tool that extends Git to handle large datasets and model files. Used in Section 47.1 for content-addressable artifact storage and in Section 47.3 for pipeline definitions.
A cloud-native experiment tracking platform with artifact versioning, run comparison, and collaborative features. Compared with MLflow in Section 47.1.
Container technology for capturing the complete runtime environment. Used in Section 47.1 for environment reproducibility.
Research and Surveys
The landmark survey revealing that over 70% of researchers failed to reproduce another scientist's experiments, motivating the infrastructure this chapter builds.
Comprehensive survey of provenance research across databases, workflows, and distributed systems. Provides the taxonomic framework for the provenance models in Section 47.2.
Documents the ML reproducibility crisis and proposes checklists and infrastructure requirements, many of which map directly to registry features built in this chapter.
Early work on automatic provenance capture for ML pipelines, introducing the concept of experiment management systems that inspired tools like MLflow and Neptune.
Data Lineage and Governance
Discusses metadata management at scale, including lineage tracking across heterogeneous data sources, relevant to the multi-tool provenance integration in Section 47.3.
Combines Git, DVC, and MLflow into a collaborative platform for ML projects. Demonstrates the integration pattern used in Section 47.3's unified registry.