Overview
Building a model is the easy part. Keeping it alive, honest, and cost-effective in production is where the real engineering begins. MLOps, LLMOps, and AgentOps represent three generations of operational discipline, each layered on the last: MLOps manages the lifecycle of trainable models (data pipelines, feature stores, experiment tracking, model registries, distribution shift monitoring). LLMOps extends the paradigm to large language models, where the "model" is a frozen API endpoint and the moving parts are prompts, retrieval contexts, evaluation harnesses, and token budgets. AgentOps adds another layer for autonomous agents, where a single user request triggers chains of tool calls, each with its own latency, cost, failure mode, and observability requirements.
This chapter traces the full lifecycle from data ingestion to retirement. We begin with model lifecycle management: data pipelines, feature stores, experiment tracking with MLflow and Weights & Biases, model registries, and the statistical machinery for detecting when production data drifts away from training data. We then move to LLMOps and AgentOps: prompt versioning, evaluation frameworks, cost optimization, trace logging with LangSmith and Arize Phoenix, tool-call monitoring, and governance through model cards and audit trails. Finally, we build a complete lifecycle tracking dashboard that integrates experiment tracking, prompt evaluation, agent tracing, and drift detection into a single Discovery Workbench component.
The connection to discovery is direct. Scientific models, like production ML models, degrade when the world changes. The distribution shift detection techniques from this chapter apply equally to a recommendation system seeing new user behavior and to a molecular property predictor encountering a novel chemical scaffold. The experiment tracking discipline connects to the provenance systems of Chapter 47. The DevOps pipelines from Chapter 21 become the deployment substrate for everything we build here. And the evaluation frameworks anticipate the agent evaluation methods of Chapter 23.
Prerequisites
Readers should have completed Chapter 21: AI for DevOps and Platform Engineering, which introduced CI/CD pipelines, infrastructure as code, and observability. Familiarity with Chapter 17: Multi-Agent Software Teams is helpful for understanding the agent orchestration patterns that AgentOps monitors. Basic knowledge of scikit-learn model training, REST APIs, and Docker containers is assumed. Experience with the Discovery Workbench from Chapter 6 is helpful for the dashboard recipe in Section 22.3.
Learning Outcomes
- Design data pipelines and feature stores that ensure reproducible model training and serving.
- Track experiments, register models, and manage lifecycle stages using MLflow and Weights & Biases.
- Detect distribution shift in production data using Maximum Mean Discrepancy (MMD) and Population Stability Index (PSI).
- Version prompts, evaluate LLM outputs, and manage token costs in LLMOps workflows.
- Instrument agent pipelines with trace logging, tool-call monitoring, and cost attribution using LangSmith and Arize Phoenix.
- Author model cards and maintain governance audit trails for deployed AI systems.
- Build a lifecycle tracking dashboard integrating experiment tracking, evaluation, tracing, and drift detection.
Sections
22.1 Model Lifecycle Management
Data pipelines and feature stores for reproducible training. Experiment tracking with MLflow. Model registries and lifecycle stages. Distribution shift detection with MMD and PSI. DVC for data versioning. Governance through model cards.
22.2 LLMOps and AgentOps
Prompt versioning and management. LLM evaluation frameworks: automated scoring, human preference, and LLM-as-judge. Cost optimization and token budgeting. Agent trace logging with LangSmith and Arize Phoenix. Tool-call monitoring, retry policies, and circuit breakers.
22.3 Building a Lifecycle Dashboard
Recipe: a complete lifecycle tracking dashboard integrating MLflow experiment data, prompt evaluation results, agent traces, and distribution shift alerts into a unified Discovery Workbench component.
Bibliography
Foundational Papers
The foundational paper on ML systems debt, showing that the model training code is a small fraction of a production ML system. Motivates the entire MLOps discipline covered in Section 22.1.
Introduces the model card framework for documenting trained models, including intended use, performance metrics, and ethical considerations. Used in Section 22.1 for governance.
An interview study of ML practitioners documenting the gap between model development and production deployment, grounding the practical challenges addressed throughout this chapter.
Tools and Libraries
Experiment tracking, model registry, and deployment platform. Central to Sections 22.1 and 22.3 for lifecycle management and the dashboard recipe.
Experiment tracking, hyperparameter sweeps, and model evaluation platform with rich visualization. Used in Section 22.1 as an alternative to MLflow.
Git-based data and model versioning that tracks large files without storing them in the repository. Covered in Section 22.1 for data pipeline reproducibility.
Tracing, evaluation, and monitoring platform for LLM and agent applications. Used in Section 22.2 for agent observability.
Open-source LLM observability with trace visualization, evaluation, and embedding analysis. Used in Section 22.2 alongside LangSmith.
Research Frontiers
Surveys the data management challenges specific to LLM development and deployment, connecting traditional MLOps data pipelines to the LLMOps paradigm.
Establishes the LLM-as-judge evaluation paradigm and benchmarks for comparing judge models, foundational to the evaluation frameworks in Section 22.2.
Proposes agents that learn from their own execution traces, connecting the trace logging infrastructure of AgentOps to agent self-improvement.
Argues that modern AI applications are compound systems (retrieval + models + tools), requiring operational practices beyond single-model MLOps.