Chapter 30: Anomaly and Novelty Discovery

Overview

Most machine learning focuses on pattern recognition: finding regularities in data. But some of the most important scientific discoveries begin with the opposite question: what does not fit the pattern? Pulsars were discovered through radio signals initially dismissed as telescope interference. The cosmic microwave background was first interpreted as antenna noise. Dark energy emerged from supernova brightness measurements that deviated from predicted values. In each case, an anomaly became a discovery.

This chapter builds a complete toolkit for detecting anomalies, novelties, and distribution shifts in scientific data. We start with the critical distinction between outlier detection (finding unusual points in existing data) and novelty detection (recognizing that new observations come from a different generating process). We then move to deep learning methods that scale these ideas to high-dimensional structured data. We tackle the challenge of distribution drift, where what counts as "normal" changes over time. Finally, we assemble a three-method anomaly discovery pipeline with ground-truth validation, the kind of system you would deploy in a real laboratory or observatory.

The ideas here connect directly to the exploratory discovery techniques of Chapter 25, the representation learning of Chapter 26 (which provides the embeddings anomaly detectors operate on), and the Bayesian uncertainty quantification of Chapter 32. They also feed forward into self-driving laboratories (Chapter 55), where anomaly detection is a core safety and discovery mechanism.

Prerequisites

Readers should be comfortable with Chapter 5: Discovery Through Data, Models, and Simulation for the data pipeline concepts, and Chapter 25: Exploratory Discovery for clustering and dimensionality reduction. Familiarity with basic neural network concepts from Chapter 26: Representation Learning is helpful for Section 30.2 on deep anomaly detection. Appendix A covers the probability and linear algebra background.

Learning Outcomes

Distinguish outlier detection, novelty detection, and out-of-distribution detection, and choose the right formulation for a given scientific problem.
Implement and compare LOF, HDBSCAN, Kernel Density Estimation, and Isolation Forest on tabular scientific data.
Build autoencoder, variational autoencoder, and normalizing flow models for deep anomaly detection on high-dimensional data.
Detect distribution drift in streaming scientific data using CUSUM and related methods.
Design a multi-method anomaly discovery pipeline with consensus scoring and ground-truth validation.
Use PyOD, HDBSCAN, scikit-learn, River, and PyTorch to build production anomaly detection systems.

Sections

30.1 Outlier vs. Novelty Detection

Statistical and geometric anomaly detection: LOF, HDBSCAN outlier scores, Kernel Density Estimation, and Isolation Forest. The taxonomy of anomaly types. When anomalies are errors and when they are discoveries.

30.2 Deep Anomaly Detection

Autoencoders, variational autoencoders, and normalizing flows for anomaly scoring. Out-of-distribution detection for neural networks. Scaling anomaly detection to images, spectra, and molecular data.

30.3 Drift and Open-World Learning

CUSUM and related drift detectors for streaming data. Concept drift versus data drift. Open-world learning: handling classes unseen during training. The connection between drift detection and scientific discovery.

30.4 Building an Anomaly Discovery Pipeline

Recipe: a three-method anomaly detection pipeline with consensus scoring, ground-truth validation, and integration into the Discovery Workbench. Evaluating anomaly detectors when labels are scarce.

What's Next

Anomaly detection tells us what is unusual, but not why. Chapter 31: Causal Discovery and Causal Inference tackles the question of causation: given that we have found an unexpected pattern, can we identify the mechanism that produced it? Where anomaly detection flags the symptom, causal inference diagnoses the cause. Together, these two chapters form the core of data-driven scientific reasoning in Part III.

Bibliography

Foundational Papers

Breunig, M. M., Kriegel, H.-P., Ng, R. T., & Sander, J. (2000). LOF: Identifying density-based local outliers. Proc. ACM SIGMOD, 93-104.

Introduced the Local Outlier Factor algorithm, establishing local density comparison as the foundation for density-based anomaly detection.

Liu, F. T., Ting, K. M., & Zhou, Z.-H. (2008). Isolation Forest. Proc. IEEE ICDM, 413-422.

The original Isolation Forest paper, introducing the elegant idea that anomalies are easier to isolate than normal points in random recursive partitions.

Campello, R. J. G. B., Moulavi, D., & Sander, J. (2013). Density-based clustering based on hierarchical density estimates. Advances in Knowledge Discovery and Data Mining, LNCS 7819.

Introduced HDBSCAN, which provides both cluster assignments and outlier scores through its hierarchical density estimation framework.

Kingma, D. P. & Welling, M. (2014). Auto-encoding variational Bayes. Proc. ICLR.

The VAE paper that enabled probabilistic anomaly scoring through learned latent distributions, foundational for deep anomaly detection.

Books

Aggarwal, C. C. (2017). Outlier Analysis, 2nd edition. Springer.

The most comprehensive textbook on outlier detection, covering statistical, proximity-based, clustering-based, and ensemble methods with extensive theoretical foundations.

Tools & Libraries

Zhao, Y., Nasrullah, Z., & Li, Z. (2019). PyOD: A Python toolbox for scalable outlier detection. JMLR, 20(96), 1-7.

The go-to Python library for outlier detection, providing 40+ algorithms with a unified API. Used extensively in Section 30.1 and the pipeline recipe in Section 30.4.

McInnes, L. & Healy, J. (2017). Accelerated hierarchical density-based clustering. Proc. IEEE ICDMW.

The high-performance HDBSCAN implementation used in this chapter for joint clustering and outlier scoring.

Montiel, J., et al. (2021). River: Machine learning for streaming data in Python. JMLR, 22(110), 1-8.

The streaming ML library used in Section 30.3 for online drift detection with CUSUM, ADWIN, and Page-Hinkley methods.

Paszke, A., et al. (2019). PyTorch: An imperative style, high-performance deep learning library. NeurIPS, 32.

The deep learning framework used for autoencoder, VAE, and normalizing flow implementations in Section 30.2.

Tutorials & Surveys

Pang, G., Shen, C., Cao, L., & van den Hengel, A. (2021). Deep learning for anomaly detection: A review. ACM Computing Surveys, 54(2), 1-38.

A thorough survey of deep anomaly detection methods covering autoencoders, GANs, flow models, and self-supervised approaches.

Yang, J., et al. (2022). OpenOOD: Benchmarking generalized out-of-distribution detection. NeurIPS Datasets and Benchmarks.

The benchmark and taxonomy for out-of-distribution detection that standardized evaluation protocols across the field.

Lu, J., Liu, A., Dong, F., Gu, F., Gama, J., & Zhang, G. (2019). Learning under concept drift: A review. IEEE TKDE, 31(12), 2346-2363.

Comprehensive review of concept drift detection and adaptation methods, essential background for Section 30.3.