"I plotted 47 histograms before breakfast. Three of them screamed. The rest just whispered politely about their distributions."
A Scatter Plot With Opinions
Overview
Exploratory data analysis (EDA) is not a preliminary chore you endure before the "real" modeling begins. It is the first genuine act of scientific inquiry on a new dataset: the moment you discover which questions are even worth asking. In this chapter, we treat EDA as a structured discovery process, moving from raw observation through pattern recognition to hypothesis generation, with modern computational tools doing the heavy lifting at every stage.
We begin by reframing EDA as inquiry rather than visualization (Section 25.1), establishing the intellectual framework and the toolchain (pandas, Polars, DuckDB) that supports it. Section 25.2 introduces the mathematical core: clustering algorithms (k-means, HDBSCAN, Gaussian mixture models) and dimensionality reduction (PCA, UMAP), showing how they partition high-dimensional data into interpretable groups and projections. Section 25.3 tackles the transition from exploration to hypothesis: cluster stability assessment (silhouette scores, DBCV), enrichment analysis, and the use of large language models to generate scientific hypotheses from discovered patterns. Finally, Section 25.4 assembles these components into a complete exploration notebook that clusters real data, projects it with UMAP, and hands the results to an LLM for automated hypothesis generation.
Prerequisites
This chapter assumes you have read Chapter 1 (Discovery as Search), where we framed discovery as navigating a state space, and Chapter 5 (Discovery Through Data, Models, and Simulation), where we introduced data-driven and theory-driven paradigms. Familiarity with Chapter 2 (Scientific Discovery) and the hypothesis-experiment cycle will also help. You should be comfortable with Python, NumPy, and basic statistics (means, variances, distributions). Prior exposure to scikit-learn is useful but not required; we introduce every algorithm from first principles.
What You Will Learn
- Distinguish question-driven exploration from routine summary statistics and apply a systematic observation-pattern-hypothesis workflow to a new dataset.
- Choose among pandas, Polars, and DuckDB for data wrangling based on dataset size and query complexity.
- Apply k-means, HDBSCAN, and Gaussian mixture models and understand the geometry each assumes.
- Use UMAP for dimensionality reduction and explain the topological intuition behind it.
- Evaluate cluster quality with silhouette scores, DBCV, and stability analysis.
- Build an end-to-end exploration notebook that feeds discovered patterns into an LLM for hypothesis generation.
Sections
25.1 Exploratory Data Analysis as Inquiry
Tukey's EDA philosophy and the observation-pattern-hypothesis workflow. The grammar of graphics for principled visualization. Data wrangling at three scales: pandas for in-memory, Polars for medium-scale lazy evaluation, DuckDB for SQL over flat files.
25.2 Clustering and Dimensionality Reduction
k-means, HDBSCAN, Gaussian mixture models, PCA, and UMAP: the geometry of partitioning and projecting high-dimensional data.
25.3 From Exploration to Hypothesis
Cluster validation, enrichment analysis, and using LLMs to transform discovered patterns into testable scientific hypotheses.
25.4 Building an Exploration Notebook
A hands-on recipe: load, cluster, project, visualize, and generate hypotheses with a complete exploration pipeline.
What's Next
Exploratory discovery reveals structure in raw data, but the representations we used (raw features, cluster labels, 2D projections) are shallow. Chapter 26: Representation Learning shows how neural networks learn rich, task-aware embeddings that capture deeper regularities. The UMAP projections and cluster analyses from this chapter become diagnostic tools for understanding learned representations, closing the loop between exploration and modeling.
Bibliography
Foundational Papers
The original UMAP paper, introducing a topological approach to dimensionality reduction that preserves both local and global structure better than t-SNE.
The HDBSCAN paper, which extends DBSCAN to handle clusters of varying density through a hierarchical approach.
Introduced the silhouette coefficient, still the most widely used internal cluster validation metric.
Books
The book that coined the term EDA. Tukey's philosophy of letting data speak before imposing models remains the foundation of modern exploratory work.
A comprehensive reference on clustering algorithms, from partitional and hierarchical methods to density-based and spectral approaches.
Tools & Libraries
The reference UMAP implementation, supporting supervised, semi-supervised, and parametric variants with GPU acceleration via cuML.
Production HDBSCAN implementation with soft clustering, outlier detection, and integration with scikit-learn's API.
A modern DataFrame library that outperforms pandas on medium-to-large datasets through lazy evaluation and columnar memory layout.
An in-process SQL database that queries CSV, Parquet, and DataFrame objects directly, ideal for analytical EDA on datasets too large for pandas.
Interactive visualization library producing zoomable, hoverable charts ideal for exploratory analysis of clusters and projections.
Tutorials & Surveys
A survey of deep learning methods for tabular data, contextualizing when neural embeddings outperform classical EDA features.
Practical guide to clustering in scikit-learn, with algorithm comparison charts and guidance on choosing the right method.
An interactive visual explainer of UMAP's hyperparameters and their effects on projection quality.