46.1 Active Learning

1. The Active Learning Loop

Imagine a chemist with ten million candidate molecules on a shelf and a budget for exactly fifty assays: which fifty vials should she open? Picking at random wastes most of the budget on molecules the model already understands, while a single well-chosen query can collapse an entire region of uncertainty at once. Active learning formalizes that choice as a loop with four components, illustrated in Figure 46.1. A learner \(\mathcal{M}\) (any supervised model) maintains predictions over a space of inputs. A pool \(\mathcal{U}\) of unlabeled examples sits waiting to be queried. A query strategy \(Q\) (also called an acquisition function, where the acquisition function is a scoring rule that ranks candidates by expected informativeness) selects which example(s) from \(\mathcal{U}\) to label next. An oracle \(\mathcal{O}\) (the wet lab, the DFT calculation, the human annotator) provides the true label for the selected example. After each label is obtained, the learner retrains and the cycle repeats. Figure 46.1.1 illustrates active learning loop with query strategies.

A query strategy is a scoring function. It ranks every unlabeled example by how much the model would benefit from learning its label, then selects the top-scoring example(s) for the oracle. A well-chosen strategy can cut labeling costs by an order of magnitude, saving dollars, CPU hours, or patient enrollments. The mechanism is straightforward: the strategy evaluates the model's current predictions over the entire pool, assigns an "informativeness" score to each candidate (using uncertainty, diversity, or expected model change), and picks the highest-scoring candidate. Use a query strategy whenever labeling is expensive relative to model inference and you have a pool of unlabeled candidates. If labels are essentially free (as in some text classification tasks with crowd workers), random sampling may suffice. It also avoids the computational overhead of scoring the entire pool each round.

Formally, at round \(t\) the learner has a labeled set \(\mathcal{L}_t\) and an unlabeled pool \(\mathcal{U}_t\). The query strategy selects a batch \(B_t \subset \mathcal{U}_t\) of size \(b\) (the batch size or acquisition batch). The oracle returns labels \(\{y_x : x \in B_t\}\). The learner updates:

The goal is to maximize model performance (accuracy, F1, calibration (how well predicted probabilities match true frequencies), or any task metric) as a function of the total labeling budget \(T \cdot b\). A good query strategy achieves target performance with fewer labels than random selection. The ratio of random-baseline labels to active-learning labels at equal performance is the sample efficiency gain, the primary metric for evaluating active learning systems. In short: the learner that chooses its own training data learns the same lesson with a fraction of the examples.

2. Uncertainty Sampling

The simplest and most widely used query strategy is uncertainty sampling (Lewis & Gale, 1994): select the example about which the current model is most uncertain. For a probabilistic classifier producing class probabilities \(p(y = c \mid x)\) for classes \(c \in \{1, \ldots, K\}\), three standard uncertainty measures are:

Least confidence: select \(x^* = \arg\max_{x \in \mathcal{U}} \left(1 - \max_c p(y = c \mid x)\right)\). The model is least confident about the example whose top predicted class has the lowest probability.

Margin sampling: select \(x^* = \arg\min_{x \in \mathcal{U}} \left(p(y = c_1 \mid x) - p(y = c_2 \mid x)\right)\), where \(c_1, c_2\) are the top two predicted classes. A small margin means the model cannot distinguish between its best two guesses.

Entropy: select \(x^* = \arg\max_{x \in \mathcal{U}} H[y \mid x] = \arg\max_{x \in \mathcal{U}} \left(-\sum_c p(y = c \mid x) \log p(y = c \mid x)\right)\). Entropy captures uncertainty across all classes simultaneously, not just the top one or two.

The three standard uncertainty measures perform differently in practice; the following code compares them before introducing methods that target epistemic uncertainty directly.

The following implementation compares all three measures on a synthetic classification task.

import numpy as np
from sklearn.gaussian_process import GaussianProcessClassifier
from sklearn.gaussian_process.kernels import RBF
from sklearn.datasets import make_moons
from sklearn.metrics import accuracy_score

def uncertainty_sampling(model, X_pool, method="entropy"):
    """Select the most uncertain example from the pool.

    Args:
        model: fitted sklearn classifier with predict_proba
        X_pool: (n_pool, n_features) unlabeled examples
        method: one of 'entropy', 'least_confidence', 'margin'

    Returns:
        Index into X_pool of the selected example.
    """
    probs = model.predict_proba(X_pool)  # (n_pool, n_classes)

    if method == "entropy":
        # H[y | x] = -sum_c p(c|x) log p(c|x)
        scores = -np.sum(probs * np.log(probs + 1e-10), axis=1)
    elif method == "least_confidence":
        scores = 1.0 - np.max(probs, axis=1)
    elif method == "margin":
        sorted_probs = np.sort(probs, axis=1)
        scores = -(sorted_probs[:, -1] - sorted_probs[:, -2])
    else:
        raise ValueError(f"Unknown method: {method}")

    return np.argmax(scores)


def run_active_learning(X, y, query_fn, n_initial=10, n_rounds=50, seed=42):
    """Run pool-based active learning and return accuracy at each round.

    Args:
        X: (n_samples, n_features) full dataset
        y: (n_samples,) labels
        query_fn: callable(model, X_pool) -> index into X_pool
        n_initial: number of initial labeled examples
        n_rounds: number of acquisition rounds

    Returns:
        List of accuracy values, one per round.
    """
    rng = np.random.RandomState(seed)
    n = len(X)
    indices = rng.permutation(n)
    labeled_idx = list(indices[:n_initial])
    pool_idx = list(indices[n_initial:])

    accuracies = []
    kernel = RBF(length_scale=1.0)

    for t in range(n_rounds):
        model = GaussianProcessClassifier(kernel=kernel, random_state=seed)
        model.fit(X[labeled_idx], y[labeled_idx])
        acc = accuracy_score(y, model.predict(X))
        accuracies.append(acc)

        if not pool_idx:
            break

        # Query the pool
        X_pool = X[pool_idx]
        selected = query_fn(model, X_pool)
        labeled_idx.append(pool_idx.pop(selected))

    return accuracies


# Generate synthetic data
X, y = make_moons(n_samples=500, noise=0.3, random_state=42)

# Compare strategies
random_accs = run_active_learning(
    X, y,
    query_fn=lambda m, Xp: np.random.randint(len(Xp)),
    n_rounds=80
)
entropy_accs = run_active_learning(
    X, y,
    query_fn=lambda m, Xp: uncertainty_sampling(m, Xp, "entropy"),
    n_rounds=80
)
margin_accs = run_active_learning(
    X, y,
    query_fn=lambda m, Xp: uncertainty_sampling(m, Xp, "margin"),
    n_rounds=80
)

# Find rounds needed to reach 90% accuracy
target = 0.90
for name, accs in [("Random", random_accs), ("Entropy", entropy_accs), ("Margin", margin_accs)]:
    rounds_to_target = next((i for i, a in enumerate(accs) if a >= target), None)
    print(f"{name}: reaches {target:.0%} accuracy at round {rounds_to_target}")
# Typical output:
# Random: reaches 90% accuracy at round 52
# Entropy: reaches 90% accuracy at round 18
# Margin: reaches 90% accuracy at round 21

The results in Listing 46.1 demonstrate the core value proposition: entropy-based uncertainty sampling reaches 90% accuracy in roughly 18 rounds, while random selection requires around 52 rounds to reach the same threshold. That is a 2.9x sample efficiency gain. In a setting where each "round" is a \$500 assay, this translates to \$9,000 instead of \$26,000 in experimental costs.

3. Query by Committee

Query by committee (QBC) (Seung et al., 1992) generalizes uncertainty sampling by maintaining a committee of \(M\) models \(\{\mathcal{M}_1, \ldots, \mathcal{M}_M\}\), each consistent with the labeled data seen so far. The query strategy selects the example on which the committee members disagree most. Disagreement is measured by the vote entropy (the Shannon entropy computed over the committee's votes, treating each member's hard prediction as a ballot):

where \(V(c, x)\) is the number of committee members that predict class \(c\) for input \(x\). Alternatively, one can use the Kullback-Leibler (KL) divergence (a measure of how one probability distribution differs from another) between each member's prediction and the committee consensus:

where \(\bar{p}(y \mid x) = \frac{1}{M}\sum_m p_m(y \mid x)\) is the average prediction. This KL-based disagreement measure is closely related to the BALD acquisition function we derive in Section 46.2: when the committee members are posterior samples, the KL divergence equals the mutual information between the prediction and the model parameters.

from sklearn.ensemble import BaggingClassifier
from sklearn.tree import DecisionTreeClassifier

def query_by_committee(committee, X_pool, method="vote_entropy"):
    """Select the example with highest committee disagreement.

    Args:
        committee: list of fitted classifiers with predict_proba
        X_pool: (n_pool, n_features)
        method: 'vote_entropy' or 'kl_divergence'

    Returns:
        Index into X_pool of the selected example.
    """
    n_pool = len(X_pool)

    if method == "vote_entropy":
        # Collect hard votes
        votes = np.array([m.predict(X_pool) for m in committee])  # (M, n_pool)
        n_classes = len(np.unique(votes))
        scores = np.zeros(n_pool)
        for i in range(n_pool):
            counts = np.bincount(votes[:, i], minlength=n_classes)
            freqs = counts / len(committee)
            scores[i] = -np.sum(freqs * np.log(freqs + 1e-10))

    elif method == "kl_divergence":
        # Collect probability predictions
        all_probs = np.array([m.predict_proba(X_pool) for m in committee])
        consensus = all_probs.mean(axis=0)  # (n_pool, n_classes)
        kl_divs = np.zeros(n_pool)
        for m_probs in all_probs:
            kl_divs += np.sum(
                m_probs * np.log((m_probs + 1e-10) / (consensus + 1e-10)),
                axis=1
            )
        scores = kl_divs / len(committee)

    return np.argmax(scores)


def build_committee(X_labeled, y_labeled, n_members=10, seed=42):
    """Build a committee of bagged decision trees."""
    bag = BaggingClassifier(
        estimator=DecisionTreeClassifier(max_depth=8),
        n_estimators=n_members,
        random_state=seed
    )
    bag.fit(X_labeled, y_labeled)
    return bag.estimators_

QBC has an important advantage over uncertainty sampling: it captures epistemic uncertainty (what the model does not know) rather than aleatoric uncertainty (inherent noise in the labels). A point near the true decision boundary may have high aleatoric uncertainty, so additional labels there will not reduce model error. QBC members trained on bootstrap samples (random subsets drawn with replacement from the labeled data) agree on such points, because all see similar noisy data. They disagree on points where the training data is genuinely insufficient. This distinction between uncertainty types is central to the Bayesian treatment in Section 46.2.

Mental Model

Think of active learning like a chef tasting a large pot of soup with a limited number of clean spoons. Random sampling is dipping spoons in at random; you might taste the same well-mixed middle region five times and miss the corner where the salt settled. Uncertainty sampling is tasting wherever the last spoonful was ambiguous ("was that too salty or not?"), which helps refine your judgment about borderline spots but can keep you fixated on one area. Core-set selection (Section 4 below) is spacing your spoons evenly across the entire pot, guaranteeing every region gets at least one taste. The best strategy, like Batch Active learning by Diverse Gradient Embeddings (BADGE) (introduced in Section 5), is tasting spots that are both far apart and ambiguous: you cover the whole pot while focusing extra attention on zones that could go either way. The key mapping: each spoon is a labeling query, the pot is your unlabeled pool, and your evolving opinion about the soup's seasoning is the model being trained.

4. Core-Set Selection

Uncertainty-based methods concentrate queries at the decision boundary. This works well when the boundary is the bottleneck, but it can fail when the model needs coverage of the entire input space to learn good representations. Core-set selection (Sener & Savarese, 2018) takes the opposite approach: it selects examples that are maximally diverse, covering the input space as uniformly as possible.

The core-set objective formalizes this as a covering problem. Given a labeled set \(\mathcal{L}\) and a pool \(\mathcal{U}\), select \(b\) points from \(\mathcal{U}\) to minimize the maximum distance from any pool point to its nearest labeled point:

This is the k-center problem (a combinatorial optimization problem that seeks to place \(k\) centers so that the farthest point from any center is minimized), which is NP-hard but admits a simple greedy 2-approximation (guaranteed to find a solution within twice the optimal cost): iteratively select the pool point that is farthest from the current labeled set.

def coreset_greedy(X_labeled, X_pool, batch_size=1):
    """Greedy core-set selection: pick the farthest point from labeled set.

    Solves the k-center problem with a 2-approximation guarantee.

    Args:
        X_labeled: (n_labeled, n_features) currently labeled points
        X_pool: (n_pool, n_features) candidate points
        batch_size: number of points to select

    Returns:
        List of indices into X_pool.
    """
    from scipy.spatial.distance import cdist

    selected = []
    all_labeled = X_labeled.copy()

    for _ in range(batch_size):
        # Distance from each pool point to nearest labeled point
        dists = cdist(X_pool, all_labeled).min(axis=1)  # (n_pool,)

        # Zero out already-selected points
        for idx in selected:
            dists[idx] = -1.0

        # Select the farthest point
        best = np.argmax(dists)
        selected.append(best)
        all_labeled = np.vstack([all_labeled, X_pool[best:best+1]])

    return selected

Core-set selection excels in representation learning settings where the model's feature space is still forming. In deep active learning for molecular property prediction, a neural network's learned embeddings evolve as training progresses. Running core-set selection in the embedding space (rather than the raw input space) ensures that each acquisition round covers distinct regions of the learned representation. This prevents the model from over-concentrating on one structural family of molecules. The molecular discovery pipelines in Chapter 49 apply this technique.

5. Expected Model Change

Uncertainty and diversity each capture one side of the coin. A direct alternative measures, for each candidate, how much the model's parameters would shift if that label were obtained.

A third family of query strategies measures the expected effect of labeling a point on the model's predictions. The expected gradient length (EGL) strategy (Settles & Craven, 2008) selects the example whose label, once obtained, would cause the largest change to the model parameters:

where \(\mathcal{L}\) is the training loss and \(\theta\) are the model parameters. The expectation is taken over the model's own predictive distribution, since the true label is unknown at query time. Points with high expected gradient length are those where the model's predictions are both uncertain and influential: labeling them would substantially update the parameters.

For neural networks, the gradient \(\nabla_\theta \mathcal{L}\) is readily available from the backward pass. The BADGE algorithm (Ash et al., 2020) uses the gradient embedding (the gradient of the loss with respect to the last layer parameters) as a feature vector, then runs k-means++ initialization (a seeding procedure that picks initial cluster centers with probability proportional to their squared distance from existing centers, promoting spread) on these gradient embeddings to select a diverse, high-gradient batch. This achieves the exploration/exploitation balance we described above without any explicit switching.

import torch
import torch.nn as nn

def expected_gradient_length(model, X_pool_tensor, n_classes):
    """Compute expected gradient length for each pool point.

    Args:
        model: PyTorch classifier (last layer must be nn.Linear)
        X_pool_tensor: (n_pool, n_features) tensor
        n_classes: number of output classes

    Returns:
        (n_pool,) array of expected gradient lengths.
    """
    model.eval()
    egl_scores = []

    with torch.no_grad():
        logits = model(X_pool_tensor)
        probs = torch.softmax(logits, dim=1)  # (n_pool, n_classes)

    # For each pool point, compute E_y[||grad||]
    for i in range(len(X_pool_tensor)):
        x_i = X_pool_tensor[i:i+1].clone().requires_grad_(True)
        grad_norms = []

        for c in range(n_classes):
            model.zero_grad()
            logits_i = model(x_i)
            loss = nn.CrossEntropyLoss()(
                logits_i,
                torch.tensor([c])
            )
            loss.backward()

            # Gradient of loss w.r.t. last layer weights
            last_layer = list(model.parameters())[-2]
            grad_norm = last_layer.grad.norm().item()
            grad_norms.append(grad_norm)

        # Expected gradient length under model's predictive distribution
        p = probs[i].numpy()
        egl = sum(p[c] * grad_norms[c] for c in range(n_classes))
        egl_scores.append(egl)

    return np.array(egl_scores)

6. Comparing Query Strategies

Uncertainty sampling typically converges fastest in the early rounds but can stall when the decision boundary is well-characterized. Core-set selection has a slower start but provides steady improvement through broad coverage. QBC combines elements of both by capturing epistemic uncertainty. EGL and BADGE tend to perform best overall but require differentiable models.

7. Active Learning with modAL

Building query strategies from scratch is instructive, but production active learning systems benefit from a framework that handles the bookkeeping. The modAL library wraps scikit-learn estimators with a pool-based active learning loop, providing built-in uncertainty sampling, QBC, and custom query strategy hooks.

from modAL.models import ActiveLearner, Committee
from modAL.uncertainty import uncertainty_sampling as modal_uncertainty
from modAL.disagreement import vote_entropy_sampling
from sklearn.ensemble import RandomForestClassifier

# Pool-based active learning in 8 lines
learner = ActiveLearner(
    estimator=RandomForestClassifier(n_estimators=100, random_state=42),
    query_strategy=modal_uncertainty,
    X_training=X[:10], y_training=y[:10]
)

X_pool, y_pool = X[10:], y[10:]
for _ in range(50):
    query_idx, _ = learner.query(X_pool)
    learner.teach(X_pool[query_idx], y_pool[query_idx])
    X_pool = np.delete(X_pool, query_idx, axis=0)
    y_pool = np.delete(y_pool, query_idx, axis=0)

print(f"Final accuracy: {learner.score(X, y):.3f}")

# Committee-based active learning
committee = Committee(
    learner_list=[
        ActiveLearner(
            estimator=RandomForestClassifier(
                n_estimators=50, random_state=seed
            ),
            X_training=X[:10], y_training=y[:10]
        )
        for seed in range(5)
    ],
    query_strategy=vote_entropy_sampling
)

8. Batch Active Learning

Selecting one example per round is optimal for information gain but impractical when the oracle can process queries in parallel. In high-throughput screening, a robotic system can run 96 assays simultaneously on a microplate. Batch active learning selects \(b\) examples per round, facing the challenge that selecting the top-\(b\) individually most uncertain points often yields a redundant batch (all points cluster near the same part of the decision boundary).

Three approaches handle batch diversity. Batch Bayesian Active Learning by Disagreement (BatchBALD) (Kirsch et al., 2019) extends the BALD acquisition function (Section 46.2) to jointly optimize the mutual information of the entire batch rather than greedily selecting individual high-BALD points. BADGE (Ash et al., 2020) runs k-means++ in gradient embedding space. Stochastic batches sample from a distribution proportional to individual acquisition scores, introducing randomness that promotes diversity.

def batch_active_learning(model, X_pool, batch_size, strategy="diverse"):
    """Select a diverse batch of points for active learning.

    Args:
        model: fitted model with predict_proba
        X_pool: (n_pool, n_features)
        batch_size: number of points to select
        strategy: 'top_k', 'diverse', or 'stochastic'

    Returns:
        List of indices into X_pool.
    """
    probs = model.predict_proba(X_pool)
    entropy = -np.sum(probs * np.log(probs + 1e-10), axis=1)

    if strategy == "top_k":
        # Naive: just pick the top-b most uncertain
        return list(np.argsort(entropy)[-batch_size:])

    elif strategy == "stochastic":
        # Sample proportional to entropy
        weights = entropy / entropy.sum()
        return list(np.random.choice(
            len(X_pool), size=batch_size, replace=False, p=weights
        ))

    elif strategy == "diverse":
        # Filtered + core-set: pre-filter top 5*batch_size by entropy,
        # then apply core-set selection for diversity
        n_candidates = min(5 * batch_size, len(X_pool))
        candidate_idx = np.argsort(entropy)[-n_candidates:]
        X_candidates = X_pool[candidate_idx]

        # Run greedy core-set on the candidates
        selected_in_candidates = coreset_greedy(
            X_pool[np.argsort(entropy)[:10]],  # Use low-entropy as "labeled"
            X_candidates,
            batch_size=batch_size
        )
        return [candidate_idx[i] for i in selected_in_candidates]

9. When Active Learning Fails

Active learning can match or underperform random sampling in several scenarios. Recognizing these failure modes prevents wasted effort.

Cold start: with too few initial labels, the model's uncertainty estimates are unreliable, and the query strategy makes poor choices. A minimum of 2-5 examples per class is typically needed before uncertainty sampling provides meaningful signal. Core-set selection is more robust to cold start because it does not depend on model predictions.

Distribution shift: if the pool distribution differs from the test distribution, concentrating queries in high-uncertainty regions of the pool may not improve test-time performance. This is particularly problematic in drug discovery when the screening library has different chemical diversity than the therapeutic target space.

Oracle-Side Failure Modes

Label noise: when the oracle provides noisy labels, uncertainty sampling can get trapped re-querying the same noisy region repeatedly. Robust active learning methods (e.g., label cleaning with cross-validation) mitigate this by estimating and correcting for label noise.

Lack of a stopping criterion: the active learning loop as presented runs for a fixed budget \(T \cdot b\), but in practice the marginal value of each new label diminishes as the model improves. Without a principled stopping rule, teams either exhaust their budget or stop too early. The Value of Information framework in Section 46.2 provides a formal criterion: stop when the expected improvement from the next query falls below the cost of obtaining it.

Overfitting to the query distribution: because active learning selects a biased sample (concentrated near decision boundaries), models trained on actively selected data can be miscalibrated. Reweighting the labeled set to correct for selection bias (using importance weights , multipliers that upweight underrepresented regions so the training distribution matches the true data distribution,) helps restore calibration.

Exercises

1. (Conceptual) Explain why uncertainty sampling with a miscalibrated model can perform worse than random sampling. Sketch a scenario with two classes where a model assigns 0.51/0.49 probabilities to all points in one region (due to miscalibration) and 0.99/0.01 to all others. Where will uncertainty sampling concentrate queries? Is this useful?
2. (Coding) Modify the run_active_learning function in Listing 46.1 to support batch acquisition (selecting \(b = 5\) points per round). Compare the "diverse" and "top_k" batch strategies from Listing 46.6 on the two-moons dataset. Plot accuracy vs. total labels acquired.
3. (Analysis) The core-set greedy algorithm has time complexity \(O(b \cdot n_{\text{pool}} \cdot n_{\text{labeled}})\) per round. Derive this, then propose and implement an optimization using a k-d tree that reduces the \(n_{\text{labeled}}\) factor. Measure the speedup for \(n_{\text{pool}} = 100{,}000\).
4. (Research) Design a hybrid query strategy that automatically balances uncertainty sampling and core-set selection based on the current round number. One approach: weight the two scores by \(\alpha_t\) and \((1 - \alpha_t)\) with \(\alpha_t = t / T\) (increasing exploitation over time). Test this on the two-moons dataset and compare to pure strategies.