26.2 Contrastive Objectives

"My entire worldview is a similarity matrix. Everything I know about the universe, I learned by asking: is this pair more alike than that pair?"

A Cosine Similarity With Philosophical Depth

1. From Mutual Information to InfoNCE

Given two electron micrographs of the same nanoparticle, captured under different staining protocols, what exactly do they share, and how would you teach a neural network to quantify that shared information without a single label?

Without contrastive objectives, teams training on unlabeled scientific data routinely watch their encoders collapse: every protein, every molecule, every micrograph maps to nearly the same vector, and downstream classifiers perform no better than random. Contrastive losses are the mathematical machinery that prevents this collapse and turns raw pairs into representations worth using.

The answer lies in mutual information (MI) maximization, the theoretical foundation of contrastive learning. Given two random variables \(X\) and \(Y\) (for example, two augmented views of the same image), mutual information measures how much knowing one tells you about the other:

If \(X\) and \(Y\) are independent, \(p(x, y) = p(x)p(y)\) and $I(X; Y) = 0$. If \(X\) fully determines \(Y\), $I(X; Y)$ equals the entropy of \(Y\). The goal of contrastive representation learning is to find encoder functions \(f\) such that $I(f(X); f(Y))$ is maximized: the representations should preserve as much information as possible about the relationship between paired views.

What. Mutual information is a measure of statistical dependence between two variables, quantifying the information shared between them in bits (or nats, depending on the log base).

Why. Maximizing MI between representations of paired views forces the encoder to capture the features that are shared across views (semantic content) while discarding features that vary across views (augmentation noise). This is precisely the inductive bias we want.

How. Direct MI estimation is intractable in high dimensions. InfoNCE provides a tractable lower bound that can be optimized with standard gradient descent.

When. The MI maximization framework applies whenever you can define meaningful positive pairs. It is most natural when the pairing captures a notion of semantic equivalence: two views of the same object, an image and its caption, a molecule and its textual description.

Computing MI directly requires knowing the joint and marginal densities, which is intractable for high-dimensional neural network representations. The Information Noise-Contrastive Estimation (InfoNCE) bound (van den Oord et al., 2018) converts MI estimation into a classification problem. Draw a positive pair \((x, y^+)\) from the joint distribution \(p(x, y)\) and \(K\) negative samples \(\{y^-_1, \ldots, y^-_K\}\) from the marginal \(p(y)\). Define a scoring function \(s(x, y)\) (typically cosine similarity of the encoded representations). InfoNCE asks: can you identify which of the \(K+1\) candidates is the true positive?

This is a \((K+1)\)-way softmax classification where the correct class is the positive pair. The critical theoretical result is that minimizing \(\mathcal{L}_{\text{InfoNCE}}\) maximizes a lower bound on mutual information:

As \(K \to \infty\), the bound tightens: more negatives yield a better estimate of MI. This explains the empirical finding that contrastive learning benefits from large numbers of negatives, whether through large batch sizes (SimCLR) or a momentum queue (MoCo). In short: contrastive learning works by converting "which of these candidates is the true partner?" into a tractable lower bound on mutual information, and the quality of that bound scales directly with the number of negatives you can afford.

2. The NT-Xent Loss: InfoNCE in Practice

NT-Xent (Normalized Temperature-scaled Cross-Entropy) is the specific instantiation of InfoNCE used by SimCLR. Given a batch of \(N\) samples, each producing two views, the scoring function is temperature-scaled cosine similarity:

NT-Xent is a contrastive loss function that trains an encoder by treating each sample's augmented partner as the single correct answer in a \((2N-1)\)-way classification problem, where every other sample in the batch serves as a distractor. It matters because it turns the abstract goal of mutual information maximization into a concrete, GPU-friendly softmax cross-entropy computation that scales linearly with batch size. The mechanism works by normalizing embeddings onto the unit hypersphere (the surface of a sphere in high-dimensional space where every vector has length 1, so that dot products equal cosine similarities), computing all pairwise cosine similarities scaled by a temperature parameter, and then applying a standard log-softmax so that the gradient pulls positive pairs together while pushing all other pairs apart. Use NT-Xent (or its close variants) when you have access to large batches of unlabeled data with a natural pairing structure; for smaller batch regimes where you cannot afford thousands of in-batch negatives, prefer a queue-based variant such as MoCo, or a non-contrastive method such as VICReg that avoids the need for explicit negatives altogether.

where \(\tau\) is the temperature parameter. For a positive pair \((z_i, z_j)\) in a batch with \(2N\) total views, the loss for anchor \(z_i\) is:

where \(\text{sim}(u, v) = u^\top v / (\|u\| \|v\|)\) is cosine similarity and \(j\) is the index of the positive partner of anchor \(i\). The full NT-Xent loss averages over all \(2N\) anchors:

The following implementation handles numerical stability and shows how temperature affects the loss landscape.

import torch
import torch.nn.functional as F

def nt_xent_loss(z1: torch.Tensor, z2: torch.Tensor,
                 temperature: float = 0.07) -> torch.Tensor:
    """Compute NT-Xent loss for two sets of L2-normalized embeddings.

    Args:
        z1: (N, D) normalized embeddings from view 1
        z2: (N, D) normalized embeddings from view 2
        temperature: scaling parameter (default 0.07)
    Returns:
        Scalar loss value
    """
    N = z1.size(0)
    z = torch.cat([z1, z2], dim=0)  # (2N, D)

    # Full 2N x 2N cosine similarity matrix
    sim = torch.mm(z, z.t()) / temperature  # (2N, 2N)

    # Remove self-similarity from the denominator
    mask = ~torch.eye(2 * N, dtype=torch.bool, device=z.device)

    # For numerical stability, subtract the max before exp
    sim_max, _ = sim.masked_fill(~mask, float('-inf')).max(
        dim=1, keepdim=True
    )
    sim = sim - sim_max.detach()

    # Denominator: sum of exp(sim) over all non-self entries
    exp_sim = torch.exp(sim) * mask.float()
    log_denom = torch.log(exp_sim.sum(dim=1))

    # Numerator: positive pair similarity
    # Positives: (i, i+N) for i in [0,N) and (i+N, i) for i in [0,N)
    pos_indices = torch.cat([
        torch.arange(N, 2 * N, device=z.device),
        torch.arange(0, N, device=z.device)
    ])
    log_num = sim[torch.arange(2 * N, device=z.device), pos_indices]

    loss = -(log_num - log_denom).mean()
    return loss

# Demonstrate how temperature affects the loss landscape
z1 = F.normalize(torch.randn(256, 64), dim=1)
z2 = F.normalize(torch.randn(256, 64), dim=1)

for tau in [0.01, 0.07, 0.1, 0.5, 1.0]:
    loss = nt_xent_loss(z1, z2, temperature=tau)
    print(f"  tau={tau:.2f}  loss={loss.item():.4f}")

With the NT-Xent loss implemented, a natural question arises: what role does the temperature parameter \(\tau\) buried inside that scoring function actually play, and why does its value matter so much more than a typical hyperparameter?

3. Temperature: The Sharpness Dial

Temperature \(\tau\) is the most underrated hyperparameter in contrastive learning. It controls the sharpness of the softmax distribution over similarity scores. To understand its effect, consider what happens at the extremes.

Low temperature (\(\tau \to 0\)). The softmax becomes a hard argmax. Only the most similar negative matters; the loss is dominated by the hardest negative in the batch. This focuses learning on fine-grained discrimination but can be unstable and sensitive to outliers.

High temperature (\(\tau \to \infty\)). The softmax becomes uniform. All negatives contribute equally regardless of similarity. The loss degenerates toward a constant, providing weak gradients. The model barely distinguishes between hard and easy negatives.

The optimal temperature depends on the data distribution and the encoder capacity. Empirically, \(\tau = 0.07\) is a common default for visual representations with normalized embeddings, while text and scientific data typically benefit from values in the range \(\tau = 0.05\) to \(\tau = 0.1\). The temperature interacts with the embedding dimension and normalization: on the unit hypersphere in \(d\) dimensions, random unit vectors have expected cosine similarity \(0\) with standard deviation \(1/\sqrt{d}\). Lower-dimensional embeddings have higher variance in random similarity, requiring higher temperature to avoid collapse.

import torch
import torch.nn.functional as F
import math

def analyze_temperature_effects(dim: int = 128, n_samples: int = 1000):
    """Show how temperature interacts with embedding dimension."""
    z = F.normalize(torch.randn(n_samples, dim), dim=1)
    sims = torch.mm(z, z.t())

    # Remove diagonal
    mask = ~torch.eye(n_samples, dtype=torch.bool)
    off_diag = sims[mask]

    # Expected statistics of random cosine similarities
    empirical_std = off_diag.std().item()
    theoretical_std = 1 / math.sqrt(dim)

    print(f"Dimension: {dim}")
    print(f"  Empirical std of cosine sims: {empirical_std:.4f}")
    print(f"  Theoretical std (1/sqrt(d)):  {theoretical_std:.4f}")

    # Show softmax concentration at different temperatures
    anchor_sims = sims[0, 1:]  # sims of sample 0 to all others
    for tau in [0.01, 0.07, 0.5]:
        probs = F.softmax(anchor_sims / tau, dim=0)
        entropy = -(probs * probs.log()).sum().item()
        max_prob = probs.max().item()
        print(f"  tau={tau:.2f}: max_prob={max_prob:.6f}, "
              f"entropy={entropy:.2f} "
              f"(uniform={math.log(n_samples - 1):.2f})")

for d in [32, 128, 512]:
    analyze_temperature_effects(dim=d)
    print()

4. Alignment and Uniformity

Wang and Isola (2020) provided an elegant decomposition of what makes contrastive representations good. They identified two independent properties, alignment and uniformity, that together explain the success of contrastive objectives.

Alignment measures how close the representations of positive pairs are. For a distribution of positive pairs \((x, x^+) \sim p_{\text{pos}}\):

Good representations have low alignment loss: positive pairs should map to nearby points. This is the "pull" force in contrastive learning.

Uniformity measures how evenly the representations are spread over the unit hypersphere. For a distribution of samples \(x \sim p_{\text{data}}\):

Good representations have low (very negative) uniformity loss: features should be spread uniformly over the sphere, not collapsed into a few clusters. This is the "push" force. Figure 26.3 illustrates how these two forces interact to produce useful representations. Figure 26.2.1 illustrates alignment and uniformity decomposition on the unit hypersphere.

Mental Model

Think of alignment and uniformity as the two forces governing how guests arrange themselves at a large cocktail party. Alignment is like name tags: people wearing matching tags (positive pairs) should find each other and stand close together. Uniformity is like the host's goal of filling the entire venue evenly, so that guests spread across the whole room rather than everyone crowding into one corner. A party where matched guests cluster together but the crowd fills the room wall to wall is a success. A party where everyone stands in the same spot (collapse) satisfies name-tag matching trivially but wastes the venue. A party where guests scatter randomly fills the room but matched pairs never meet. The contrastive loss is the invisible social pressure that simultaneously nudges tagged pairs closer and keeps the overall crowd dispersed.

The NT-Xent loss implicitly optimizes both objectives. The numerator (positive pair similarity) corresponds to alignment. The denominator (sum over all pairs including negatives) corresponds to uniformity. Decomposing these components during training provides a powerful diagnostic: if alignment is good but uniformity is poor, the model may be collapsing (mapping many distinct inputs to the same representation). If uniformity is good but alignment is poor, the model may not be learning the semantic relationship between positive pairs.

import torch
import torch.nn.functional as F

def alignment_loss(z1: torch.Tensor, z2: torch.Tensor,
                   alpha: float = 2.0) -> torch.Tensor:
    """Alignment: expected distance between positive pairs.

    Args:
        z1, z2: (N, D) L2-normalized embeddings of positive pairs
        alpha: exponent (default 2 for squared distance)
    """
    return (z1 - z2).norm(dim=1).pow(alpha).mean()

def uniformity_loss(z: torch.Tensor, t: float = 2.0) -> torch.Tensor:
    """Uniformity: log of expected Gaussian kernel over all pairs.

    Args:
        z: (N, D) L2-normalized embeddings
        t: kernel bandwidth (default 2)
    """
    sq_pdist = torch.pdist(z, p=2).pow(2)
    return sq_pdist.mul(-t).exp().mean().log()

def monitor_contrastive_quality(encoder, data_pairs, device='cpu'):
    """Track alignment and uniformity during training."""
    encoder.eval()
    with torch.no_grad():
        x1, x2 = data_pairs
        z1 = F.normalize(encoder(x1.to(device)), dim=1)
        z2 = F.normalize(encoder(x2.to(device)), dim=1)

        align = alignment_loss(z1, z2).item()
        z_all = torch.cat([z1, z2], dim=0)
        uniform = uniformity_loss(z_all).item()

    encoder.train()
    return {
        'alignment': align,      # lower is better
        'uniformity': uniform,   # lower (more negative) is better
    }

# Demonstrate: random vs collapsed vs good encoder
dim = 64
N = 500

# Random encoder: decent uniformity, poor alignment
z_rand_1 = F.normalize(torch.randn(N, dim), dim=1)
z_rand_2 = F.normalize(torch.randn(N, dim), dim=1)
print("Random encoder:")
print(f"  Alignment:  {alignment_loss(z_rand_1, z_rand_2):.4f}")
print(f"  Uniformity: {uniformity_loss(z_rand_1):.4f}")

# Collapsed encoder: perfect alignment, terrible uniformity
z_collapse = F.normalize(torch.ones(N, dim), dim=1)
z_collapse_noisy = F.normalize(
    z_collapse + 1e-6 * torch.randn(N, dim), dim=1
)
print("\nCollapsed encoder:")
print(f"  Alignment:  {alignment_loss(z_collapse, z_collapse):.4f}")
print(f"  Uniformity: {uniformity_loss(z_collapse_noisy):.4f}")

# Good encoder: both metrics small
z_good_1 = F.normalize(torch.randn(N, dim), dim=1)
z_good_2 = F.normalize(z_good_1 + 0.1 * torch.randn(N, dim), dim=1)
print("\nGood encoder:")
print(f"  Alignment:  {alignment_loss(z_good_1, z_good_2):.4f}")
print(f"  Uniformity: {uniformity_loss(z_good_1):.4f}")

Alignment and uniformity explain what a good representation looks like, but they also reveal where the training signal comes from: the negatives in the denominator of the loss are the force that prevents collapse and drives uniformity, which raises the question of whether all negatives contribute equally.

5. Hard Negatives and Mining Strategies

Not all negatives are equally informative. An easy negative (a protein from a completely different family compared to your anchor) provides almost no gradient signal because the encoder can already distinguish them. A hard negative (a protein from the same superfamily that performs a different function) forces the encoder to learn subtle discriminative features.

The gradient of the NT-Xent loss with respect to the anchor representation \(z_i\) is dominated by the negatives that have the highest similarity to the anchor. Formally, the contribution of negative \(k\) to the gradient is proportional to:

This is exactly the softmax weight of negative \(k\). At low temperature, the gradient is dominated by the single hardest negative. At high temperature, all negatives contribute equally. This means temperature implicitly controls hard negative mining: lower temperature is equivalent to harder mining.

Explicit hard negative mining strategies include:

• In-batch hard negatives: Within a batch, identify the most similar negatives and up-weight their contribution. This is free but limited by batch diversity.
• Cross-batch negatives (MoCo): The momentum queue accumulates negatives from recent batches, increasing the chance of encountering hard negatives.
• Curriculum negatives: Start training with easy negatives and progressively introduce harder ones as the model improves.
• Mined negatives: Use a separate retrieval index to find the hardest negatives offline, then construct batches that pair each anchor with its hardest negatives from the full dataset.

import torch
import torch.nn.functional as F

def mine_hard_negatives(query_embs: torch.Tensor,
                        corpus_embs: torch.Tensor,
                        labels: torch.Tensor,
                        k: int = 5) -> torch.Tensor:
    """Mine the k hardest negatives for each query.

    A hard negative has high similarity but a different label.

    Args:
        query_embs: (N, D) normalized query embeddings
        corpus_embs: (M, D) normalized corpus embeddings
        labels: (N,) query labels for filtering true positives
        k: number of hard negatives per query

    Returns:
        (N, k) indices into corpus_embs of hardest negatives
    """
    # Compute all pairwise similarities
    sims = torch.mm(query_embs, corpus_embs.t())  # (N, M)

    # Mask out same-label pairs (true positives should not be negatives)
    label_match = labels.unsqueeze(1) == labels.unsqueeze(0)
    sims[label_match[:, :corpus_embs.size(0)]] = -float('inf')

    # Top-k most similar among different-label samples
    _, hard_neg_indices = sims.topk(k, dim=1)
    return hard_neg_indices

# Example: mine hard negatives for scientific abstract embeddings
N = 200
dim = 64
embeddings = F.normalize(torch.randn(N, dim), dim=1)
topic_labels = torch.randint(0, 10, (N,))  # 10 topics

hard_negs = mine_hard_negatives(
    embeddings, embeddings, topic_labels, k=3
)
print(f"Hard negative indices shape: {hard_negs.shape}")  # (200, 3)

# Verify: hard negatives have different labels than the query
for i in range(5):
    neg_labels = topic_labels[hard_negs[i]]
    neg_sims = embeddings[i] @ embeddings[hard_negs[i]].t()
    print(f"  Query {i} (label={topic_labels[i].item()}): "
          f"neg labels={neg_labels.tolist()}, "
          f"sims={[f'{s:.3f}' for s in neg_sims.tolist()]}")

Hard negative mining improves the quality of the training signal flowing through the contrastive loss, but the architecture of the network that sits between the encoder and that loss turns out to matter just as much.

6. The Projection Head Mystery

One of the most surprising findings in contrastive learning is the role of the projection head. SimCLR showed that adding a small multilayer perceptron (MLP) (typically two layers) between the encoder and the contrastive loss improved downstream performance by over 10%, yet the projection head itself is discarded after training. Downstream tasks use the encoder representations \(h = f(x)\), not the projected representations \(z = g(h)\).

Why does this work? The projection head acts as an information bottleneck that absorbs invariance pressure. The contrastive loss forces projected representations to become augmentation-invariant: two crops of the same image must map to the same \(z\). This process discards information (color distribution, spatial location of the crop) that downstream tasks may need. Placing this pressure on \(z = g(h)\) rather than on \(h = f(x)\) lets the encoder features \(h\) retain that information while the contrastive objective still shapes them.

The features \(h\) contain both augmentation-invariant information (object identity, semantic content) and augmentation-variant information (color, position, scale). The projection head performs the lossy selection that the contrastive loss demands, leaving encoder features richer than a direct contrastive objective would allow.

In practice, projection head design follows a few conventions that have emerged from ablation studies. A two-layer MLP with a hidden dimension matching the encoder output (for example, 2048 for a ResNet-50) and a lower output dimension (128 or 256) is the most common configuration. Adding a third layer rarely helps, and a single linear layer underperforms the nonlinear variant because it cannot absorb as much augmentation-specific information. The output dimension of the projection head controls the granularity of the contrastive comparison: too low, and fine-grained distinctions are lost; too high, and the benefits of the bottleneck diminish.

7. Architectural Comparison: SimCLR, MoCo, and CLIP

The three architectures we introduced in Section 26.1 make different tradeoffs along several axes. Figure 26.4 summarizes the data flow and negative-sourcing strategy of each architecture. Understanding these tradeoffs guides architecture selection for scientific applications.

Negative source. SimCLR draws negatives from the current batch, requiring large batch sizes (4096+) and multi-GPU training. MoCo draws negatives from a momentum queue (65K+), enabling single-GPU training. CLIP draws negatives from cross-modal pairings within the batch, scaling with batch size but benefiting from natural diversity in image-text pairs.

Encoder consistency. SimCLR uses a single encoder for both views (perfectly consistent). MoCo uses a momentum encoder for keys, introducing a small inconsistency that smooths out over training. CLIP uses two separate encoders for the two modalities, which is necessary because images and text require fundamentally different architectures.

Augmentation requirements. SimCLR and MoCo require carefully designed augmentations for a single modality. CLIP requires paired data across modalities (image-caption pairs) but avoids the augmentation design problem because the natural variation between modalities provides the contrastive signal.

Transfer mechanism. SimCLR and MoCo produce a single-modality encoder whose representations transfer via fine-tuning or linear probing (training a single linear classifier on top of frozen encoder features to evaluate representation quality). CLIP produces aligned cross-modal representations that enable zero-shot transfer: classify images by comparing their embeddings to text descriptions of classes, with no training on the target task. We explore this zero-shot paradigm further in Chapter 27.

import torch
import torch.nn as nn
import torch.nn.functional as F

class CLIPDualEncoder(nn.Module):
    """Simplified CLIP-style dual encoder for text-molecule pairing."""

    def __init__(self, text_encoder: nn.Module, mol_encoder: nn.Module,
                 text_dim: int, mol_dim: int, proj_dim: int = 256,
                 temperature: float = 0.07):
        super().__init__()
        self.text_encoder = text_encoder
        self.mol_encoder = mol_encoder

        # Project both modalities to shared space
        self.text_proj = nn.Linear(text_dim, proj_dim, bias=False)
        self.mol_proj = nn.Linear(mol_dim, proj_dim, bias=False)

        # Learnable temperature (CLIP uses log-parameterization)
        self.log_temp = nn.Parameter(
            torch.tensor(1.0 / temperature).log()
        )

    def forward(self, text_features: torch.Tensor,
                mol_features: torch.Tensor):
        """Compute CLIP-style symmetric contrastive loss.

        Pairs are assumed aligned: text[i] describes mol[i].
        """
        # Project and normalize
        t = F.normalize(self.text_proj(text_features), dim=1)
        m = F.normalize(self.mol_proj(mol_features), dim=1)

        # Temperature-scaled similarity
        temp = self.log_temp.exp()
        logits = torch.mm(t, m.t()) * temp  # (N, N)

        # Symmetric loss: text->mol and mol->text
        labels = torch.arange(t.size(0), device=t.device)
        loss_t2m = F.cross_entropy(logits, labels)
        loss_m2t = F.cross_entropy(logits.t(), labels)

        return (loss_t2m + loss_m2t) / 2

    @torch.no_grad()
    def zero_shot_classify(self, mol_features: torch.Tensor,
                           class_text_features: torch.Tensor):
        """Zero-shot: classify molecules by similarity to text."""
        m = F.normalize(self.mol_proj(mol_features), dim=1)
        t = F.normalize(self.text_proj(class_text_features), dim=1)
        sims = torch.mm(m, t.t())  # (N_mol, N_classes)
        return sims.argmax(dim=1)

# Example usage with simple encoders
text_enc = nn.Sequential(nn.Linear(768, 256), nn.ReLU())
mol_enc = nn.Sequential(nn.Linear(512, 256), nn.ReLU())

clip_model = CLIPDualEncoder(
    text_enc, mol_enc, text_dim=256, mol_dim=256, proj_dim=128
)

# Simulated paired data: descriptions and molecular fingerprints
text_feats = torch.randn(64, 768)
mol_feats = torch.randn(64, 512)

loss = clip_model(text_enc(text_feats), mol_enc(mol_feats))
print(f"CLIP loss: {loss.item():.4f}")

import open_clip

model, preprocess_train, preprocess_val = (
    open_clip.create_model_and_transforms(
        'ViT-B-32', pretrained='laion2b_s34b_b79k'
    )
)
tokenizer = open_clip.get_tokenizer('ViT-B-32')

# Zero-shot classification in 3 lines
image_features = model.encode_image(
    preprocess_val(image).unsqueeze(0)
)
text_features = model.encode_text(tokenizer(["a dog", "a cat"]))
probs = (image_features @ text_features.T).softmax(dim=-1)

8. When Contrastive Objectives Fail

Contrastive objectives are powerful but not universal. They can fail in several well-characterized ways that practitioners should watch for.

False negatives. Contrastive losses assume that all non-paired samples in a batch are true negatives, meaning they are semantically unrelated to the anchor. When the dataset contains many semantically similar samples, this assumption breaks: random negatives from the batch may actually be positives. In a dataset of drug molecules, two random molecules might share the same mechanism of action. Treating them as negatives pushes apart representations that should be close. This is especially problematic in scientific domains with continuous similarity (protein sequence space, chemical property space) rather than discrete categories.

Augmentation misspecification. If augmentations are too weak, the model can distinguish views by augmentation artifacts rather than semantic content (a shortcut solution). If augmentations are too strong, they destroy the semantic information you want to preserve. For scientific data, the appropriate augmentation depends heavily on the downstream task, as we discussed in Section 26.1.

Dimensional collapse. Full representation collapse, where every input maps to the same point, is the extreme failure. A subtler variant is dimensional collapse, where the model uses only a subset of the available embedding dimensions while the rest carry no information, effectively reducing the capacity of the embedding space. The uniformity metric from subsection 4 can detect this: dimensional collapse manifests as poor uniformity even when alignment is good. Centering and whitening the representations (as in the W-MSE variant, which replaces cosine similarity with whitened mean squared error to decorrelate embedding dimensions) can mitigate this failure mode.

Section 26.3 develops quantitative tools for detecting all three failure modes as part of a broader treatment of representation evaluation.