Analyzing And Improving Representations With The Soft Nearest Neighbor Loss

Abstract

We explore and expand the \(\textit\{Soft Nearest Neighbor Loss\}\) to measure the \(\textit\{entanglement\}\) of class manifolds in representation space: i.e., how close pairs of points from the same class are relative to pairs of points from different classes. We demonstrate several use cases of the loss. As an analytical tool, it provides insights into the evolution of class similarity structures during learning. Surprisingly, we find that \(\textit\{maximizing\}\) the entanglement of representations of different classes in the hidden layers is beneficial for discrimination in the final layer, possibly because it encourages representations to identify class-independent similarity structures. Maximizing the soft nearest neighbor loss in the hidden layers leads not only to improved generalization but also to better-calibrated estimates of uncertainty on outlier data. Data that is not from the training distribution can be recognized by observing that in the hidden layers, it has fewer

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyfrosst2019analyzing

Related papers

• Learning Distributed Representations With Efficient Softmax Normalization (2023)0.00
• Mixture Of Experts With Soft Nearest Neighbor Loss: Resolving Expert Collapse Via Representation Disentanglement (2026)0.00
• On High-dimensional Modifications Of The Nearest Neighbor Classifier (2024)0.00
• Von Mises-fisher Loss: An Exploration Of Embedding Geometries For Supervised Learning (2021)10.07
• Constellation Loss: Improving The Efficiency Of Deep Metric Learning Loss Functions For Optimal Embedding (2019)0.00
• With A Little Help From My Friends: Nearest-neighbor Contrastive Learning Of Visual Representations (2021)18.76
• Connecting Neural Models Latent Geometries With Relative Geodesic Representations (2025)0.00
• For Manifold Learning, Deep Neural Networks Can Be Locality Sensitive Hash Functions (2021)0.00