Softstep: Learning Sparse Similarity Powers Deep Neighbor-based Regression

Abstract

Neighbor-based methods are a natural alternative to linear prediction for tabular data when relationships between inputs and targets exhibit complexity such as nonlinearity, periodicity, or heteroscedasticity. Yet in deep learning on unstructured data, nonparametric neighbor-based approaches are rarely implemented in lieu of simple linear heads. This is primarily due to the ability of systems equipped with linear regression heads to co-learn internal representations along with the linear head's parameters. To unlock the full potential of neighbor-based methods in neural networks we introduce SoftStep, a parametric module that learns sparse instance-wise similarity measures directly from data. When integrated with existing neighbor-based methods, SoftStep enables regression models that consistently outperform linear heads across diverse architectures, domains, and training scenarios. We focus on regression tasks, where we show theoretically that neighbor-based prediction with a mean squ

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keysusman2025softstep

Related papers

• Explaining The Success Of Nearest Neighbor Methods In Prediction (2025)18.63
• Efficient Learning Of Neighbor Representations For Boundary Trees And Forests (2018)0.00
• Representation Learning By Reconstructing Neighborhoods (2018)0.00
• Analyzing And Improving Representations With The Soft Nearest Neighbor Loss (2019)0.00
• Kernel Similarity Matching With Hebbian Neural Networks (2022)0.00
• Efficient Optimization Methods For Extreme Similarity Learning With Nonlinear Embeddings (2020)3.58
• Deep Metric Learning Using Similarities From Nonlinear Rank Approximations (2019)2.26
• Learning Deep Nearest Neighbor Representations Using Differentiable Boundary Trees (2017)0.00