Stationary Representations: Optimally Approximating Compatibility And Implications For Improved Model Replacements

Abstract

Learning compatible representations enables the interchangeable use of semantic features as models are updated over time. This is particularly relevant in search and retrieval systems where it is crucial to avoid reprocessing of the gallery images with the updated model. While recent research has shown promising empirical evidence, there is still a lack of comprehensive theoretical understanding about learning compatible representations. In this paper, we demonstrate that the stationary representations learned by the \(d\)-Simplex fixed classifier optimally approximate compatibility representation according to the two inequality constraints of its formal definition. This not only establishes a solid foundation for future works in this line of research but also presents implications that can be exploited in practical learning scenarios. An exemplary application is the now-standard practice of downloading and fine-tuning new pre-trained models. Specifically, we show the strengths and cri

Stats

• citations2
• S2 citations—
• github stars13
• HF likes0
• heat score5.87
• arxiv keybiondi2024stationary

Related papers

• Learning Compatible Embeddings (2021)13.80
• Towards Backward-compatible Representation Learning (2020)12.93
• Forward Compatible Training For Large-scale Embedding Retrieval Systems (2021)8.09
• Backward-compatible Aligned Representations Via An Orthogonal Transformation Layer (2024)0.00
• Unified Representation Learning For Cross Model Compatibility (2020)5.24
• Fastfill: Efficient Compatible Model Update (2023)0.00
• Metric Compatible Training For Online Backfilling In Large-scale Retrieval (2023)2.26
• Boundary-aware Backward-compatible Representation Via Adversarial Learning In Image Retrieval (2023)9.21