Analyzing Local Representations Of Self-supervised Vision Transformers

Abstract

In this paper, we present a comparative analysis of various self-supervised Vision Transformers (ViTs), focusing on their local representative power. Inspired by large language models, we examine the abilities of ViTs to perform various computer vision tasks with little to no fine-tuning. We design evaluation framework to analyze the quality of local, i.e.\ patch-level, representations in the context of few-shot semantic segmentation, instance identification, object retrieval and tracking. We discover that contrastive learning based methods like DINO produce more universal patch representations that can be immediately applied for downstream tasks with no parameter tuning, compared to masked image modeling. The embeddings learned using the latter approach, e.g. in masked autoencoders, have high variance features that harm distance-based algorithms, such as k-NN, and do not contain useful information for most downstream tasks. Furthermore, we demonstrate that removing these high-variance

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyvanyan2023analyzing

Related papers

• Self-supervised Vision Transformers For Writer Retrieval (2024)5.24
• Visual Representation Learning With Self-supervised Attention For Low-label High-data Regime (2022)5.49
• Boosting Vision Transformers For Image Retrieval (2022)15.28
• Decomposing And Interpreting Image Representations Via Text In Vits Beyond CLIP (2024)7.28
• Revisiting Contrastive Methods For Unsupervised Learning Of Visual Representations (2021)3.91
• Hypergraph Vision Transformers: Images Are More Than Nodes, More Than Edges (2025)4.52
• Understanding The Effect Of Using Semantically Meaningful Tokens For Visual Representation Learning (2024)0.00
• Training Vision Transformers For Image Retrieval (2021)0.00