Towards A Smaller Student: Capacity Dynamic Distillation For Efficient Image Retrieval

Abstract

Previous Knowledge Distillation based efficient image retrieval methods employs a lightweight network as the student model for fast inference. However, the lightweight student model lacks adequate representation capacity for effective knowledge imitation during the most critical early training period, causing final performance degeneration. To tackle this issue, we propose a Capacity Dynamic Distillation framework, which constructs a student model with editable representation capacity. Specifically, the employed student model is initially a heavy model to fruitfully learn distilled knowledge in the early training epochs, and the student model is gradually compressed during the training. To dynamically adjust the model capacity, our dynamic framework inserts a learnable convolutional layer within each residual block in the student model as the channel importance indicator. The indicator is optimized simultaneously by the image retrieval loss and the compression loss, and a retrieval-gui

Stats

• citations21
• S2 citations—
• github stars0
• HF likes0
• heat score10.07
• arxiv keyxie2023towards

Related papers

• Data-efficient Ranking Distillation For Image Retrieval (2020)0.00
• Context Unaware Knowledge Distillation For Image Retrieval (2022)0.60
• Embeddistill: A Geometric Knowledge Distillation For Information Retrieval (2023)0.00
• Dynamic Contrastive Distillation For Image-text Retrieval (2022)11.76
• Curriculum Learning For Dense Retrieval Distillation (2022)11.49
• Coarse-to-fine: Learning Compact Discriminative Representation For Single-stage Image Retrieval (2023)9.35
• Let All Be Whitened: Multi-teacher Distillation For Efficient Visual Retrieval (2023)8.86
• Dns: Distill-and-select For Efficient And Accurate Video Indexing And Retrieval (2021)14.99