Queryadapter: Rapid Adaptation Of Vision-language Models In Response To Natural Language Queries

Abstract

A domain shift exists between the large-scale, internet data used to train a Vision-Language Model (VLM) and the raw image streams collected by a robot. Existing adaptation strategies require the definition of a closed-set of classes, which is impractical for a robot that must respond to diverse natural language queries. In response, we present QueryAdapter; a novel framework for rapidly adapting a pre-trained VLM in response to a natural language query. QueryAdapter leverages unlabelled data collected during previous deployments to align VLM features with semantic classes related to the query. By optimising learnable prompt tokens and actively selecting objects for training, an adapted model can be produced in a matter of minutes. We also explore how objects unrelated to the query should be dealt with when using real-world data for adaptation. In turn, we propose the use of object captions as negative class labels, helping to produce better calibrated confidence scores during adaptati

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keychapman2025queryadapter

Related papers

• Ucdr-adapter: Exploring Adaptation Of Pre-trained Vision-language Models For Universal Cross-domain Retrieval (2024)4.52
• Understanding Retrieval-augmented Task Adaptation For Vision-language Models (2024)0.00
• Cross-modal Adapter: Parameter-efficient Transfer Learning Approach For Vision-language Models (2024)6.77
• Probvlm: Probabilistic Adapter For Frozen Vision-language Models (2023)13.41
• Dynamic Adapter With Semantics Disentangling For Cross-lingual Cross-modal Retrieval (2024)2.26
• AVION: Aerial Vision-language Instruction From Offline Teacher To Prompt-tuned Network (2026)0.00
• RAVEN: Multitask Retrieval Augmented Vision-language Learning (2024)0.00
• Vision-language Modelling For Radiological Imaging And Reports In The Low Data Regime (2023)0.00