Image Retrieval With Mixed Initiative And Multimodal Feedback

Abstract

How would you search for a unique, fashionable shoe that a friend wore and you want to buy, but you didn't take a picture? Existing approaches propose interactive image search as a promising venue. However, they either entrust the user with taking the initiative to provide informative feedback, or give all control to the system which determines informative questions to ask. Instead, we propose a mixed-initiative framework where both the user and system can be active participants, depending on whose initiative will be more beneficial for obtaining high-quality search results. We develop a reinforcement learning approach which dynamically decides which of three interaction opportunities to give to the user: drawing a sketch, providing free-form attribute feedback, or answering attribute-based questions. By allowing these three options, our system optimizes both the informativeness and exploration capabilities allowing faster image retrieval. We outperform three baselines on three dataset

Stats

• citations11
• S2 citations—
• github stars0
• HF likes0
• heat score8.09
• arxiv keymurrugarrallerena2018image

Related papers

• Dialog-based Interactive Image Retrieval (2018)0.00
• Give Me A Hint! Navigating Image Databases Using Human-in-the-loop Feedback (2018)7.16
• Ask&confirm: Active Detail Enriching For Cross-modal Retrieval With Partial Query (2021)11.68
• Enhancing Product Search Interfaces With Sketch-guided Diffusion And Language Agents (2025)0.00
• Revisiting Relevance Feedback For Clip-based Interactive Image Retrieval (2024)4.52
• Conversational Fashion Image Retrieval Via Multiturn Natural Language Feedback (2021)11.85
• Advancing Image Retrieval With Few-shot Learning And Relevance Feedback (2023)0.00
• Image Retrieval With A Bayesian Model Of Relevance Feedback (2016)0.00