Dialog-based Interactive Image Retrieval

Abstract

Existing methods for interactive image retrieval have demonstrated the merit of integrating user feedback, improving retrieval results. However, most current systems rely on restricted forms of user feedback, such as binary relevance responses, or feedback based on a fixed set of relative attributes, which limits their impact. In this paper, we introduce a new approach to interactive image search that enables users to provide feedback via natural language, allowing for more natural and effective interaction. We formulate the task of dialog-based interactive image retrieval as a reinforcement learning problem, and reward the dialog system for improving the rank of the target image during each dialog turn. To mitigate the cumbersome and costly process of collecting human-machine conversations as the dialog system learns, we train our system with a user simulator, which is itself trained to describe the differences between target and candidate images. The efficacy of our approach is demon

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyguo2018dialog

Related papers

• Image Retrieval With Mixed Initiative And Multimodal Feedback (2018)8.09
• Interactive Video Retrieval With Dialog (2019)8.09
• Chatting Makes Perfect: Chat-based Image Retrieval (2023)5.29
• Interactive Text-to-image Retrieval With Large Language Models: A Plug-and-play Approach (2024)10.24
• Revisiting Relevance Feedback For Clip-based Interactive Image Retrieval (2024)4.52
• Give Me A Hint! Navigating Image Databases Using Human-in-the-loop Feedback (2018)7.16
• Learning To Retrieve Videos By Asking Questions (2022)8.82
• Ask&confirm: Active Detail Enriching For Cross-modal Retrieval With Partial Query (2021)11.68