The Curious Layperson: Fine-grained Image Recognition Without Expert Labels

Abstract

Most of us are not experts in specific fields, such as ornithology. Nonetheless, we do have general image and language understanding capabilities that we use to match what we see to expert resources. This allows us to expand our knowledge and perform novel tasks without ad-hoc external supervision. On the contrary, machines have a much harder time consulting expert-curated knowledge bases unless trained specifically with that knowledge in mind. Thus, in this paper we consider a new problem: fine-grained image recognition without expert annotations, which we address by leveraging the vast knowledge available in web encyclopedias. First, we learn a model to describe the visual appearance of objects using non-expert image descriptions. We then train a fine-grained textual similarity model that matches image descriptions with documents on a sentence-level basis. We evaluate the method on two datasets and compare with several strong baselines and the state of the art in cross-modal retrieva

Stats

• citations8
• S2 citations—
• github stars25
• HF likes0
• heat score9.99
• arxiv keychoudhury2021the

Related papers

• Grafit: Learning Fine-grained Image Representations With Coarse Labels (2020)12.93
• How A General-purpose Commonsense Ontology Can Improve Performance Of Learning-based Image Retrieval (2017)6.34
• Improving Image Recognition By Retrieving From Web-scale Image-text Data (2023)9.41
• Learning From One And Only One Shot (2022)5.24
• Fine-tuning CNN Image Retrieval With No Human Annotation (2017)22.75
• Exploiting Unlabelled Photos For Stronger Fine-grained SBIR (2023)10.61
• Fine-grained Apparel Classification And Retrieval Without Rich Annotations (2018)0.00
• Caption-matching: A Multimodal Approach For Cross-domain Image Retrieval (2024)0.00