MALM: Mask Augmentation Based Local Matching For Food-recipe Retrieval

Abstract

Image-to-recipe retrieval is a challenging vision-to-language task of significant practical value. The main challenge of the task lies in the ultra-high redundancy in the long recipe and the large variation reflected in both food item combination and food item appearance. A de-facto idea to address this task is to learn a shared feature embedding space in which a food image is aligned better to its paired recipe than other recipes. However, such supervised global matching is prone to supervision collapse, i.e., only partial information that is necessary for distinguishing training pairs can be identified, while other information that is potentially useful in generalization could be lost. To mitigate such a problem, we propose a mask-augmentation-based local matching network (MALM), where an image-text matching module and a masked self-distillation module benefit each other mutually to learn generalizable cross-modality representations. On one hand, we perform local matching between the

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyvoutharoja2023malm

Related papers

• SIMMER: Cross-modal Food Image--recipe Retrieval Via Mllm-based Embedding (2026)0.00
• Enhancing Recipe Retrieval With Foundation Models: A Data Augmentation Perspective (2023)6.77
• Images & Recipes: Retrieval In The Cooking Context (2018)3.58
• Cross-modal Food Retrieval: Learning A Joint Embedding Of Food Images And Recipes With Semantic Consistency And Attention Mechanism (2020)12.10
• Transformer Decoders With Multimodal Regularization For Cross-modal Food Retrieval (2022)14.17
• Recipe1m+: A Dataset For Learning Cross-modal Embeddings For Cooking Recipes And Food Images (2018)17.24
• Cross-modal Retrieval In The Cooking Context: Learning Semantic Text-image Embeddings (2018)0.00
• Cross-lingual Adaptation For Recipe Retrieval With Mixup (2022)5.84