Cross-modal Retrieval And Synthesis (X-MRS): Closing The Modality Gap In Shared Representation Learning
2020 · Ricardo Guerrero, Hai Xuan Pham, Vladimir Pavlovic
Abstract
Computational food analysis (CFA) naturally requires multi-modal evidence of a particular food, e.g., images, recipe text, etc. A key to making CFA possible is multi-modal shared representation learning, which aims to create a joint representation of the multiple views (text and image) of the data. In this work we propose a method for food domain cross-modal shared representation learning that preserves the vast semantic richness present in the food data. Our proposed method employs an effective transformer-based multilingual recipe encoder coupled with a traditional image embedding architecture. Here, we propose the use of imperfect multilingual translations to effectively regularize the model while at the same time adding functionality across multiple languages and alphabets. Experimental analysis on the public Recipe1M dataset shows that the representation learned via the proposed method significantly outperforms the current state-of-the-arts (SOTA) on retrieval tasks. Furthermore,
Authors
(none)
Tags
Stats
Related papers
- Cross-modal Retrieval In The Cooking Context: Learning Semantic Text-image Embeddings (2018)0.00
- Transformer Decoders With Multimodal Regularization For Cross-modal Food Retrieval (2022)14.17
- Cross-modal Food Retrieval: Learning A Joint Embedding Of Food Images And Recipes With Semantic Consistency And Attention Mechanism (2020)12.10
- SIMMER: Cross-modal Food Image--recipe Retrieval Via Mllm-based Embedding (2026)0.00
- Mitigating Cross-modal Representation Bias For Multicultural Image-to-recipe Retrieval (2025)0.00
- CHEF: Cross-modal Hierarchical Embeddings For Food Domain Retrieval (2021)8.35
- MCEN: Bridging Cross-modal Gap Between Cooking Recipes And Dish Images With Latent Variable Model (2020)13.39
- Towards Unbiased Cross-modal Representation Learning For Food Image-to-recipe Retrieval (2025)0.00