Learning From Multiview Correlations In Open-domain Videos

Abstract

An increasing number of datasets contain multiple views, such as video, sound and automatic captions. A basic challenge in representation learning is how to leverage multiple views to learn better representations. This is further complicated by the existence of a latent alignment between views, such as between speech and its transcription, and by the multitude of choices for the learning objective. We explore an advanced, correlation-based representation learning method on a 4-way parallel, multimodal dataset, and assess the quality of the learned representations on retrieval-based tasks. We show that the proposed approach produces rich representations that capture most of the information shared across views. Our best models for speech and textual modalities achieve retrieval rates from 70.7% to 96.9% on open-domain, user-generated instructional videos. This shows it is possible to learn reliable representations across disparate, unaligned and noisy modalities, and encourages using the

Stats

• citations5
• S2 citations—
• github stars0
• HF likes0
• heat score5.84
• arxiv keyholzenberger2018learning

Related papers

• Multi-view Document Representation Learning For Open-domain Dense Retrieval (2022)10.21
• Generalized Multi-view Embedding For Visual Recognition And Cross-modal Retrieval (2016)14.69
• Self-weighted Multiview Metric Learning By Maximizing The Cross Correlations (2019)0.00
• Learning Shared Semantic Space With Correlation Alignment For Cross-modal Event Retrieval (2019)10.21
• See, Hear, And Read: Deep Aligned Representations (2017)0.00
• Domain Adaptation In Multi-view Embedding For Cross-modal Video Retrieval (2021)0.00
• Learning Joint Embedding For Cross-modal Retrieval (2019)5.84
• Cross-modal Discrete Representation Learning (2021)10.61