VALOR: Vision-audio-language Omni-perception Pretraining Model And Dataset

Abstract

In this paper, we propose a Vision-Audio-Language Omni-peRception pretraining model (VALOR) for multi-modal understanding and generation. Different from widely-studied vision-language pretraining models, VALOR jointly models relationships of vision, audio and language in an end-to-end manner. It contains three separate encoders for single modality representations, and a decoder for multimodal conditional text generation. We design two pretext tasks to pretrain VALOR model, including Multimodal Grouping Alignment (MGA) and Multimodal Grouping Captioning (MGC). MGA projects vision, language and audio to the same common space, building vision-language, audio-language and audiovisual-language alignment simultaneously. MGC learns how to generate text tokens in conditions of vision, audio or their both. To promote vision-audio-language pretraining research, we construct a large-scale high-quality tri-modality dataset named VALOR-1M, which contains 1M audiable videos with human annotated audi

Stats

• citations25
• S2 citations—
• github stars0
• HF likes0
• heat score10.61
• arxiv keyliu2023valor

Related papers

• VAST: A Vision-audio-subtitle-text Omni-modality Foundation Model And Dataset (2023)14.55
• OLKAVS: An Open Large-scale Korean Audio-visual Speech Dataset (2023)4.52
• VITA: Towards Open-source Interactive Omni Multimodal LLM (2024)0.00
• Capybara-omni: An Efficient Paradigm For Building Omni-modal Language Models (2025)0.00
• Modality-independent Teachers Meet Weakly-supervised Audio-visual Event Parser (2023)4.77
• Unified Video-language Pre-training With Synchronized Audio (2024)0.00
• OMCAT: Omni Context Aware Transformer (2024)0.00
• Omni-avsr: Towards Unified Multimodal Speech Recognition With Large Language Models (2025)2.26