VX2TEXT: End-to-end Learning Of Video-based Text Generation From Multimodal Inputs

Abstract

We present \textsc\{Vx2Text\}, a framework for text generation from multimodal inputs consisting of video plus text, speech, or audio. In order to leverage transformer networks, which have been shown to be effective at modeling language, each modality is first converted into a set of language embeddings by a learnable tokenizer. This allows our approach to perform multimodal fusion in the language space, thus eliminating the need for ad-hoc cross-modal fusion modules. To address the non-differentiability of tokenization on continuous inputs (e.g., video or audio), we utilize a relaxation scheme that enables end-to-end training. Furthermore, unlike prior encoder-only models, our network includes an autoregressive decoder to generate open-ended text from the multimodal embeddings fused by the language encoder. This renders our approach fully generative and makes it directly applicable to different "video+\(x\) to text" problems without the need to design specialized network heads for eac

Stats

• citations41
• S2 citations—
• github stars0
• HF likes0
• heat score12.17
• arxiv keylin2021vx2text

Related papers

• Deepaudio-v1:towards Multi-modal Multi-stage End-to-end Video To Speech And Audio Generation (2025)0.00
• 3mdit: Unified Tri-modal Diffusion Transformer For Text-driven Synchronized Audio-video Generation (2025)0.00
• I-code V2: An Autoregressive Generation Framework Over Vision, Language, And Speech Data (2023)3.58
• End-to-end Generative Pretraining For Multimodal Video Captioning (2022)15.85
• Taming Text-to-sounding Video Generation Via Advanced Modality Condition And Interaction (2025)0.00
• Diverse And Aligned Audio-to-video Generation Via Text-to-video Model Adaptation (2023)11.19
• Multimodal Transformer Networks For End-to-end Video-grounded Dialogue Systems (2019)17.12
• Mechanisms Of Multimodal Synchronization: Insights From Decoder-based Video-text-to-speech Synthesis (2024)0.00