End-to-end Generative Pretraining For Multimodal Video Captioning

Abstract

Recent video and language pretraining frameworks lack the ability to generate sentences. We present Multimodal Video Generative Pretraining (MV-GPT), a new pretraining framework for learning from unlabelled videos which can be effectively used for generative tasks such as multimodal video captioning. Unlike recent video-language pretraining frameworks, our framework trains both a multimodal video encoder and a sentence decoder jointly. To overcome the lack of captions in unlabelled videos, we leverage the future utterance as an additional text source and propose a bidirectional generation objective -- we generate future utterances given the present mulitmodal context, and also the present utterance given future observations. With this objective, we train an encoder-decoder model end-to-end to generate a caption from raw pixels and transcribed speech directly. Our model achieves state-of-the-art performance for multimodal video captioning on four standard benchmarks, as well as for othe

Stats

• citations129
• S2 citations—
• github stars0
• HF likes0
• heat score15.85
• arxiv keyseo2022end

Related papers

• Mm-narrator: Narrating Long-form Videos With Multimodal In-context Learning (2023)10.35
• Classifier-guided Captioning Across Modalities (2025)0.00
• A Better Use Of Audio-visual Cues: Dense Video Captioning With Bi-modal Transformer (2020)10.61
• VX2TEXT: End-to-end Learning Of Video-based Text Generation From Multimodal Inputs (2021)12.17
• Taming Text-to-sounding Video Generation Via Advanced Modality Condition And Interaction (2025)0.00
• Multimodal Transformer Networks For End-to-end Video-grounded Dialogue Systems (2019)17.12
• Learning Audio-video Modalities From Image Captions (2022)12.54
• Diverse Audio Captioning Via Adversarial Training (2021)0.00