Hierarchical Multimodal Transformer To Summarize Videos

Abstract

Although video summarization has achieved tremendous success benefiting from Recurrent Neural Networks (RNN), RNN-based methods neglect the global dependencies and multi-hop relationships among video frames, which limits the performance. Transformer is an effective model to deal with this problem, and surpasses RNN-based methods in several sequence modeling tasks, such as machine translation, video captioning, *etc*. Motivated by the great success of transformer and the natural structure of video (frame-shot-video), a hierarchical transformer is developed for video summarization, which can capture the dependencies among frame and shots, and summarize the video by exploiting the scene information formed by shots. Furthermore, we argue that both the audio and visual information are essential for the video summarization task. To integrate the two kinds of information, they are encoded in a two-stream scheme, and a multimodal fusion mechanism is developed based on the hierarchical transfor

Stats

• citations90
• S2 citations—
• github stars0
• HF likes0
• heat score14.69
• arxiv keyzhao2021hierarchical

Related papers

• Multimodal Frame-scoring Transformer For Video Summarization (2022)0.00
• A Better Use Of Audio-visual Cues: Dense Video Captioning With Bi-modal Transformer (2020)10.61
• Realizing Video Summarization From The Path Of Language-based Semantic Understanding (2024)0.00
• Efficient Selective Audio Masked Multimodal Bottleneck Transformer For Audio-video Classification (2024)0.00
• Meta-transformer: A Unified Framework For Multimodal Learning (2023)6.44
• Role Of Audio In Audio-visual Video Summarization (2022)0.00
• Multimodal Transformer Networks For End-to-end Video-grounded Dialogue Systems (2019)17.12
• Maivar-t: Multimodal Audio-image And Video Action Recognizer Using Transformers (2023)0.00