OMCAT: Omni Context Aware Transformer

Abstract

Large Language Models (LLMs) have made significant strides in text generation and comprehension, with recent advancements extending into multimodal LLMs that integrate visual and audio inputs. However, these models continue to struggle with fine-grained, cross-modal temporal understanding, particularly when correlating events across audio and video streams. We address these challenges with two key contributions: a new dataset and model, called OCTAV and OMCAT respectively. OCTAV (Omni Context and Temporal Audio Video) is a novel dataset designed to capture event transitions across audio and video. Second, OMCAT (Omni Context Aware Transformer) is a powerful model that leverages RoTE (Rotary Time Embeddings), an innovative extension of RoPE, to enhance temporal grounding and computational efficiency in time-anchored tasks. Through a robust three-stage training pipeline-feature alignment, instruction tuning, and OCTAV-specific training-OMCAT excels in cross-modal temporal understanding.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keygoel2024omcat

Related papers

• Capybara-omni: An Efficient Paradigm For Building Omni-modal Language Models (2025)0.00
• Omni-captioner: Data Pipeline, Models, And Benchmark For Omni Detailed Perception (2025)0.00
• M2-omni: Advancing Omni-mllm For Comprehensive Modality Support With Competitive Performance (2025)0.00
• Omni-avsr: Towards Unified Multimodal Speech Recognition With Large Language Models (2025)2.26
• VALOR: Vision-audio-language Omni-perception Pretraining Model And Dataset (2023)10.61
• CTAL: Pre-training Cross-modal Transformer For Audio-and-language Representations (2021)7.50
• VAST: A Vision-audio-subtitle-text Omni-modality Foundation Model And Dataset (2023)14.55
• Mixture-of-transformers: A Sparse And Scalable Architecture For Multi-modal Foundation Models (2024)0.00