Capybara-omni: An Efficient Paradigm For Building Omni-modal Language Models

Abstract

With the development of Multimodal Large Language Models (MLLMs), numerous outstanding accomplishments have emerged within the open-source community. Due to the complexity of creating and training multimodal data pairs, it is still a computational and time-consuming process to build powerful MLLMs. In this work, we introduce Capybara-OMNI, an MLLM that trains in a lightweight and efficient manner and supports understanding text, image, video, and audio modalities. We present in detail the framework design, the data construction, and the training recipe, to develop an MLLM step-by-step to obtain competitive performance. We also provide exclusive benchmarks utilized in our experiments to show how to properly verify understanding capabilities across different modalities. Results show that by following our guidance, we can efficiently build an MLLM that achieves competitive performance among models of the same scale on various multimodal benchmarks. Additionally, to enhance the multimodal

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyji2025capybara

Related papers

• M2-omni: Advancing Omni-mllm For Comprehensive Modality Support With Competitive Performance (2025)0.00
• Macaw-llm: Multi-modal Language Modeling With Image, Audio, Video, And Text Integration (2023)0.00
• Omni-captioner: Data Pipeline, Models, And Benchmark For Omni Detailed Perception (2025)0.00
• Omni-avsr: Towards Unified Multimodal Speech Recognition With Large Language Models (2025)2.26
• Multimodal Large Language Models: A Survey (2023)0.00
• Omhbench: Benchmarking Balanced And Grounded Omni-modal Multi-hop Reasoning (2026)0.00
• OMCAT: Omni Context Aware Transformer (2024)0.00
• Omni-c: Compressing Heterogeneous Modalities Into A Single Dense Encoder (2026)0.00