Macaw-llm: Multi-modal Language Modeling With Image, Audio, Video, And Text Integration

Abstract

Although instruction-tuned large language models (LLMs) have exhibited remarkable capabilities across various NLP tasks, their effectiveness on other data modalities beyond text has not been fully studied. In this work, we propose Macaw-LLM, a novel multi-modal LLM that seamlessly integrates visual, audio, and textual information. Macaw-LLM consists of three main components: a modality module for encoding multi-modal data, a cognitive module for harnessing pretrained LLMs, and an alignment module for harmonizing diverse representations. Our novel alignment module seamlessly bridges multi-modal features to textual features, simplifying the adaptation process from the modality modules to the cognitive module. In addition, we construct a large-scale multi-modal instruction dataset in terms of multi-turn dialogue, including 69K image instances and 50K video instances. We have made our data, code and model publicly available, which we hope can pave the way for future research in multi-modal

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keylyu2023macaw

Related papers

• A Review Of Multi-modal Large Language And Vision Models (2024)0.00
• Capybara-omni: An Efficient Paradigm For Building Omni-modal Language Models (2025)0.00
• Large Language Models Are Strong Audio-visual Speech Recognition Learners (2024)9.59
• Mmmmodal -- Multi-images Multi-audio Multi-turn Multi-modal (2024)0.00
• Omni-avsr: Towards Unified Multimodal Speech Recognition With Large Language Models (2025)2.26
• C3LLM: Conditional Multimodal Content Generation Using Large Language Models (2024)0.00
• Multimodal Large Language Models For End-to-end Affective Computing: Benchmarking And Boosting With Generative Knowledge Prompting (2025)0.00
• Uniaudio 1.5: Large Language Model-driven Audio Codec Is A Few-shot Audio Task Learner (2024)0.00