Mmmmodal -- Multi-images Multi-audio Multi-turn Multi-modal

Abstract

Our contribution introduces a groundbreaking multimodal large language model designed to comprehend multi-images, multi-audio, and multi-images-multi-audio within a single multiturn session. Leveraging state-of-the-art models, we utilize the SigLIP encoder for visual inputs and the Whisper Encoder for audio inputs. Notably, this multimodal large language model is bilingual, proficient in understanding both English and Malay simultaneously. We proudly unveil two versions of this model: TinyLlama with 1.1B parameters, and Mistral with 7B parameters. With its ability to navigate diverse modalities and languages, our model represents a significant advancement for the Malaysian context and beyond. All models released at https://huggingface.co/collections/mesolitica/multimodal-malaysian-llm-65c6f893e03f78fa9e5c8859

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyzolkepli2024mmmmodal

Related papers

• Towards Multi-modal Mastery: A 4.5B Parameter Truly Multi-modal Small Language Model (2024)2.26
• Macaw-llm: Multi-modal Language Modeling With Image, Audio, Video, And Text Integration (2023)0.00
• Meralion-audiollm: Bridging Audio And Language With Large Language Models (2024)0.00
• A Review Of Multi-modal Large Language And Vision Models (2024)0.00
• Discrete Multimodal Transformers With A Pretrained Large Language Model For Mixed-supervision Speech Processing (2024)0.00
• Large Language Models Are Strong Audio-visual Speech Recognition Learners (2024)9.59
• Multimodal Large Language Models: A Survey (2023)0.00
• X-LLM: Bootstrapping Advanced Large Language Models By Treating Multi-modalities As Foreign Languages (2023)0.00