Llava-mole: Sparse Mixture Of Lora Experts For Mitigating Data Conflicts In Instruction Finetuning Mllms

Abstract

Instruction finetuning on a variety of image-text instruction data is the key to obtaining a versatile Multimodal Large Language Model (MLLM), and different configurations of the instruction data can lead to finetuned models with different capabilities. However, we have discovered that data conflicts are inevitable when mixing instruction data from distinct domains, which can result in performance