Data-centric Improvements For Enhancing Multi-modal Understanding In Spoken Conversation Modeling

Abstract

Conversational assistants are increasingly popular across diverse real-world applications, highlighting the need for advanced multimodal speech modeling. Speech, as a natural mode of communication, encodes rich user-specific characteristics such as speaking rate and pitch, making it critical for effective interaction. Our work introduces a data-centric customization approach for efficiently enhancing multimodal understanding in conversational speech modeling. Central to our contributions is a novel multi-task learning paradigm that involves designing auxiliary tasks to utilize a small amount of speech data. Our approach achieves state-of-the-art performance on the Spoken-SQuAD benchmark, using only 10% of the training data with open-weight models, establishing a robust and efficient framework for audio-centric conversational modeling. We also introduce ASK-QA, the first dataset for multi-turn spoken dialogue with ambiguous user requests and dynamic evaluation inputs. Code and data fort

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keychen2024data

Related papers

• Improving Multimodal Speech Recognition By Data Augmentation And Speech Representations (2022)9.03
• Attentive Contextual Carryover For Multi-turn End-to-end Spoken Language Understanding (2021)7.16
• Resource-efficient Adaptation Of Speech Foundation Models For Multi-speaker ASR (2024)3.58
• S3: A Simple Strong Sample-effective Multimodal Dialog System (2024)0.00
• An Adversarial Learning Based Multi-step Spoken Language Understanding System Through Human-computer Interaction (2021)0.00
• Audiosetmix: Enhancing Audio-language Datasets With Llm-assisted Augmentations (2024)0.00
• A Multimodal Approach To Device-directed Speech Detection With Large Language Models (2024)7.16
• Speechcaps: Advancing Instruction-based Universal Speech Models With Multi-talker Speaking Style Captioning (2024)2.86