Modalitymirror: Improving Audio Classification In Modality Heterogeneity Federated Learning With Multimodal Distillation

Abstract

Multimodal Federated Learning frequently encounters challenges of client modality heterogeneity, leading to undesired performances for secondary modality in multimodal learning. It is particularly prevalent in audiovisual learning, with audio is often assumed to be the weaker modality in recognition tasks. To address this challenge, we introduce ModalityMirror to improve audio model performance by leveraging knowledge distillation from an audiovisual federated learning model. ModalityMirror involves two phases: a modality-wise FL stage to aggregate uni-modal encoders; and a federated knowledge distillation stage on multi-modality clients to train an unimodal student model. Our results demonstrate that ModalityMirror significantly improves the audio classification compared to the state-of-the-art FL methods such as Harmony, particularly in audiovisual FL facing video missing. Our approach unlocks the potential for exploiting the diverse modality spectrum inherent in multi-modal FL.

Stats

• citations1
• S2 citations—
• github stars0
• HF likes0
• heat score2.26
• arxiv keyfeng2024modalitymirror

Related papers

• Audio-visual Representation Learning Via Knowledge Distillation From Speech Foundation Models (2025)7.81
• Cross-modal Distillation For Widely Differing Modalities (2025)0.00
• Enhancing Multimodal Sentiment Analysis For Missing Modality Through Self-distillation And Unified Modality Cross-attention (2024)6.71
• Audio Representation Learning By Distilling Video As Privileged Information (2023)0.00
• Quantifying Multimodal Imbalance: A Gmm-guided Adaptive Loss For Audio-visual Learning (2025)0.00
• Hunyuanvideo-foley: Multimodal Diffusion With Representation Alignment For High-fidelity Foley Audio Generation (2025)0.00
• Enhancing Modal Fusion By Alignment And Label Matching For Multimodal Emotion Recognition (2024)6.34
• Self-supervised Audio-visual Speech Representations Learning By Multimodal Self-distillation (2022)0.00