Rate-adaptive Coding Mechanism For Semantic Communications With Multi-modal Data

Abstract

Recently, the ever-increasing demand for bandwidth in multi-modal communication systems requires a paradigm shift. Powered by deep learning, semantic communications are applied to multi-modal scenarios to boost communication efficiency and save communication resources. However, the existing end-to-end neural network (NN) based framework without the channel encoder/decoder is incompatible with modern digital communication systems. Moreover, most end-to-end designs are task-specific and require re-design and re-training for new tasks, which limits their applications. In this paper, we propose a distributed multi-modal semantic communication framework incorporating the conventional channel encoder/decoder. We adopt NN-based semantic encoder and decoder to extract correlated semantic information contained in different modalities, including speech, text, and image. Based on the proposed framework, we further establish a general rate-adaptive coding mechanism for various types of multi-modal

Stats

• citations38
• S2 citations—
• github stars0
• HF likes0
• heat score11.93
• arxiv keyhe2023rate

Related papers

• Semantic Communications For Speech Recognition (2021)11.93
• Optimizing Neural Speech Codec For Low-bitrate Compression Via Multi-scale Encoding (2024)0.00
• Semantic Communications For Speech Signals (2020)14.35
• Deep Learning Enabled Semantic Communications With Speech Recognition And Synthesis (2022)17.85
• Spatialcodec: Neural Spatial Speech Coding (2023)3.69
• Neural Speech And Audio Coding: Modern AI Technology Meets Traditional Codecs (2024)7.16
• Task And Perception-aware Distributed Source Coding For Correlated Speech Under Bandwidth-constrained Channels (2025)0.00
• Cascaded Cross-module Residual Learning Towards Lightweight End-to-end Speech Coding (2019)11.49