Spatialcodec: Neural Spatial Speech Coding

Abstract

In this work, we address the challenge of encoding speech captured by a microphone array using deep learning techniques with the aim of preserving and accurately reconstructing crucial spatial cues embedded in multi-channel recordings. We propose a neural spatial audio coding framework that achieves a high compression ratio, leveraging single-channel neural sub-band codec and SpatialCodec. Our approach encompasses two phases: (i) a neural sub-band codec is designed to encode the reference channel with low bit rates, and (ii), a SpatialCodec captures relative spatial information for accurate multi-channel reconstruction at the decoder end. In addition, we also propose novel evaluation metrics to assess the spatial cue preservation: (i) spatial similarity, which calculates cosine similarity on a spatially intuitive beamspace, and (ii), beamformed audio quality. Our system shows superior spatial performance compared with high bitrate baselines and black-box neural architecture. Demos are

Stats

• citations0
• S2 citations—
• github stars69
• HF likes0
• heat score3.69
• arxiv keyxu2023spatialcodec

Related papers

• Optimizing Neural Speech Codec For Low-bitrate Compression Via Multi-scale Encoding (2024)0.00
• Spatialnet: Extensively Learning Spatial Information For Multichannel Joint Speech Separation, Denoising And Dereverberation (2023)13.88
• A Neural Speech Codec For Noise Robust Speech Coding (2023)0.00
• Pscodec: A Series Of High-fidelity Low-bitrate Neural Speech Codecs Leveraging Prompt Encoders (2024)0.00
• Latent-domain Predictive Neural Speech Coding (2022)12.15
• Stftcodec: High-fidelity Audio Compression Through Time-frequency Domain Representation (2025)2.26
• Freecodec: A Disentangled Neural Speech Codec With Fewer Tokens (2024)4.52
• Neural Speech And Audio Coding: Modern AI Technology Meets Traditional Codecs (2024)7.16