Disentangling Textual And Acoustic Features Of Neural Speech Representations

Abstract

Neural speech models build deeply entangled internal representations, which capture a variety of features (e.g., fundamental frequency, loudness, syntactic category, or semantic content of a word) in a distributed encoding. This complexity makes it difficult to track the extent to which such representations rely on textual and acoustic information, or to suppress the encoding of acoustic features that may pose privacy risks (e.g., gender or speaker identity) in critical, real-world applications. In this paper, we build upon the Information Bottleneck principle to propose a disentanglement framework that separates complex speech representations into two distinct components: one encoding content (i.e., what can be transcribed as text) and the other encoding acoustic features relevant to a given downstream task. We apply and evaluate our framework to emotion recognition and speaker identification downstream tasks, quantifying the contribution of textual and acoustic features at each model

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keymohebbi2024disentangling

Related papers

• Disentangling Speech And Non-speech Components For Building Robust Acoustic Models From Found Data (2019)0.00
• Speaker Disentanglement Of Speech Pre-trained Model Based On Interpretability (2025)0.00
• Disentangling Prosody Representations With Unsupervised Speech Reconstruction (2022)0.00
• Disentangled Feature Learning For Real-time Neural Speech Coding (2022)0.00
• Disentangling Voice And Content With Self-supervision For Speaker Recognition (2023)2.26
• Contentvec: An Improved Self-supervised Speech Representation By Disentangling Speakers (2022)0.00
• Parsing Speech: A Neural Approach To Integrating Lexical And Acoustic-prosodic Information (2017)8.60
• Intra-class Variation Reduction Of Speaker Representation In Disentanglement Framework (2020)8.35