A Large-scale Probing Analysis Of Speaker-specific Attributes In Self-supervised Speech Representations

Abstract

Enhancing explainability in speech self-supervised learning (SSL) is important for developing reliable SSL-based speech processing systems. This study probes how speech SSL models encode speaker-specific information via a large-scale probing analysis of 11 models, decomposing identity into acoustic, prosodic, and paralinguistic attributes. The results confirm a general hierarchy wherein initial layers encode fundamental acoustics and middle layers synthesise abstract traits. Crucially, the consensus that final layers purely abstract linguistic content is challenged. It is discovered that larger models unexpectedly recover speaker identity in their deep layers. Furthermore, the intermediate representations of speech SSL models are found to capture dynamic prosody better than specialised speaker embeddings. These insights decode the complex internal mechanics of SSL models, providing guidelines for selecting interpretable and task-optimal representations.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keychiu2025a

Related papers

• What Do Self-supervised Speech And Speaker Models Learn? New Findings From A Cross Model Layer-wise Analysis (2024)8.09
• Speech Self-supervised Representations Benchmarking: A Case For Larger Probing Heads (2023)2.26
• Evidence Of Vocal Tract Articulation In Self-supervised Learning Of Speech (2022)9.41
• Speech Representation Analysis Based On Inter- And Intra-model Similarities (2024)2.26
• Analyzing The Factors Affecting Usefulness Of Self-supervised Pre-trained Representations For Speech Recognition (2022)0.00
• An Empirical Analysis Of Speech Self-supervised Learning At Multiple Resolutions (2024)0.00
• Understanding Self-supervised Learning Of Speech Representation Via Invariance And Redundancy Reduction (2023)0.00
• Unispeech-sat: Universal Speech Representation Learning With Speaker Aware Pre-training (2021)0.00