Improving Speech Inversion Through Self-supervised Embeddings And Enhanced Tract Variables

Abstract

The performance of deep learning models depends significantly on their capacity to encode input features efficiently and decode them into meaningful outputs. Better input and output representation has the potential to boost models' performance and generalization. In the context of acoustic-to-articulatory speech inversion (SI) systems, we study the impact of utilizing speech representations acquired via self-supervised learning (SSL) models, such as HuBERT compared to conventional acoustic features. Additionally, we investigate the incorporation of novel tract variables (TVs) through an improved geometric transformation model. By combining these two approaches, we improve the Pearson product-moment correlation (PPMC) scores which evaluate the accuracy of TV estimation of the SI system from 0.7452 to 0.8141, a 6.9% increase. Our findings underscore the profound influence of rich feature representations from SSL models and improved geometric transformations with target TVs on the enhance

Stats

• citations4
• S2 citations—
• github stars0
• HF likes0
• heat score5.24
• arxiv keyattia2023improving

Related papers

• Self-supervised Models Of Speech Infer Universal Articulatory Kinematics (2023)0.00
• Evidence Of Vocal Tract Articulation In Self-supervised Learning Of Speech (2022)9.41
• Efficient Infusion Of Self-supervised Representations In Automatic Speech Recognition (2024)0.00
• Unispeech-sat: Universal Speech Representation Learning With Speaker Aware Pre-training (2021)0.00
• Pushing The Limits Of Unsupervised Unit Discovery For SSL Speech Representation (2023)6.34
• Investigation Of Ensemble Features Of Self-supervised Pretrained Models For Automatic Speech Recognition (2022)9.41
• Contentvec: An Improved Self-supervised Speech Representation By Disentangling Speakers (2022)0.00
• Automatic Pronunciation Assessment Using Self-supervised Speech Representation Learning (2022)0.00