TIPAA-SSL: Text Independent Phone-to-audio Alignment Based On Self-supervised Learning And Knowledge Transfer

Abstract

In this paper, we present a novel approach for text independent phone-to-audio alignment based on phoneme recognition, representation learning and knowledge transfer. Our method leverages a self-supervised model (wav2vec2) fine-tuned for phoneme recognition using a Connectionist Temporal Classification (CTC) loss, a dimension reduction model and a frame-level phoneme classifier trained thanks to forced-alignment labels (using Montreal Forced Aligner) to produce multi-lingual phonetic representations, thus requiring minimal additional training. We evaluate our model using synthetic native data from the TIMIT dataset and the SCRIBE dataset for American and British English, respectively. Our proposed model outperforms the state-of-the-art (charsiu) in statistical metrics and has applications in language learning and speech processing systems. We leave experiments on other languages for future work but the design of the system makes it easily adaptable to other languages.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keytits2024tipaa

Related papers

• Phone-to-audio Alignment Without Text: A Semi-supervised Approach (2021)16.74
• Vae-based Phoneme Alignment Using Gradient Annealing And SSL Acoustic Features (2024)0.00
• An Adapter Based Pre-training For Efficient And Scalable Self-supervised Speech Representation Learning (2021)8.35
• Automatic Pronunciation Assessment Using Self-supervised Speech Representation Learning (2022)0.00
• Iterative Pseudo-forced Alignment By Acoustic CTC Loss For Self-supervised ASR Domain Adaptation (2022)0.00
• Speaker- And Text-independent Estimation Of Articulatory Movements And Phoneme Alignments From Speech (2024)2.26
• Joint Masked CPC And CTC Training For ASR (2020)8.60
• Lyrics-to-audio Alignment By Unsupervised Discovery Of Repetitive Patterns In Vowel Acoustics (2017)6.34