Discrete Audio Representation As An Alternative To Mel-spectrograms For Speaker And Speech Recognition

Abstract

Discrete audio representation, aka audio tokenization, has seen renewed interest driven by its potential to facilitate the application of text language modeling approaches in audio domain. To this end, various compression and representation-learning based tokenization schemes have been proposed. However, there is limited investigation into the performance of compression-based audio tokens compared to well-established mel-spectrogram features across various speaker and speech related tasks. In this paper, we evaluate compression based audio tokens on three tasks: Speaker Verification, Diarization and (Multi-lingual) Speech Recognition. Our findings indicate that (i) the models trained on audio tokens perform competitively, on average within \(1%\) of mel-spectrogram features for all the tasks considered, and do not surpass them yet. (ii) these models exhibit robustness for out-of-domain narrowband data, particularly in speaker tasks. (iii) audio tokens allow for compression to 20x compa

Stats

• citations13
• S2 citations—
• github stars0
• HF likes0
• heat score8.60
• arxiv keypuvvada2023discrete

Related papers

• Dmel: Speech Tokenization Made Simple (2024)0.00
• DASB - Discrete Audio And Speech Benchmark (2024)0.00
• Wavtokenizer: An Efficient Acoustic Discrete Codec Tokenizer For Audio Language Modeling (2024)6.22
• Almtokenizer: A Low-bitrate And Semantic-rich Audio Codec Tokenizer For Audio Language Modeling (2025)0.00
• Dm-codec: Distilling Multimodal Representations For Speech Tokenization (2024)3.53
• Exploring Speech Recognition, Translation, And Understanding With Discrete Speech Units: A Comparative Study (2023)0.00
• Tokensplit: Using Discrete Speech Representations For Direct, Refined, And Transcript-conditioned Speech Separation And Recognition (2023)7.50
• Acoustic BPE For Speech Generation With Discrete Tokens (2023)6.77