Decoder-only Architecture For Speech Recognition With CTC Prompts And Text Data Augmentation

Abstract

Collecting audio-text pairs is expensive; however, it is much easier to access text-only data. Unless using shallow fusion, end-to-end automatic speech recognition (ASR) models require architecture modifications or additional training schemes to use text-only data. Inspired by recent advances in decoder-only language models (LMs), such as GPT-3 and PaLM adopted for speech-processing tasks, we propose using a decoder-only architecture for ASR with simple text augmentation. To provide audio information, encoder features compressed by CTC prediction are used as prompts for the decoder, which can be regarded as refining CTC prediction using the decoder-only model. Because the decoder architecture is the same as an autoregressive LM, it is simple to enhance the model by leveraging external text data with LM training. An experimental comparison using LibriSpeech and Switchboard shows that our proposed models with text augmentation training reduced word error rates from ordinary CTC by 0.3% a

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keytsunoo2023decoder

Related papers

• CJST: CTC Compressor Based Joint Speech And Text Training For Decoder-only ASR (2024)3.58
• Linguistic-enhanced Transformer With CTC Embedding For Speech Recognition (2022)2.26
• On Decoder-only Architecture For Speech-to-text And Large Language Model Integration (2023)0.00
• An Improved Hybrid Ctc-attention Model For Speech Recognition (2018)0.00
• You Do Not Need More Data: Improving End-to-end Speech Recognition By Text-to-speech Data Augmentation (2020)11.49
• Pre-training Transformer Decoder For End-to-end ASR Model With Unpaired Speech Data (2022)13.47
• Multitask Training With Text Data For End-to-end Speech Recognition (2020)7.50
• Back-translation-style Data Augmentation For End-to-end ASR (2018)13.11