State-of-the-art Speech Recognition With Sequence-to-sequence Models
2017 · Chung-Cheng Chiu, Tara N. Sainath, Yonghui Wu, et al.
Abstract
Attention-based encoder-decoder architectures such as Listen, Attend, and Spell (LAS), subsume the acoustic, pronunciation and language model components of a traditional automatic speech recognition (ASR) system into a single neural network. In previous work, we have shown that such architectures are comparable to state-of-theart ASR systems on dictation tasks, but it was not clear if such architectures would be practical for more challenging tasks such as voice search. In this work, we explore a variety of structural and optimization improvements to our LAS model which significantly improve performance. On the structural side, we show that word piece models can be used instead of graphemes. We also introduce a multi-head attention architecture, which offers improvements over the commonly-used single-head attention. On the optimization side, we explore synchronous training, scheduled sampling, label smoothing, and minimum word error rate optimization, which are all shown to improve acc
Authors
(none)
Tags
Stats
Related papers
- An Online Attention-based Model For Speech Recognition (2018)9.59
- Attention-based Sequence-to-sequence Model For Speech Recognition: Development Of State-of-the-art System On Librispeech And Its Application To Non-native English (2018)0.00
- Multi-dialect Speech Recognition With A Single Sequence-to-sequence Model (2017)13.79
- Listen Attentively, And Spell Once: Whole Sentence Generation Via A Non-autoregressive Architecture For Low-latency Speech Recognition (2020)10.07
- Towards Better Decoding And Language Model Integration In Sequence To Sequence Models (2016)15.67
- Audio-attention Discriminative Language Model For ASR Rescoring (2019)9.23
- Single Headed Attention Based Sequence-to-sequence Model For State-of-the-art Results On Switchboard (2020)0.00
- High Performance Sequence-to-sequence Model For Streaming Speech Recognition (2020)3.58