Improving Speech Recognition By Revising Gated Recurrent Units

Abstract

Speech recognition is largely taking advantage of deep learning, showing that substantial benefits can be obtained by modern Recurrent Neural Networks (RNNs). The most popular RNNs are Long Short-Term Memory (LSTMs), which typically reach state-of-the-art performance in many tasks thanks to their ability to learn long-term dependencies and robustness to vanishing gradients. Nevertheless, LSTMs have a rather complex design with three multiplicative gates, that might impair their efficient implementation. An attempt to simplify LSTMs has recently led to Gated Recurrent Units (GRUs), which are based on just two multiplicative gates. This paper builds on these efforts by further revising GRUs and proposing a simplified architecture potentially more suitable for speech recognition. The contribution of this work is two-fold. First, we suggest to remove the reset gate in the GRU design, resulting in a more efficient single-gate architecture. Second, we propose to replace tanh with ReLU acti

Stats

• citations30
• S2 citations—
• github stars0
• HF likes0
• heat score11.19
• arxiv keyravanelli2017improving

Related papers

• Light Gated Recurrent Units For Speech Recognition (2018)18.90
• Memory Visualization For Gated Recurrent Neural Networks In Speech Recognition (2016)11.76
• Stabilising And Accelerating Light Gated Recurrent Units For Automatic Speech Recognition (2023)0.00
• Investigating Gated Recurrent Neural Networks For Speech Synthesis (2016)0.00
• Dynamic Gated Recurrent Neural Network For Compute-efficient Speech Enhancement (2024)8.35
• Semi-tied Units For Efficient Gating In LSTM And Highway Networks (2018)3.58
• A Comparison Of Adaptation Techniques And Recurrent Neural Network Architectures (2018)3.58
• Restricted Recurrent Neural Networks (2019)7.50