Learning Compact Recurrent Neural Networks

Abstract

Recurrent neural networks (RNNs), including long short-term memory (LSTM) RNNs, have produced state-of-the-art results on a variety of speech recognition tasks. However, these models are often too large in size for deployment on mobile devices with memory and latency constraints. In this work, we study mechanisms for learning compact RNNs and LSTMs via low-rank factorizations and parameter sharing schemes. Our goal is to investigate redundancies in recurrent architectures where compression can be admitted without losing performance. A hybrid strategy of using structured matrices in the bottom layers and shared low-rank factors on the top layers is found to be particularly effective, reducing the parameters of a standard LSTM by 75%, at a small cost of 0.3% increase in WER, on a 2,000-hr English Voice Search task.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keylu2016learning

Related papers

• Restricted Recurrent Neural Networks (2019)7.50
• Long Short-term Memory Based Convolutional Recurrent Neural Networks For Large Vocabulary Speech Recognition (2016)6.77
• High Order Recurrent Neural Networks For Acoustic Modelling (2018)8.60
• Memory Visualization For Gated Recurrent Neural Networks In Speech Recognition (2016)11.76
• Lightweight And Efficient End-to-end Speech Recognition Using Low-rank Transformer (2019)0.00
• Lattice Rescoring Strategies For Long Short Term Memory Language Models In Speech Recognition (2017)9.76
• Bidirectional Quaternion Long-short Term Memory Recurrent Neural Networks For Speech Recognition (2018)9.41
• Improving RNN Transducer Modeling For End-to-end Speech Recognition (2019)0.00