Semi-tied Units For Efficient Gating In LSTM And Highway Networks

Abstract

Gating is a key technique used for integrating information from multiple sources by long short-term memory (LSTM) models and has recently also been applied to other models such as the highway network. Although gating is powerful, it is rather expensive in terms of both computation and storage as each gating unit uses a separate full weight matrix. This issue can be severe since several gates can be used together in e.g. an LSTM cell. This paper proposes a semi-tied unit (STU) approach to solve this efficiency issue, which uses one shared weight matrix to replace those in all the units in the same layer. The approach is termed "semi-tied" since extra parameters are used to separately scale each of the shared output values. These extra scaling factors are associated with the network activation functions and result in the use of parameterised sigmoid, hyperbolic tangent, and rectified linear unit functions. Speech recognition experiments using British English multi-genre broadcast data sh

Stats

• citations2
• S2 citations—
• github stars0
• HF likes0
• heat score3.58
• arxiv keyzhang2018semi

Related papers

• Improving Speech Recognition By Revising Gated Recurrent Units (2017)11.19
• Memory Visualization For Gated Recurrent Neural Networks In Speech Recognition (2016)11.76
• Light Gated Recurrent Units For Speech Recognition (2018)18.90
• Investigating Gated Recurrent Neural Networks For Speech Synthesis (2016)0.00
• Language Modeling With Highway LSTM (2017)10.21
• Stabilising And Accelerating Light Gated Recurrent Units For Automatic Speech Recognition (2023)0.00
• From Nodes To Networks: Evolving Recurrent Neural Networks (2018)0.00
• Dynamic Gated Recurrent Neural Network For Compute-efficient Speech Enhancement (2024)8.35