Future Word Contexts In Neural Network Language Models

Abstract

Recently, bidirectional recurrent network language models (bi-RNNLMs) have been shown to outperform standard, unidirectional, recurrent neural network language models (uni-RNNLMs) on a range of speech recognition tasks. This indicates that future word context information beyond the word history can be useful. However, bi-RNNLMs pose a number of challenges as they make use of the complete previous and future word context information. This impacts both training efficiency and their use within a lattice rescoring framework. In this paper these issues are addressed by proposing a novel neural network structure, succeeding word RNNLMs (su-RNNLMs). Instead of using a recurrent unit to capture the complete future word contexts, a feedforward unit is used to model a finite number of succeeding, future, words. This model can be trained much more efficiently than bi-RNNLMs and can also be used for lattice rescoring. Experimental results on a meeting transcription task (AMI) show the proposed mod

Stats

• citations12
• S2 citations—
• github stars0
• HF likes0
• heat score8.35
• arxiv keychen2017future

Related papers

• Gated Recurrent Unit Based Acoustic Modeling With Future Context (2018)7.16
• Context-aware RNNLM Rescoring For Conversational Speech Recognition (2020)4.52
• Lattice Rescoring Strategies For Long Short Term Memory Language Models In Speech Recognition (2017)9.76
• End-to-end Speech Recognition With Word-based RNN Language Models (2018)0.00
• Towards ASR Robust Spoken Language Understanding Through In-context Learning With Word Confusion Networks (2024)0.00
• Improved Neural Language Model Fusion For Streaming Recurrent Neural Network Transducer (2020)8.82
• End-to-end Speech Recognition Contextualization With Large Language Models (2023)0.00
• Towards Language Modelling In The Speech Domain Using Sub-word Linguistic Units (2021)0.00