Whisperner: Unified Open Named Entity And Speech Recognition

Abstract

Integrating named entity recognition (NER) with automatic speech recognition (ASR) can significantly enhance transcription accuracy and informativeness. In this paper, we introduce WhisperNER, a novel model that allows joint speech transcription and entity recognition. WhisperNER supports open-type NER, enabling recognition of diverse and evolving entities at inference. Building on recent advancements in open NER research, we augment a large synthetic dataset with synthetic speech samples. This allows us to train WhisperNER on a large number of examples with diverse NER tags. During training, the model is prompted with NER labels and optimized to output the transcribed utterance along with the corresponding tagged entities. To evaluate WhisperNER, we generate synthetic speech for commonly used NER benchmarks and annotate existing ASR datasets with open NER tags. Our experiments demonstrate that WhisperNER outperforms natural baselines on both out-of-domain open type NER and supervised

Stats

• citations1
• S2 citations—
• github stars0
• HF likes0
• heat score2.26
• arxiv keyayache2024whisperner

Related papers

• A Multitask Training Approach To Enhance Whisper With Contextual Biasing And Open-vocabulary Keyword Spotting (2023)0.00
• "i've Heard Of You!": Generate Spoken Named Entity Recognition Data For Unseen Entities (2024)2.26
• End-to-end Named Entity Extraction From Speech (2018)0.00
• DEXTER: Deep Encoding Of External Knowledge For Named Entity Recognition In Virtual Assistants (2021)2.26
• Enriching Under-represented Named-entities To Improve Speech Recognition Performance (2020)0.00
• M2r-whisper: Multi-stage And Multi-scale Retrieval Augmentation For Enhancing Whisper (2024)6.77
• End-to-end Whisper To Natural Speech Conversion Using Modified Transformer Network (2020)0.00
• Retraining-free Customized ASR For Enharmonic Words Based On A Named-entity-aware Model And Phoneme Similarity Estimation (2023)4.52