Frame-wise Streaming End-to-end Speaker Diarization With Non-autoregressive Self-attention-based Attractors

Abstract

This work proposes a frame-wise online/streaming end-to-end neural diarization (FS-EEND) method in a frame-in-frame-out fashion. To frame-wisely detect a flexible number of speakers and extract/update their corresponding attractors, we propose to leverage a causal speaker embedding encoder and an online non-autoregressive self-attention-based attractor decoder. A look-ahead mechanism is adopted to allow leveraging some future frames for effectively detecting new speakers in real time and adaptively updating speaker attractors. The proposed method processes the audio stream frame by frame, and has a low inference latency caused by the look-ahead frames. Experiments show that, compared with the recently proposed block-wise online methods, our method FS-EEND achieves state-of-the-art diarization results, with a low inference latency and computational cost.

Stats

• citations1
• S2 citations—
• github stars0
• HF likes0
• heat score2.26
• arxiv keyliang2023frame

Related papers

• LS-EEND: Long-form Streaming End-to-end Neural Diarization With Online Attractor Extraction (2024)3.58
• Encoder-decoder Based Attractors For End-to-end Neural Diarization (2021)13.05
• Online End-to-end Neural Diarization With Speaker-tracing Buffer (2020)10.74
• Online Streaming End-to-end Neural Diarization Handling Overlapping Speech And Flexible Numbers Of Speakers (2021)0.00
• Speech-aware Neural Diarization With Encoder-decoder Attractor Guided By Attention Constraints (2024)0.00
• BW-EDA-EEND: Streaming End-to-end Neural Speaker Diarization For A Variable Number Of Speakers (2020)10.74
• Online Neural Diarization Of Unlimited Numbers Of Speakers Using Global And Local Attractors (2022)10.07
• Speakers Unembedded: Embedding-free Approach To Long-form Neural Diarization (2024)3.58