Tokensplit: Using Discrete Speech Representations For Direct, Refined, And Transcript-conditioned Speech Separation And Recognition
2023 · Hakan Erdogan, Scott Wisdom, Xuankai Chang, et al.
Abstract
We present TokenSplit, a speech separation model that acts on discrete token sequences. The model is trained on multiple tasks simultaneously: separate and transcribe each speech source, and generate speech from text. The model operates on transcripts and audio token sequences and achieves multiple tasks through masking of inputs. The model is a sequence-to-sequence encoder-decoder model that uses the Transformer architecture. We also present a "refinement" version of the model that predicts enhanced audio tokens from the audio tokens of speech separated by a conventional separation model. Using both objective metrics and subjective MUSHRA listening tests, we show that our model achieves excellent performance in terms of separation, both with or without transcript conditioning. We also measure the automatic speech recognition (ASR) performance and provide audio samples of speech synthesis to demonstrate the additional utility of our model.
Authors
(none)
Tags
Stats
Related papers
- Speechsplit 2.0: Unsupervised Speech Disentanglement For Voice Conversion Without Tuning Autoencoder Bottlenecks (2022)4.27
- Transmask: A Compact And Fast Speech Separation Model Based On Transformer (2021)8.82
- Directed Speech Separation For Automatic Speech Recognition Of Long Form Conversational Speech (2021)2.26
- Multi-dimensional And Multi-scale Modeling For Speech Separation Optimized By Discriminative Learning (2023)0.00
- Discrete Audio Representation As An Alternative To Mel-spectrograms For Speaker And Speech Recognition (2023)8.60
- Monaural Multi-speaker Speech Separation Using Efficient Transformer Model (2023)0.00
- Separator-transducer-segmenter: Streaming Recognition And Segmentation Of Multi-party Speech (2022)0.00
- Tokenverse: Towards Unifying Speech And NLP Tasks Via Transducer-based ASR (2024)1.40