Streaming Parrotron For On-device Speech-to-speech Conversion

Abstract

We present a fully on-device streaming Speech2Speech conversion model that normalizes a given input speech directly to synthesized output speech. Deploying such a model on mobile devices pose significant challenges in terms of memory footprint and computation requirements. We present a streaming-based approach to produce an acceptable delay, with minimal loss in speech conversion quality, when compared to a reference state of the art non-streaming approach. Our method consists of first streaming the encoder in real time while the speaker is speaking. Then, as soon as the speaker stops speaking, we run the spectrogram decoder in streaming mode along the side of a streaming vocoder to generate output speech. To achieve an acceptable delay-quality trade-off, we propose a novel hybrid approach for look-ahead in the encoder which combines a look-ahead feature stacker with a look-ahead self-attention. We show that our streaming approach is almost 2x faster than real time on the Pixel4 CPU.

Stats

• citations1
• S2 citations—
• github stars0
• HF likes0
• heat score2.26
• arxiv keyrybakov2022streaming

Related papers

• Parallel Rescoring With Transformer For Streaming On-device Speech Recognition (2020)7.50
• Dualvc 3: Leveraging Language Model Generated Pseudo Context For End-to-end Low Latency Streaming Voice Conversion (2024)0.00
• Streamspeech: Simultaneous Speech-to-speech Translation With Multi-task Learning (2024)7.81
• High Quality Streaming Speech Synthesis With Low, Sentence-length-independent Latency (2021)8.60
• Streamvoice: Streamable Context-aware Language Modeling For Real-time Zero-shot Voice Conversion (2024)7.16
• Mobilespeech: A Fast And High-fidelity Framework For Mobile Zero-shot Text-to-speech (2024)0.00
• Livespeech: Low-latency Zero-shot Text-to-speech Via Autoregressive Modeling Of Audio Discrete Codes (2024)5.84
• High Performance Sequence-to-sequence Model For Streaming Speech Recognition (2020)3.58