Bytes Are All You Need: End-to-end Multilingual Speech Recognition And Synthesis With Bytes

Abstract

We present two end-to-end models: Audio-to-Byte (A2B) and Byte-to-Audio (B2A), for multilingual speech recognition and synthesis. Prior work has predominantly used characters, sub-words or words as the unit of choice to model text. These units are difficult to scale to languages with large vocabularies, particularly in the case of multilingual processing. In this work, we model text via a sequence of Unicode bytes, specifically, the UTF-8 variable length byte sequence for each character. Bytes allow us to avoid large softmaxes in languages with large vocabularies, and share representations in multilingual models. We show that bytes are superior to grapheme characters over a wide variety of languages in monolingual end-to-end speech recognition. Additionally, our multilingual byte model outperform each respective single language baseline on average by 4.4% relatively. In Japanese-English code-switching speech, our multilingual byte model outperform our monolingual baseline by 38.6% rela

Stats

• citations76
• S2 citations—
• github stars0
• HF likes0
• heat score14.15
• arxiv keyli2018bytes

Related papers

• Multilingual Byte2speech Models For Scalable Low-resource Speech Synthesis (2021)0.00
• An Investigation Of Phone-based Subword Units For End-to-end Speech Recognition (2020)9.59
• Massively Multilingual Adversarial Speech Recognition (2019)11.93
• Large-scale Multilingual Speech Recognition With A Streaming End-to-end Model (2019)14.97
• Hybrid Ctc-attention Based End-to-end Speech Recognition Using Subword Units (2018)10.85
• Multilingual Speech Recognition With A Single End-to-end Model (2017)16.05
• Allost: Low-resource Speech Translation Without Source Transcription (2021)7.81
• Phoneme Based Neural Transducer For Large Vocabulary Speech Recognition (2020)9.59