Neural2speech: A Transfer Learning Framework For Neural-driven Speech Reconstruction

Abstract

Reconstructing natural speech from neural activity is vital for enabling direct communication via brain-computer interfaces. Previous efforts have explored the conversion of neural recordings into speech using complex deep neural network (DNN) models trained on extensive neural recording data, which is resource-intensive under regular clinical constraints. However, achieving satisfactory performance in reconstructing speech from limited-scale neural recordings has been challenging, mainly due to the complexity of speech representations and the neural data constraints. To overcome these challenges, we propose a novel transfer learning framework for neural-driven speech reconstruction, called Neural2Speech, which consists of two distinct training phases. First, a speech autoencoder is pre-trained on readily available speech corpora to decode speech waveforms from the encoded speech representations. Second, a lightweight adaptor is trained on the small-scale neural recordings to align the

Stats

• citations4
• S2 citations—
• github stars0
• HF likes0
• heat score5.24
• arxiv keyli2023neural2speech

Related papers

• Utilizing Neural Transducers For Two-stage Text-to-speech Via Semantic Token Prediction (2024)0.00
• Deep Neural Networks For Automatic Speech Processing: A Survey From Large Corpora To Limited Data (2020)0.00
• Transfer Learning-based Deep Residual Learning For Speech Recognition In Clean And Noisy Environments (2025)3.58
• Improved Speech Reconstruction From Silent Video (2017)13.34
• Dualsep: A Light-weight Dual-encoder Convolutional Recurrent Network For Real-time In-car Speech Separation (2024)0.00
• Translatotron 2: High-quality Direct Speech-to-speech Translation With Voice Preservation (2021)0.00
• Almost Unsupervised Text To Speech And Automatic Speech Recognition (2019)0.00
• Vid2speech: Speech Reconstruction From Silent Video (2017)14.15