Ghostvec: A New Threat To Speaker Privacy Of End-to-end Speech Recognition System

Abstract

Speaker adaptation systems face privacy concerns, for such systems are trained on private datasets and often overfitting. This paper demonstrates that an attacker can extract speaker information by querying speaker-adapted speech recognition (ASR) systems. We focus on the speaker information of a transformer-based ASR and propose GhostVec, a simple and efficient attack method to extract the speaker information from an encoder-decoder-based ASR system without any external speaker verification system or natural human voice as a reference. To make our results quantitative, we pre-process GhostVec using singular value decomposition (SVD) and synthesize it into waveform. Experiment results show that the synthesized audio of GhostVec reaches 10.83% EER and 0.47 minDCF with target speakers, which suggests the effectiveness of the proposed method. We hope the preliminary discovery in this study to catalyze future speech recognition research on privacy-preserving topics.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keychen2023ghostvec

Related papers

• You Are What You Say: Exploiting Linguistic Content For Voiceprivacy Attacks (2025)3.58
• Speaker De-identification System Using Autoencoders And Adversarial Training (2020)0.00
• Inaudible Adversarial Perturbations For Targeted Attack In Speaker Recognition (2020)12.33
• Hiddenspeaker: Generate Imperceptible Unlearnable Audios For Speaker Verification System (2024)2.26
• Adversarial Speech For Voice Privacy Protection From Personalized Speech Generation (2024)8.09
• Impact Of Phonetics On Speaker Identity In Adversarial Voice Attack (2025)0.00
• Adversarial Sample Detection For Speaker Verification By Neural Vocoders (2021)0.00
• Reprogramming Self-supervised Learning-based Speech Representations For Speaker Anonymization (2023)2.26