Abstract
Automatic lipreading (ALR) systems decode speech by analyzing lip movements in the absence of audio, which is particularly critical in noisy environments. These systems have undergone remarkable improvements by utilizing deep neural networks and large-scale databases, enabling progression from recognizing simple speech units to interpreting complete sentences. However, only a few studies have considered the vulnerability of DNN-trained lipreading systems to adversarial attacks, particularly in white-box settings where attackers have full access to the model. This paper presents an adversarial attack framework for DNN-based lipreading models that addresses the limitations of existing approaches. Firstly, we present a gradient-based attack framework for the white-box setting. The fundamental idea of this framework explicitly targets the sequence-to-sequence alignment mechanism to enable more effective attacks on variable-length outputs by introducing lip-specific spatial masking, which restricts perturbations to semantically relevant regions (the lips) rather than applying uniformly across the entire frame. As a result, it ensures temporal consistency, enhances imperceptibility, and computational efficiency. Furthermore, in black-box settings, where the model's internals are inaccessible, we propose a gradient-free optimization technique using genetic algorithms that enables the generation of adversarial examples. Our experimental results show that both our methods achieve 100% success with lip-masked perturbations, while reducing the attack space by 90%, resulting in efficient and imperceptible adversarial examples.