Multiformer: A Head-configurable Transformer-based Model For Direct Speech Translation

Abstract

Transformer-based models have been achieving state-of-the-art results in several fields of Natural Language Processing. However, its direct application to speech tasks is not trivial. The nature of this sequences carries problems such as long sequence lengths and redundancy between adjacent tokens. Therefore, we believe that regular self-attention mechanism might not be well suited for it. Different approaches have been proposed to overcome these problems, such as the use of efficient attention mechanisms. However, the use of these methods usually comes with a cost, which is a performance reduction caused by information loss. In this study, we present the Multiformer, a Transformer-based model which allows the use of different attention mechanisms on each head. By doing this, the model is able to bias the self-attention towards the extraction of more diverse token interactions, and the information loss is reduced. Finally, we perform an analysis of the head contributions, and we obse

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keysant2022multiformer

Related papers

• Speechformer: Reducing Information Loss In Direct Speech Translation (2021)7.16
• Exploring Self-attention Mechanisms For Speech Separation (2022)12.54
• Streaming Transformer-based Acoustic Models Using Self-attention With Augmented Memory (2020)0.00
• Dual-decoder Transformer For Joint Automatic Speech Recognition And Multilingual Speech Translation (2020)13.73
• Mixture-of-transformers: A Sparse And Scalable Architecture For Multi-modal Foundation Models (2024)0.00
• Transmask: A Compact And Fast Speech Separation Model Based On Transformer (2021)8.82
• Streaming Simultaneous Speech Translation With Augmented Memory Transformer (2020)6.77
• Diffspeaker: Speech-driven 3D Facial Animation With Diffusion Transformer (2024)5.24