Mamba-360: Survey Of State Space Models As Transformer Alternative For Long Sequence Modelling: Methods, Applications, And Challenges
2024 · Badri Narayana Patro, Vijay Srinivas Agneeswaran
Abstract
Sequence modeling is a crucial area across various domains, including Natural Language Processing (NLP), speech recognition, time series forecasting, music generation, and bioinformatics. Recurrent Neural Networks (RNNs) and Long Short Term Memory Networks (LSTMs) have historically dominated sequence modeling tasks like Machine Translation, Named Entity Recognition (NER), etc. However, the advancement of transformers has led to a shift in this paradigm, given their superior performance. Yet, transformers suffer from \(O(N^2)\) attention complexity and challenges in handling inductive bias. Several variations have been proposed to address these issues which use spectral networks or convolutions and have performed well on a range of tasks. However, they still have difficulty in dealing with long sequences. State Space Models(SSMs) have emerged as promising alternatives for sequence modeling paradigms in this context, especially with the advent of S4 and its variants, such as S4nd, Hippo,
Authors
(none)
Tags
Stats
Related papers
- Advancing Intelligent Sequence Modeling: Evolution, Trade-offs, And Applications Of State- Space Architectures From S4 To Mamba (2025)0.00
- A Comparative Study On Transformer Vs RNN In Speech Applications (2019)20.07
- Audio Mamba: Bidirectional State Space Model For Audio Representation Learning (2024)11.58
- Samba-asr: State-of-the-art Speech Recognition Leveraging Structured State-space Models (2025)0.00
- Dual-path Mamba: Short And Long-term Bidirectional Selective Structured State Space Models For Speech Separation (2024)4.12
- S-transformer: Segment-transformer For Robust Neural Speech Synthesis (2020)0.00
- SSAMBA: Self-supervised Audio Representation Learning With Mamba State Space Model (2024)0.00
- Audio Mamba: Selective State Spaces For Self-supervised Audio Representations (2024)9.23