An Empirical Evaluation Of Generic Convolutional And Recurrent Networks For Sequence Modeling

Abstract

For most deep learning practitioners, sequence modeling is synonymous with recurrent networks. Yet recent results indicate that convolutional architectures can outperform recurrent networks on tasks such as audio synthesis and machine translation. Given a new sequence modeling task or dataset, which architecture should one use? We conduct a systematic evaluation of generic convolutional and recurrent architectures for sequence modeling. The models are evaluated across a broad range of standard tasks that are commonly used to benchmark recurrent networks. Our results indicate that a simple convolutional architecture outperforms canonical recurrent networks such as LSTMs across a diverse range of tasks and datasets, while demonstrating longer effective memory. We conclude that the common association between sequence modeling and recurrent networks should be reconsidered, and convolutional networks should be regarded as a natural starting point for sequence modeling tasks. To assist relat

Stats

• citations0
• S2 citations—
• github stars4510
• HF likes0
• heat score7.31
• arxiv keybai2018an

Related papers

• Re-examination Of The Role Of Latent Variables In Sequence Modeling (2019)0.00
• Rethinking Recurrent Latent Variable Model For Music Composition (2018)7.50
• Learning The Sequential Temporal Information With Recurrent Neural Networks (2018)0.00
• On A Novel Training Algorithm For Sequence-to-sequence Predictive Recurrent Networks (2021)0.00
• A Comparison Of Adaptation Techniques And Recurrent Neural Network Architectures (2018)3.58
• Investigating Gated Recurrent Neural Networks For Speech Synthesis (2016)0.00
• From Nodes To Networks: Evolving Recurrent Neural Networks (2018)0.00
• Memory Visualization For Gated Recurrent Neural Networks In Speech Recognition (2016)11.76