Autoclip: Adaptive Gradient Clipping For Source Separation Networks

Abstract

Clipping the gradient is a known approach to improving gradient descent, but requires hand selection of a clipping threshold hyperparameter. We present AutoClip, a simple method for automatically and adaptively choosing a gradient clipping threshold, based on the history of gradient norms observed during training. Experimental results show that applying AutoClip results in improved generalization performance for audio source separation networks. Observation of the training dynamics of a separation network trained with and without AutoClip show that AutoClip guides optimization into smoother parts of the loss landscape. AutoClip is very simple to implement and can be integrated readily into a variety of applications across multiple domains.

Stats

• citations34
• S2 citations—
• github stars0
• HF likes0
• heat score11.58
• arxiv keyseetharaman2020autoclip

Related papers

• Revisit Micro-batch Clipping: Adaptive Data Pruning Via Gradient Manipulation (2024)0.00
• Efficiently Train ASR Models That Memorize Less And Perform Better With Per-core Clipping (2024)0.00
• End-to-end Source Separation With Adaptive Front-ends (2017)12.17
• APPLADE: Adjustable Plug-and-play Audio Declipper Combining DNN With Sparse Optimization (2022)5.84
• Image To Image Translation Based On Convolutional Neural Network Approach For Speech Declipping (2019)0.00
• A Style Transfer Approach To Source Separation (2019)3.58
• Remix-cycle-consistent Learning On Adversarially Learned Separator For Accurate And Stable Unsupervised Speech Separation (2022)3.58
• End-to-end Non-negative Autoencoders For Sound Source Separation (2019)2.26