Gradient Coupling: The Hidden Barrier To Generalization In Agentic Reinforcement Learning

Abstract

Reinforcement learning (RL) is a dominant paradigm for training autonomous agents, yet these agents often exhibit poor generalization, failing to adapt to scenarios not seen during training. In this work, we identify a fundamental cause of this brittleness, a phenomenon which we term "gradient coupling." We hypothesize that in complex agentic tasks, the high similarity between distinct states leads to destructive interference between gradients. Specifically, a gradient update that reinforces an optimal action in one state can inadvertently increase the likelihood of a suboptimal action in a similar, yet different, state. To solve this, we propose a novel objective where the actor is trained to simultaneously function as a classifier that separates good and bad actions. This auxiliary pressure compels the model to learn disentangled embeddings for positive and negative actions, which mitigates negative gradient interference and improve the generalization performance. Extensive experimen

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyliu2025gradient

Related papers

• Dynamics Generalization Via Information Bottleneck In Deep Reinforcement Learning (2020)0.00
• Understanding What Affects The Generalization Gap In Visual Reinforcement Learning: Theory And Empirical Evidence (2024)5.84
• Efficient Reinforcement Learning Via Decoupling Exploration And Utilization (2023)2.56
• Improving Generalization In Reinforcement Learning With Mixture Regularization (2020)0.00
• The Principle Of Unchanged Optimality In Reinforcement Learning Generalization (2019)0.00
• Improving Generalization In Meta Reinforcement Learning Using Learned Objectives (2019)0.00
• Generalizing Skills With Semi-supervised Reinforcement Learning (2016)0.00
• Training Generalizable Collaborative Agents Via Strategic Risk Aversion (2026)0.00