When Should We Prefer Decision Transformers For Offline Reinforcement Learning?

Abstract

Offline reinforcement learning (RL) allows agents to learn effective, return-maximizing policies from a static dataset. Three popular algorithms for offline RL are Conservative Q-Learning (CQL), Behavior Cloning (BC), and Decision Transformer (DT), from the class of Q-Learning, Imitation Learning, and Sequence Modeling respectively. A key open question is: which algorithm is preferred under what conditions? We study this question empirically by exploring the performance of these algorithms across the commonly used D4RL and Robomimic benchmarks. We design targeted experiments to understand their behavior concerning data suboptimality, task complexity, and stochasticity. Our key findings are: (1) DT requires more data than CQL to learn competitive policies but is more robust; (2) DT is a substantially better choice than both CQL and BC in sparse-reward and low-quality data settings; (3) DT and BC are preferable as task horizon increases, or when data is obtained from human demonstrators;

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keybhargava2023when

Related papers

• Q-learning Decision Transformer: Leveraging Dynamic Programming For Conditional Sequence Modelling In Offline RL (2022)0.00
• When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning? (2022)0.00
• Q-value Regularized Decision Convformer For Offline Reinforcement Learning (2024)0.00
• Solving Continual Offline Reinforcement Learning With Decision Transformer (2024)0.00
• Double Check My Desired Return: Transformer With Target Alignment For Offline Reinforcement Learning (2025)0.00
• Belief-based Offline Reinforcement Learning For Delay-robust Policy Optimization (2025)0.00
• Quantum Decision Transformers (QDT): Synergistic Entanglement And Interference For Offline Reinforcement Learning (2025)0.00
• Representation Matters: Offline Pretraining For Sequential Decision Making (2021)0.00