Learning With Imperfect Models: When Multi-step Prediction Mitigates Compounding Error

Abstract

Compounding error, where small prediction mistakes accumulate over time, presents a major challenge in learning-based control. For example, this issue often limits the performance of model-based reinforcement learning and imitation learning. One common approach to mitigate compounding error is to train multi-step predictors directly, rather than relying on autoregressive rollout of a single-step model. However, it is not well understood when the benefits of multi-step prediction outweigh the added complexity of learning a more complicated model. In this work, we provide a rigorous analysis of this trade-off in the context of linear dynamical systems. We show that when the model class is well-specified and accurately captures the system dynamics, single-step models achieve lower asymptotic prediction error. On the other hand, when the model class is misspecified due to partial observability, direct multi-step predictors can significantly reduce bias and thus outperform single-step appro

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keysomalwar2025learning

Related papers

• Learning To Combat Compounding-error In Model-based Reinforcement Learning (2019)0.00
• Towards A Simple Approach To Multi-step Model-based Reinforcement Learning (2018)0.00
• Models As Agents: Optimizing Multi-step Predictions Of Interactive Local Models In Model-based Multi-agent Reinforcement Learning (2023)6.77
• Any-step Dynamics Model Improves Future Predictions For Online And Offline Reinforcement Learning (2024)0.00
• A Note On Loss Functions And Error Compounding In Model-based Reinforcement Learning (2024)0.00
• Self-correcting Models For Model-based Reinforcement Learning (2016)0.00
• Plan To Predict: Learning An Uncertainty-foreseeing Model For Model-based Reinforcement Learning (2023)0.00
• Diminishing Return Of Value Expansion Methods In Model-based Reinforcement Learning (2023)0.00