Re:frame -- Retrieving Experience From Associative Memory

Abstract

Offline reinforcement learning (RL) often deals with suboptimal data when collecting large expert datasets is unavailable or impractical. This limitation makes it difficult for agents to generalize and achieve high performance, as they must learn primarily from imperfect or inconsistent trajectories. A central challenge is therefore how to best leverage scarce expert demonstrations alongside abundant but lower-quality data. We demonstrate that incorporating even a tiny amount of expert experience can substantially improve RL agent performance. We introduce Re:Frame (Retrieving Experience From Associative Memory), a plug-in module that augments a standard offline RL policy (e.g., Decision Transformer) with a small external Associative Memory Buffer (AMB) populated by expert trajectories drawn from a separate dataset. During training on low-quality data, the policy learns to retrieve expert data from the Associative Memory Buffer (AMB) via content-based associations and integrate them in

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyzelezetsky2025re

Related papers

• Adamemento: Adaptive Memory-assisted Policy Optimization For Reinforcement Learning (2024)0.00
• Remember And Forget For Experience Replay (2018)0.00
• Augmented Replay Memory In Reinforcement Learning With Continuous Control (2019)5.24
• AWAC: Accelerating Online Reinforcement Learning With Offline Datasets (2020)0.00
• Adaptive Replay Buffer For Offline-to-online Reinforcement Learning (2025)0.00
• Prioritized Trajectory Replay: A Replay Memory For Data-driven Reinforcement Learning (2023)0.00
• Replay For Safety (2021)0.00
• Stabilising Experience Replay For Deep Multi-agent Reinforcement Learning (2017)0.00