Zipfian Environments For Reinforcement Learning

Abstract

As humans and animals learn in the natural world, they encounter distributions of entities, situations and events that are far from uniform. Typically, a relatively small set of experiences are encountered frequently, while many important experiences occur only rarely. The highly-skewed, heavy-tailed nature of reality poses particular learning challenges that humans and animals have met by evolving specialised memory systems. By contrast, most popular RL environments and benchmarks involve approximately uniform variation of properties, objects, situations or tasks. How will RL algorithms perform in worlds (like ours) where the distribution of environment features is far less uniform? To explore this question, we develop three complementary RL environments where the agent's experience varies according to a Zipfian (discrete power law) distribution. On these benchmarks, we find that standard Deep RL architectures and algorithms acquire useful knowledge of common situations and tasks, but

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keychan2022zipfian

Related papers

• Emergent Complexity And Zero-shot Transfer Via Unsupervised Environment Design (2020)0.00
• Discovering Minimal Reinforcement Learning Environments (2024)0.00
• DRED: Zero-shot Transfer In Reinforcement Learning Via Data-regularised Environment Design (2024)1.81
• Jumanji: A Diverse Suite Of Scalable Reinforcement Learning Environments In JAX (2023)0.00
• Eden: A Unified Environment Framework For Booming Reinforcement Learning Algorithms (2021)0.00
• Learning Curricula In Open-ended Worlds (2023)0.00
• Ecological Reinforcement Learning (2020)8.35
• From Two-dimensional To Three-dimensional Environment With Q-learning: Modeling Autonomous Navigation With Reinforcement Learning And No Libraries (2024)0.00