Simple Agent, Complex Environment: Efficient Reinforcement Learning With Agent States

Abstract

We design a simple reinforcement learning (RL) agent that implements an optimistic version of \(Q\)-learning and establish through regret analysis that this agent can operate with some level of competence in any environment. While we leverage concepts from the literature on provably efficient RL, we consider a general agent-environment interface and provide a novel agent design and analysis. This level of generality positions our results to inform the design of future agents for operation in complex real environments. We establish that, as time progresses, our agent performs competitively relative to policies that require longer times to evaluate. The time it takes to approach asymptotic performance is polynomial in the complexity of the agent's state representation and the time required to evaluate the best policy that the agent can represent. Notably, there is no dependence on the complexity of the environment. The ultimate per-period performance loss of the agent is bounded by a con

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keydong2021simple

Related papers

• Computationally Efficient Reinforcement Learning: Targeted Exploration Leveraging Simple Rules (2022)2.26
• A Strongly Asymptotically Optimal Agent In General Environments (2019)0.00
• An Agent Design With Goal Reaching Guarantees For Enhancement Of Learning (2024)0.00
• The Effective Horizon Explains Deep RL Performance In Stochastic Environments (2023)3.42
• Sample-efficient Robust Multi-agent Reinforcement Learning In The Face Of Environmental Uncertainty (2024)0.00
• Improving Generalization To New Environments And Removing Catastrophic Forgetting In Reinforcement Learning By Using An Eco-system Of Agents (2022)0.00
• Strategically Efficient Exploration In Competitive Multi-agent Reinforcement Learning (2021)0.00
• On Value Functions And The Agent-environment Boundary (2019)0.00