Rl-star: Theoretical Analysis Of Reinforcement Learning Frameworks For Self-taught Reasoner

Abstract

The reasoning abilities of large language models (LLMs) have improved with chain-of-thought (CoT) prompting, allowing models to solve complex tasks stepwise. However, training CoT capabilities requires detailed reasoning data, which is often scarce. The self-taught reasoner (STaR) framework addresses this by using reinforcement learning to automatically generate reasoning steps, reducing reliance on human-labeled data. Although STaR and its variants have demonstrated empirical success, a theoretical foundation explaining these improvements is lacking. This work provides a theoretical framework for understanding the effectiveness of reinforcement learning on CoT reasoning and STaR. Our contributions are: (1) criteria for the quality of pre-trained models necessary to initiate effective reasoning improvement; (2) an analysis of policy improvement, showing why LLM reasoning improves iteratively with STaR; (3) conditions for convergence to an optimal reasoning policy; and (4) an examinatio

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keychang2024rl

Related papers

• Think In Games: Learning To Reason In Games Via Reinforcement Learning With Large Language Models (2025)0.00
• MARSHAL: Incentivizing Multi-agent Reasoning Via Self-play With Strategic Llms (2025)0.00
• Reinforcement Learning With Knowledge Representation And Reasoning: A Brief Survey (2023)0.00
• From Laws To Motivation: Guiding Exploration Through Law-based Reasoning And Rewards (2024)0.00
• DYSTIL: Dynamic Strategy Induction With Large Language Models For Reinforcement Learning (2025)0.00
• Scheduling Your LLM Reinforcement Learning With Reasoning Trees (2026)0.00
• Mental Modeling Of Reinforcement Learning Agents By Language Models (2024)0.00
• Free Energy-driven Reinforcement Learning With Adaptive Advantage Shaping For Unsupervised Reasoning In Llms (2026)0.00