CAPO: Counterfactual Credit Assignment In Sequential Cooperative Teams

Abstract

In cooperative teams where agents act in a fixed order and share a single team reward, it is hard to know how much each agent contributed, and harder still when agents are updated one at a time because data collected earlier no longer reflects the new policies. We introduce the Sequential Aristocrat Utility (SeqAU), the unique per-agent learning signal that maximizes the individual learnability of each agent's action, extending the classical framework of Wolpert and Tumer (2002) to this sequential setting. From SeqAU we derive CAPO (Counterfactual Advantage Policy Optimization), a critic-free policy-gradient algorithm. CAPO fits a per-agent reward decomposition from group rewards and computes the per-agent advantage in closed form plus a handful of forward passes through the current policy, requiring no extra environment calls beyond the initial batch. We give analytic bias and variance bounds and validate them on a controlled sequential bandit, where CAPO's advantage over standard bas

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keydeshmukh2026capo

Related papers

• Cooperative Game-theoretic Credit Assignment For Multi-agent Policy Gradients Via The Core (2025)0.00
• Assigning Credit With Partial Reward Decoupling In Multi-agent Proximal Policy Optimization (2024)0.00
• Multi-agent Policy Optimization With Approximatively Synchronous Advantage Estimation (2020)0.00
• Counterfactual Multi-agent Policy Gradients (2017)0.00
• Solving Common-payoff Games With Approximate Policy Iteration (2021)3.58
• MACRPO: Multi-agent Cooperative Recurrent Policy Optimization (2021)0.00
• Agent-temporal Credit Assignment For Optimal Policy Preservation In Sparse Multi-agent Reinforcement Learning (2024)0.00
• Order Matters: Agent-by-agent Policy Optimization (2023)0.00