Sparse Actiongen: Accelerating Diffusion Policy With Real-time Pruning

Abstract

Diffusion Policy has dominated action generation due to its strong capabilities for modeling multi-modal action distributions, but its multi-step denoising processes make it impractical for real-time visuomotor control. Existing caching-based acceleration methods typically rely on $\text{{}static\}$ schedules that fail to adapt to the $\text{{}dynamics\}$ of robot-environment interactions, thereby leading to suboptimal performance. In this paper, we propose $\underline{\{}\text{{}S\}\}$parse $\underline{\{}\text{{}A\}\}$ction$\underline{\{}\text{{}G\}\}$en ($\text{{}SAG\}$) for extremely sparse action generation. To accommodate the iterative interactions, SAG customizes a rollout-adaptive prune-then-reuse mechanism that first identifies prunable computations globally and then reuses cached activations to substitute them during action diffusion. To capture the rollout dynamics, SAG parameterizes an observation-conditioned diffusion pruner for environment-aware adaptation and