Abstract
Q-value iteration (Q-VI) is usually analyzed through the γ-contraction of the Bellman operator. This argument proves convergence to Q^*, but it gives only a coarse account of when the induced greedy policy becomes optimal. We study discounted Q-VI as a switching system and focus on the practically optimal solution set (POSS), the set of Q-functions whose tie-broken greedy policies are optimal. The main result shows that Q-VI reaches the optimal action class in finite time by entering an invariant tube around X₁=Q^*+span(1), which is contained in the POSS. For every ε>0, the distance to X₁ satisfies an exponential bound with rate (ρ+ε)^k, where ρ is the joint spectral radius of the projected switching family restricted to directions transverse to X₁. When ρ<γ, this transverse convergence is faster than the classical contraction rate. The analysis separates fast policy identification from the subsequent convergence to Q^*, which may still be governed by the all-ones mode. We also give spectral and graph-theoretic conditions under which the strict inequality ρ<γ holds or fails.