Sign-Separated Finite-Time Error Analysis of Q-Learning

Constant step-size Q-learning updates are known to overestimate action values due to the Bellman max operator, but the asymmetry between how positive and negative errors propagate has lacked a clean theoretical treatment. This work decomposes Q-learning error componentwise into negative and positive parts, showing the negative side can be bounded by a stable LTI system tied to the optimal policy — which converges at least as fast as the positive-side bound governed by a linear switching system. The key finding is that positive errors get selected and amplified by the Bellman maximum while negative errors do not, giving a formal finite-time explanation for Q-learning's overestimation bias in both deterministic and stochastic settings.

No production traction yet. The GitHub repos referencing it are generic arxiv aggregators, not implementations or downstream research tools. Zero citations on Semantic Scholar. This is pure theoretical RL analysis, useful as a reference for researchers working on Q-learning convergence guarantees but not something builders are picking up.