STHTD-MP: Faster Convergence in Off-Policy Prediction with TD

Temporal difference (TD) learning has been the backbone of prediction in reinforcement learning for decades, yet tuning its behavior in off-policy scenarios remains an open problem with real practical consequences. A new preprint published on May 29, 2026 on arXiv (cs.AI) proposes STHTD-MP, a variant that replaces the standard feature covariance metric with geometric information extracted directly from the behavior policy. The result is a more informed update that, according to the theoretical analysis provided, converges with guarantees under standard assumptions.

What the paper proposes

Gradient-TD methods, such as GTD2 or TDC, solve the off-policy instability problem by framing prediction as a saddle-point problem between primal and auxiliary variables. Their stability is well understood, but practical speed depends heavily on the metric defining the space of auxiliary variables. The usual choice is the feature covariance matrix, which is generic and does not exploit the structure of the specific problem.

What STHTD-MP does is replace that metric with the symmetric part of the Bellman matrix induced by the behavior policy. This choice is not arbitrary: hybrid TD methods had already shown that including information from behavior policy transitions produces richer update geometries. The novel step here is integrating that idea within the Mirror-Prox framework, a saddle-point optimization algorithm that alternates between a prediction phase and a correction phase, hence the MP suffix, rather than following the gradient naively.

An important operational advantage: STHTD-MP maintains a single learning rate for both primal and auxiliary variables, which significantly simplifies hyperparameter tuning in practice.

Why convergence analysis matters

The paper does not stop at the empirical proposal. It offers a formal convergence proof for linear prediction with fixed policy, organized in three steps:

1. Positive definiteness of the induced metric: the symmetric part of the behavior policy's Bellman matrix is proved to be positive definite under standard conditions, guaranteeing the metric is valid.
2. Hurwitz mean system: the system of ordinary differential equations (ODE) associated with the stochastic process has all eigenvalues with negative real part, a necessary condition for classical ODE analysis of stochastic approximation to apply.
3. Boundedness via Lyapunov and process convergence: the stochastic recursion remains bounded thanks to a Lyapunov function argument, from which convergence follows.

This type of theoretical guarantee remains scarce in modern TD literature, where many practical advances are published without robust convergence proofs. That the paper includes them is relevant for those who need certainty before deploying the method in critical environments.

Who benefits from this work

This work appeals primarily to three profiles:

• Reinforcement learning researchers working on off-policy prediction who need methods with solid theoretical foundations, not just empirical heuristics.
• Engineers designing control or recommendation systems with reinforcement learning, where the policy generating the data (behavior) differs from the one being evaluated (target), a classic off-policy scenario.
• Teams integrating RL into pipelines with linear function approximation, where computational efficiency matters and reducing the number of hyperparameters to tune has direct value.

That said, the paper works in the linear approximation regime and fixed policy setting, which excludes cases with deep neural networks or non-stationary policies. Extrapolation to those contexts would require additional work that the authors do not address here.

Context in the research ecosystem

The gradient-TD family of methods has been active since the work of Sutton, Maei, and Szepesvári in the late 2000s. Since then, variants have appeared such as ETD, Emphatic TD, or hybrid methods that partially inspired this work. STHTD-MP does not aim to replace all of them, but to occupy a specific niche: combining the stability of the Mirror-Prox framework with more expressive geometry without increasing configuration complexity.

From ClaudeWave we observe with interest how theoretical research in TD continues to produce concrete contributions, though the path from a preprint with linear guarantees to a usable production library is typically longer than the pace of arXiv publication suggests.