Quantile Q-Learning: Revisiting Offline Extreme Q-Learning with Quantile Regression

AI summary

Overview

• Research area: Offline reinforcement learning (RL), specifically in-sample value-based methods that learn from fixed, pre-collected datasets without environment interaction.
• Technical level: Advanced — familiarity with reinforcement learning, Bellman equations, Q-functions, Extreme Value Theory, and quantile regression is assumed to fully appreciate the theory, though the high-level ideas are accessible to intermediate readers.
• Scope in one sentence: The paper proposes Quantile Q-Learning (QQL), a method that removes XQL's brittle, dataset-specific temperature hyperparameter by estimating it with quantile regression and stabilizes training with value regularization, achieving state-of-the-art offline RL performance using one fixed hyperparameter set across all tasks.

What This Paper Is About

Offline RL lets agents learn policies from data alone, which is critical when trial-and-error in the real world is expensive or dangerous. Extreme Q-Learning (XQL) is a strong but notoriously finicky offline RL algorithm: its behavior hinges on a temperature coefficient β that must be hand-tuned for each individual dataset and domain, and even its stabilized variant (MXQL) can be unstable during training. This paper's goal is to make XQL work robustly with a single, fixed hyperparameter setting across all tasks by learning β automatically from data.

Key Contributions

1. A principled, data-driven estimator for the temperature coefficient. The authors show that under mild (Gumbel-noise) assumptions, β can be generalized to a state-dependent function β(s) and recovered from the difference between two value functions via the Euler–Mascheroni constant. This removes the need for per-dataset tuning.
2. A quantile-regression formulation of the value functions. They prove that the relevant value functions correspond to specific quantiles of the Q-function's distribution and learn them directly with a quantile-regression loss, yielding a β-free variant of XQL.
3. Value regularization with mild generalization. To counter the over-conservatism of purely in-sample methods, they add a mild generalization term with conservative estimation, improving stability and mitigating overestimation of out-of-distribution actions.
4. Empirical validation with a single hyperparameter set. QQL matches or beats XQL, MXQL, and other baselines (BC, TD3+BC, CQL, IQL, EDAC, BCQ) on D4RL and NeoRL2 without any per-task tuning.

Main Findings

• Consistent hyperparameters match tuned baselines. Using λ = 1.0 and ζ = 1.0 everywhere, QQL scores competitively with—and often above—baselines that were individually tuned per dataset (e.g., 112.5 vs. 110.7 for MXQL on hopper-med-exp; 90.2 on walker2d-med-rep vs. 83.6 for MXQL).
• XQL collapses without dataset-specific tuning; QQL does not. In a direct comparison, XQL drops substantially when forced to use one shared hyperparameter set across domains (e.g., Gym Locomotion falls from 83.7 to 73.6), whereas QQL stays strong (85.6).
• Improved training stability. QQL shows smoother training curves and lower variance than MXQL, with reduced Q-value overestimation.
• Both new components matter. Ablations show removing value regularization or conservative estimation degrades performance; removing both causes substantial collapse.
• Low sensitivity to the remaining hyperparameters. Varying λ or ζ over {0.25, 0.5, 1.0, 2.0, 4.0} changes scores only marginally.
• The key assumption holds empirically. A toy example (with Gumbel-fitting and goodness-of-fit tests) and D4RL residual analyses support the claim that β(s) is consistent between the two Gumbel-noise assumptions.
• Modest computational cost. QQL requires roughly 1.6× XQL's wall-clock training time (4 vs. 2.5 hours for 1M steps on an RTX 4070) due to the extra quantile regression and value estimation.

Methodology in Plain English

The researchers start from XQL's central weakness: a temperature term β inside exponential expressions that appears in both the value loss and the policy update, and whose optimal value varies wildly across environments. Rather than tune it, they derive it.

1. Model the estimation error. They assume the gap between the learned Q-function and the true optimal Q-function follows a Gumbel distribution (a standard model for max-operator errors), with a state-dependent scale β(s).
2. Relate β(s) to value functions. Using properties of the Gumbel distribution, they prove that β(s) is proportional to the difference between two specific value functions—one tied to the optimal value and one to the estimated value. The proportionality constant turns out to be the Euler–Mascheroni constant (≈ 0.577).
3. Learn those value functions with quantile regression. They show each value function equals a fixed quantile (α₁ and α₂) of the Q-function's distribution, so they can be trained with a standard quantile (pinball) loss instead of an explicit β parameter.
4. Rewrite the Q and policy losses without β. The Q-function is trained with mean-squared Bellman error using the estimated value functions; the policy is updated with a modified advantage-weighted regression that no longer needs a hand-set temperature.
5. Add mild generalization. To avoid excessive conservatism, they also fit the value networks on actions sampled from the current policy (not just dataset actions), using conservative quantile levels for those out-of-sample estimates—essentially a pessimism-controlled extension of doubly mild generalization.
6. Test broadly. They evaluate on D4RL (Gym Locomotion, Adroit, AntMaze) and the harder NeoRL2 benchmark, keeping hyperparameters fixed the whole time.

Why This Matters

This work removes a major practical obstacle to deploying offline RL: the need for expensive, dataset-specific hyperparameter searches. Because XQL-style methods are already competitive, making them robust and self-calibrating substantially lowers the barrier to real-world use.

Real-world applications:

• Healthcare treatment planning — learning safe treatment policies from historical patient records without risking live patients.
• Robotics and industrial control — training manipulation or navigation policies from prior teleoperation logs, where new data collection is costly or hazardous.
• Autonomous driving — learning from recorded driving data without exposing vehicles or pedestrians to exploration risk.
• Recommendation and finance — sequential decision-making from logged user or market data where online exploration is expensive or regulated.

Industry relevance: Any organization sitting on large but static datasets (hospitals, robotics firms, autonomous vehicle companies, ad platforms) can benefit from offline RL methods that do not require interactive experimentation or per-deployment hyperparameter tuning. A single robust configuration is far easier to validate, ship, and maintain in production systems where reliability is critical.

Future Directions

• Extension to online RL. The authors propose adapting quantile-based β estimation and value regularization to online Extreme Q-Learning, potentially improving interactive training stability.
• Tighter theory. The consistency assumption between the β(s) scales of the two Gumbel models is currently supported empirically (toy examples, residual analysis); a formal proof would strengthen the foundation.
• Reducing computational overhead. The extra 1.6× training cost from quantile regression and dual value networks could be trimmed through shared representations or lighter estimators.
• Broader benchmarking. Testing on additional domains (e.g., real-world robotics logs, large-scale recommender systems) would clarify how far the "one hyperparameter set fits all" claim generalizes.

Target Audience

This paper is best suited for machine learning researchers and graduate students working on offline reinforcement learning, distributional RL, or in-sample value-based methods, as well as practitioners who need to deploy offline RL in cost-sensitive or safety-critical settings and want an algorithm that works out of the box. Readers should be comfortable with Q-learning, Bellman backups, and basic probability; the theoretical sections benefit from familiarity with Extreme Value Theory and quantile regression.

Authors’ abstract

Offline reinforcement learning (RL) enables policy learning from fixed datasets without further environment interaction, making it particularly valuable in high-risk or costly domains. Extreme $Q$-Learning (XQL) is a recent offline RL method that models Bellman errors using the Extreme Value Theorem, yielding strong empirical performance. However, XQL and its stabilized variant MXQL suffer from notable limitations: both require extensive hyperparameter tuning specific to each dataset and domain, and also exhibit instability during training. To address these issues, we proposed a principled method to estimate the temperature coefficient $β$ via quantile regression under mild assumptions. To further improve training stability, we introduce a value regularization technique with mild generalization, inspired by recent advances in constrained value learning. Experimental results demonstrate that the proposed algorithm achieves competitive or superior performance across a range of benchmark tasks, including D4RL and NeoRL2, while maintaining stable training dynamics and using a consistent set of hyperparameters across all datasets and domains.