Boosting CVaR Policy Optimization with Quantile Gradients

AI summary

Overview

Research area: Risk-averse reinforcement learning (RARL), specifically policy-gradient optimization of static Conditional Value-at-Risk (CVaR) and its augmentation with the related quantile / Value-at-Risk (VaR) objective.

Technical level: Advanced. The paper is heavily mathematical, building on nested Bellman operators, contraction-mapping proofs, quantile regression losses, and the elicitability of quantiles.

Scope in one sentence: The paper proposes augmenting trajectory-based CVaR policy gradient with a quantile (VaR) policy-gradient term, because the quantile objective admits a dynamic-programming formulation that uses all sampled data, thereby fixing the sample inefficiency of CVaR-PG while leaving the CVaR objective unchanged.

What This Paper Is About

Optimizing CVaR with policy gradients is sample inefficient: the standard estimator uses only the worst-performing α fraction of a batch of N trajectories and throws the rest away, so it is blind to good outcomes and can even suffer vanishing gradients. The authors' goal is to accelerate CVaR policy learning without changing the CVaR objective, by adding an expected-quantile term — justified because CVaR is exactly the expectation of the tail quantiles (Eq. 1, CVaR_α[x̃] = (1/α)∫₀^α VaR_β[x̃] dβ). They then derive a VaR policy-gradient method that works inside the Markovian policy class, so it can be combined with existing CVaR-PG.

Key Contributions

1. A new CVaR objective with an added quantile term. The authors reformulate the problem as max_{π̄ ∈ Π_M} (1 − ω) CVaR_α[g̃₀] + ω E_{β ∼ U[0,α]}[VaR_β[g̃₀]] (Eq. 8), where ω ∈ (0, 1) is a trade-off parameter. The augmentation does not alter the CVaR objective, since CVaR corresponds to the expectation of quantile over the tail.

2. A new Bellman optimality operator for the state-only quantile value function. Motivated by the elicitability of quantiles and by the soft quantile loss of Hau et al. (2025), the paper defines T*_{ε,κ} v(s, α) (Eq. 9), using a TD-like error δ_v(s, α, r, s′, u) = r + γ v(s′, u) − v(s, α) with u ∼ U[0,1]. It proves the operator is a contraction for step size η ∈ (0, κ] (Proposition 3.1) and that the optimal quantile value v*(s, α) is its unique fixed point when ε = 0 and κ = 0 (Proposition 3.2). A practical version using the simple quantile loss is also given. Notably, this operator cannot be obtained by simply taking the maximum over a of the Q-learning update, because l_α(·) is non-linear.

3. Reduction of VaR optimization to the Markovian policy class. The authors prove that an optimal history-dependent (non-Markovian) VaR policy can still be executed from the quantile value function q^{π̄*} of a stationary Markovian policy by tracking cumulative rewards (Algorithm 3, Proposition 3.3), and they derive a corresponding optimization objective analogous to an advantage function (Proposition 3.4, Eq. 12). This turns the problem into a Markovian VaR-PG update, with the quantile level at each state determined by α_t = min{β | v(s_t, β) ≥ (v(s_{t−1}, α_{t−1}) − r_{t−1})/γ} (Eq. 13).

4. A combined algorithm with practical design choices. The final method combines VaR-PG and CVaR-PG. It discretizes quantile levels uniformly, with a value network that outputs I quantile values at the levels {½((i−1)/I + i/I)}_{i=1}^{I}; to prevent quantile crossing, the first output is a base value and the remaining I − 1 outputs are non-negative deltas produced through a softplus function (a simple monotonicity mechanism replacing prior approaches such as Zhou et al. 2020 and Luo et al. 2022). It also generalizes one-step advantage estimation to multi-step rollout with a weighted average controlled by λ ∈ (0, 1) (Eq. 15).

Main Findings

• Diagnosis of CVaR-PG's inefficiency: The estimator (Eq. 3) discards the 1 − α portion of sampled trajectories, focuses exclusively on worst-performing outcomes (termed "blindness to success" by Greenberg et al. 2022), and can produce a zero gradient when the left tail of the return distribution is flat, since the weights I{R(τᵢ) ≤ q̂_α}(R(τᵢ) − q̂_α) can vanish.

• Why quantiles help: Unlike static CVaR, static quantile (VaR) optimization admits a nested dynamic-programming formulation (following Hau et al. 2025), so all sampled data can be used for learning, and good-performing trajectories are captured by the quantile component instead of being ignored.

• Theoretical guarantees: The new VaR Bellman optimality operator is a contraction for η ∈ (0, κ] with v* as its unique fixed point at ε = 0, κ = 0; the policy-evaluation operator under π̂ is likewise shown to be a contraction, with its fixed point characterized when ε = 0 and κ = 0; and for finite action spaces with softmax parameterization, the fixed point of the policy operator is shown (in Appendix A.6) to be pointwise monotonically increasing.

• Markovian VaR execution is possible: Proposition 3.3 shows that running Algorithm 3 with the quantile value function of a stationary Markovian α₀-VaR-optimal policy results in executing that policy, using cumulative-reward tracking as in the history-dependent procedure.

• Empirical claims: The abstract states that in "domains with verifiable risk-averse behavior" the algorithm within the Markovian policy class substantially improves upon CVaR-PG and consistently outperforms other existing methods, and the introduction adds that it can learn a risk-averse policy with high return and low risk in terms of CVaR when others may fail. The specific environments, baselines, dataset sizes, and numeric results are not reported in the available (truncated) content of this paper — the text ends mid-sentence in Section 3.3 before the experimental section.

Methodology in Plain English

The authors start from a known weakness: to optimize the average of a policy's worst-case returns, the standard policy-gradient method samples a batch of trajectories and updates using only the worst α share of them, ignoring everything else. One way out is to look at a closely related quantity — the quantile, or Value-at-Risk — because CVaR is just the average of the quantiles in the tail. Unlike CVaR, the quantile has a recursive (Bellman-style) structure, which means every sampled transition can contribute to learning.

The authors first write down such a recursive equation for the quantile value function and prove it behaves well (it is a contraction with a unique fixed point). Rather than learning an action-value function — which tends to be high variance — they derive a state-only version that plugs directly into modern actor-critic policy updates, defining an advantage-like signal as the derivative of the quantile-regression loss applied to the TD error.

Then they confront a practical obstacle: the exact VaR-optimal policy is history-dependent, and executing it requires an (s, α)-indexed policy plus a procedure that tracks accumulated reward. Under function approximation this is hard to learn reliably. They prove instead that a simpler stationary, Markovian policy suffices if you track cumulative rewards and use the Markovian policy's own quantile value function to decide which quantile level applies at each visited state. This yields a Markovian policy-gradient update where each transition is tagged with a risk level α_t computed on the fly.

Finally they blend the two gradients — the CVaR term weighted by (1 − ω) and the averaged quantile term weighted by ω — into one objective. Engineering details make this trainable: a value network that outputs many quantile values and enforces monotonic ordering via accumulated softplus deltas (to avoid "quantile crossing"), and a multi-step advantage estimator that trades off bias and variance through λ.

Why This Matters

• Impact on research: The paper offers a different route to sample-efficient risk-averse RL than existing fixes. Where Greenberg et al. (2022) require control over environment stochasticity, Luo et al. (2024) work mainly when risk-neutral and risk-averse actions coincide across most states, and Kim and Min (2024, PCVaR-PG) and Mead et al. (2025, RET-CAP) reshape trajectory returns into per-step rewards, this work instead augments the gradient itself with a quantile term that is provably unchanged in its CVaR component. It also extends Markovian execution guarantees for VaR beyond the tabular settings where Hau et al. (2025) validated them.

• Real-world applications: The paper does not enumerate specific application domains, but risk-averse sequential decision-making of the kind studied here (CVaR-optimal policies that avoid catastrophic outcomes) is relevant to, for example:

  • Safety-critical control and robotics, where the worst-case outcome matters more than the average.
  • Financial decision-making and portfolio or trading policies where tail losses must be bounded.
  • Clinical or treatment-sequencing decisions, where harmful outcomes carry asymmetric cost.
  • Energy and infrastructure operations, where rare failure events are extremely expensive.

• Industry relevance: Any deployment setting where a policy must respect a tail-risk budget while still being trained from a realistically limited number of samples benefits from the sample-efficiency angle. The practical machinery in the paper — monotone quantile network outputs, multi-step advantage estimation, a single trade-off parameter ω — is designed to slot into standard deep actor-critic pipelines rather than requiring a bespoke training loop.

Future Directions

• Convergence of the Markovian VaR update. The authors state explicitly that although π̄* attains the maximum of the objective in Eq. 12 under v^{π̄*}, it remains an open question whether iterative updates starting from an arbitrary π̄ and v converge to π̄*.

• Robustness under function approximation. The α-selection step (Eq. 13) relies on the learned quantile value function being monotone in the risk level and accurate enough to invert; how reliably this holds with deep networks — and how it interacts with the discretized quantile grid over [0,1] — is not settled in the presented theory.

• Choosing the augmentation weight. The trade-off parameter ω ∈ (0,1) in Eq. 8 has no stated automatic selection rule; understanding how it trades off CVaR optimality against the quantile auxiliary signal is an open design question.

• Broader risk measures. The paper focuses narrowly on static CVaR and static VaR (quantile); whether the same augmentation idea applies to other risk metrics mentioned in the introduction — variance, Gini deviation, entropic risk, entropic VaR — is left open.

Target Audience

This paper is aimed at reinforcement learning researchers and graduate students working on risk-sensitive or risk-averse RL, constrained and safe RL, and distributional RL, as well as theoretically inclined readers comfortable with Bellman operators, contraction arguments, and quantile regression. Practitioners building safety-critical decision systems who want a drop-in way to make CVaR policy gradient training more sample-efficient will find the algorithmic construction and engineering details useful, though the derivation itself assumes a strong mathematical background.

Authors’ abstract

Optimizing Conditional Value-at-risk (CVaR) using policy gradient (a.k.a CVaR-PG) faces significant challenges of sample inefficiency. This inefficiency stems from the fact that it focuses on tail-end performance and overlooks many sampled trajectories. We address this problem by augmenting CVaR with an expected quantile term. Quantile optimization admits a dynamic programming formulation that leverages all sampled data, thus improves sample efficiency. This does not alter the CVaR objective since CVaR corresponds to the expectation of quantile over the tail. Empirical results in domains with verifiable risk-averse behavior show that our algorithm within the Markovian policy class substantially improves upon CVaR-PG and consistently outperforms other existing methods.