Multi-Agent Regime-Conditioned Diffusion (MARCD) for CVaR-Constrained Portfolio Decisions

AI summary

Overview

Research area: Machine learning for finance — generative time-series modeling (diffusion models), regime detection (hidden Markov models), and convex risk-constrained portfolio optimization (CVaR).

Technical level: Advanced. The paper combines a variance-preserving DDPM, a mixture-of-experts denoiser, a Rockafellar–Uryasev CVaR epigraph quadratic program, and proof sketches in the appendix.

Scope (one sentence): The paper introduces MARCD, a pipeline that chains a regime-conditioned diffusion generator to a CVaR-constrained convex allocator, and evaluates it against equal-weight, risk-parity, and Black–Litterman baselines on ten liquid ETFs over an out-of-sample window of 2020–2025.

What This Paper Is About

Financial returns switch abruptly between calm and crisis regimes, and standard mean–variance or Black–Litterman optimization tends to underestimate the left-tail co-movements that produce large peak-to-trough losses. The authors argue that most modern generative return models are trained with plain mean-squared error, are often unconditioned, and are decoupled from the decision that ultimately consumes their samples. MARCD's goal is to close that gap: generate scenarios that are explicitly conditioned on inferred market regimes, bias training toward adverse outcomes, and feed those scenarios into a convex, turnover-aware CVaR allocator under strict walk-forward evaluation.

Key Contributions

1. Tail-weighted diffusion objective. A training objective, L_tail, that multiplies the diffusion denoising error by (1 + η · 1{ℓ̃ ≤ Q_q(ℓ̃)}), where ℓ̃ = −min_j r_j is a portfolio-free proxy for the worst single-asset one-step loss, with q ∈ [0.05, 0.10] and η ∈ [1, 3]. The design explicitly avoids target leakage by not using portfolio weights during training.
2. Regime-expert (MoE) denoiser. A two-expert denoiser (base plus crisis head) mixed by a gate g_t = σ(MLP(z_t)) whose value increases with the HMM crisis posterior, intended to enrich co-crash structure without degrading calm-regime fit.
3. A multi-agent generative-to-decision pipeline. Regime inference (Gaussian HMM posteriors) → regime-conditioned scenario generation → signal extraction via blended, shrunk moments → allocation, with an explicit walk-forward protocol and audit logs derived from the allocator's KKT system (active constraints, tail weights, duals).
4. A governed CVaR epigraph QP allocator. A convex quadratic program combining a mean–variance regularizer, the Rockafellar–Uryasev CVaR term at level α = 0.95, budget/box constraints, and an ℓ₁ turnover cap, with O(N) linear epigraph constraints and an optional turnover penalty κ‖w − w_{t−1}‖₁.

Main Findings

• Drawdown reduction: Over the 2020–2025 out-of-sample window, MARCD records a MaxDD of 9.3% versus 14.1% for Black–Litterman (reported as a 34% reduction in the abstract and 34.1% in the conclusion) and 21.2% for equal-weight, described as 56% lower than EW, with absolute differences of 4.8% and 11.9%.
• Risk-adjusted performance: MARCD attains a Sharpe of 1.23 versus 1.02 for BL, 0.88 for risk parity, and 0.72 for EW; Sortino is 1.69 versus 1.50 (BL), 1.37 (RP), and 1.09 (EW); Calmar is 1.11 versus 0.70 (BL) and 0.38 (EW); return is 10.3% versus 9.9% (BL) and volatility is 8.4% versus 9.7% (BL).
• Scenario calibration: MARCD records the lowest KS (0.154), ES (0.289), VS (0.247), and CVaR₀.₉₅ error (15 bps) among the compared generators, but its LB p(|r|) of 0.50 and VaR₀.₉₅ UC p of 0.58 are slightly below TimeVAE's 0.52 and 0.63, which the paper calls "competitive, slightly below TimeVAE."
• Statistical significance: A stationary block bootstrap with B = 1000 and block length 20 indicates MARCD's Sharpe exceeds BL and EW at two-sided p < 0.05; the reported Sharpe uplift confidence intervals are 0.51 [0.31, 0.71] versus EW, 0.21 [0.07, 0.35] versus BL, and 0.35 [0.18, 0.51] versus RP.
• Ablation sensitivity: Removing regime conditioning lowers Sharpe to 1.13 and raises MaxDD to 11.3%; removing the CVaR term lowers Sharpe to 1.12 and raises MaxDD to 14.6%; setting λ = 0.0 gives Sharpe 1.13 / MaxDD 12.1%, and λ = 1.0 gives Sharpe 1.12 / MaxDD 12.8%. VaR₀.₉₅ UC p falls from 0.58 to 0.49, 0.46, 0.52, and 0.48 respectively, and CVaR₀.₉₅ error rises from 15 to 22, 27, 21, and 24 bps.
• Regime count selection: Rolling BIC favors K = 3 (ΔBIC 0 versus +18 for K = 2 and +9 for K = 4), and K = 3 also yields the best OOS Sharpe (1.23) with MaxDD 9.3% and VaR₀.₉₅ UC p of 0.58.
• Robustness: Results are reported as stable across blend weights λ ∈ [0.3, 0.7] and CVaR trade-off γ ∈ [0.5, 1.5], and robust under doubled transaction costs and a varied turnover cap; turnover across ablations stays in the 15.3–15.9% range.
• Stress behavior: In the COVID-19 window (Feb–Apr 2020) and the 2022 inflation shock (Jun–Oct 2022), MARCD's drawdowns are smaller than BL and EW; removing regime conditioning lowers crisis Sharpe and weakens VaR coverage.

Methodology in Plain English

The authors build a four-stage assembly line that runs at each monthly rebalance. First, a Gaussian hidden Markov model with K states is fitted only on data available up to that date, producing posterior probabilities π_t,k that describe which regime the market is probably in; these posteriors become a conditioning vector z_t. Second, a variance-preserving diffusion model (a conditional DDPM with a UNet-style denoiser, 4 down and 4 up blocks, roughly 1–2M parameters, cosine noise schedule, EMA = 0.999, AdamW at 1e-4, batch 256, 250k steps, seed 2020) generates N = 1024 next-period return scenarios conditioned on z_t. Training uses the tail-weighted loss so that samples where the worst single-asset return falls in the lower q-quantile receive extra emphasis (η multiplies their error), and a gate mixes a general expert with a crisis-specific expert. Third, the Signal Agent blends synthetic and historical moments with weight λ (and applies covariance shrinkage toward the identity), yielding μ̂_t and Σ̂_t. Fourth, the Allocation Agent solves a convex quadratic program at CVaR level α = 0.95 over d = 10 assets, minimizing the negative expected return plus a variance penalty plus the Rockafellar–Uryasev CVaR expression, subject to full investment, box bounds, and an ℓ₁ turnover cap of 0.20; interior-point complexity is O(d³ + Nd²). Backtests rebalance monthly, charge identical 10 bps costs across all strategies, and compare against equal-weight, risk parity, and Black–Litterman run under the same partial-rebalance projection.

Why This Matters

The paper argues that scenario quality is the bottleneck for tail-risk-aware portfolios: robust and CVaR formulations exist, but they "hinge on scenario quality." MARCD's claim is that biasing generation toward adverse outcomes and conditioning on regime state improves exactly the left-tail co-movements that drive realized drawdowns — and because the allocator is convex and its KKT duals are logged, the pipeline is auditable rather than a black box.

Real-world applications:

• Multi-asset portfolio construction where drawdown control matters more than raw return, using the ten-ETF liquid universe studied here.
• Risk management and model governance, via the solver-side audit trail linking regime posteriors and tail weights to realized trades.
• Risk-profile customization, since the paper reports five application profiles (Conservative, Crisis-Focused, Aggressive, Balanced, Momentum) trading off volatility, drawdown, and return.
• Stress-testing and scenario analysis, using the regime-conditioned generator to produce crisis-conditional scenario sets rather than unconditioned historical bootstraps.

Industry relevance: The framework is aimed at governed, walk-forward institutional workflows — explicit budget, box, and turnover constraints, consistent transaction cost assumptions, and logging suitable for model-risk and compliance review. The paper's own limitations section notes that diffusion sampling and QP solves impose nontrivial compute, which constrains intraday deployment.

Future Directions

1. Decision-aware (end-to-end) training that differentiates through a spectral CVaR objective, turnover, and holdings penalties — the appendix sketches a bilevel program with an envelope hypergradient and implicit differentiation through the QP's KKT system.
2. Short multi-step scenario generation with a convex pathwise drawdown CVaR objective, extending beyond the current one-step-ahead setting.
3. Distributional robustness and deeper-tail dependence, for example Wasserstein-DRO and copula reshaping.
4. Online regime updates with risk overlays and safe fallbacks, plus faster inference via diffusion distillation or consistency models to approach real-time use. The paper also leaves spectral risk measures other than Rockafellar–Uryasev CVaR to future work.

Target Audience

Quantitative researchers and practitioners working at the intersection of generative machine learning and portfolio optimization; risk managers and model-governance teams interested in auditable, constraint-explicit allocation pipelines; and machine learning researchers studying conditional diffusion, mixture-of-experts denoisers, and tail-weighted training objectives. Readers need comfort with hidden Markov models, denoising diffusion, and convex optimization with CVaR to follow the methodology and appendix proofs.

Authors’ abstract

We examine whether regime-conditioned generative scenarios combined with a convex CVaR allocator improve portfolio decisions under regime shifts. We present MARCD, a generative-to-decision framework with: (i) a Gaussian HMM to infer latent regimes; (ii) a diffusion generator that produces regime-conditioned scenarios; (iii) signal extraction via blended, shrunk moments; and (iv) a governed CVaR epigraph quadratic program. Contributions: Within the Scenario stage we introduce a tail-weighted diffusion objective that up-weights low-quantile outcomes relevant for drawdowns and a regime-expert (MoE) denoiser whose gate increases with crisis posteriors; both are evaluated end-to-end through the allocator. Under strict walk-forward on liquid multi-asset ETFs (2005-2025), MARCD exhibits stronger scenario calibration and materially smaller drawdowns: MaxDD 9.3% versus 14.1% for BL (a 34% reduction) over 2020-2025 out-of-sample. The framework provides an auditable pipeline with explicit budget, box, and turnover constraints, demonstrating the value of decision-aware generative modeling in finance.