Uncertainty Quantification for Reduced-Order Surrogate Models Applied to Cloud Microphysics

AI summary

Overview

Research area: Uncertainty quantification (UQ) for machine-learning surrogate models, specifically latent-space reduced-order models (ROMs), applied to atmospheric cloud microphysics.

Technical level: Advanced. The paper assumes familiarity with reduced-order modeling, autoencoders, latent dynamics, and conformal prediction.

Scope: The paper introduces a post hoc, model-agnostic conformal-prediction framework that produces statistically valid prediction intervals for the reconstruction, latent-dynamics, and end-to-end components of a latent-space ROM, demonstrated on an autoencoder–SINDy surrogate for warm-rain droplet-size distributions.

What This Paper Is About

Reduced-order models can simulate high-dimensional physical systems cheaply, but practitioners have had no unified way to say how much to trust their predictions. Existing UQ methods for these models are usually tied to specific architectures, require expensive retraining, or assume a parametric form for the errors. This paper builds a framework that bolts onto any already-trained latent-space ROM and delivers calibrated prediction intervals for each part of the pipeline, and it demonstrates that framework on a cloud-microphysics surrogate that predicts how droplet-size distributions evolve during coalescence and rain formation.

Key Contributions

1. A post hoc, model-agnostic UQ framework for latent-space ROMs that requires no modification to the underlying architecture or training procedure, and no parametric assumptions about the data distribution.
2. Component-wise uncertainty estimates covering three distinct stages of the ROM pipeline: reconstruction (encoder–decoder), latent dynamics, and the end-to-end model.
3. Application of conformal prediction to latent-space ROMs, using tailwise (asymmetric) quantiles of signed residuals for the non-negative, often zero-adjacent droplet-size-distribution (DSD) outputs, and a Mahalanobis-distance conformity score with a Ledoit–Wolf shrinkage covariance estimator for the multivariate, correlated latent variables.
4. A systematic comparison of three conformal prediction variants—vanilla, split (60-20-20 train-validation-test), and CV+ with k = 20 folds—on the AE–SINDy cloud microphysics surrogate, including empirical coverage statistics across 90%, 95%, 98%, and 99% nominal levels.

Main Findings

• Coverage is near nominal for reconstruction and end-to-end predictions. For all conformal methods, empirical coverage was generally close to nominal, with mean coverages typically within 1%–2% of the target and standard deviations typically under roughly 4% (aside from the 1 − α = 90% case). For example, end-to-end vanilla coverages were 88.65 ± 3.35 at 90%, 93.73 ± 2.55 at 95%, 96.79 ± 1.96 at 98%, and 97.94 ± 1.54 at 99%.

• CV+ is the most accurate but the most computationally costly. CV+ consistently produced empirical coverages closer to nominal levels in both mean and median, an improvement most apparent in the end-to-end model. The trade-off is that CV+ requires retraining the surrogate model k times (k = 20 in this study), versus once for vanilla or split conformal.

• Latent dynamics behave differently from the other components. Vanilla and split CP stayed near nominal for latent predictions but with larger variability across samples. CV+ produced systematically conservative intervals for latent dynamics: mean coverage around 95% versus a 90% nominal level, mean coverages exceeding 98% at the 95–99% nominal levels, and medians that saturated at 100% for the two highest nominal coverages.

• Uncertainty migrates from small to large droplets over time. End-to-end predictive uncertainty increases with time, and the prediction-interval "peak" at the sub-50 μm cloud-droplet scale contracts while intervals expand markedly at larger rain-droplet scales. This trend held even for a unimodal cloud-droplet population with negligible collisional growth (sample 7); uncertainty in larger rain bins still grew with time. The authors interpret this as the inherent difficulty of predicting rain onset (emergence of a secondary right-hand peak).

• The three components show distinct uncertainty-growth patterns. Reconstruction uncertainty, which characterizes the autoencoder alone, stays consistent across time. Latent-dynamics uncertainty shows rapid incipient growth that slows over time, reflecting the cloud-to-rain transition. The full ROM produces nearly linear growth in end-to-end predictive error.

• Latent errors are "smoothed out" by the autoencoder. The authors identify this as a key advantage of component-wise analysis: despite growing latent error, end-to-end error growth remains linear because propagation is mediated by the reconstruction back to physical space. They conclude that for this ROM, reducing structural uncertainty in the autoencoder would likely improve overall performance more than refining the dynamical model.

• Smaller miscoverage rates destabilize the intervals. Across all three network subsets, predictive errors became less consistent as α → 0, because convergence of empirical quantiles to their true values depends strongly on the miscoverage rate; extreme quantiles need larger calibration sets.

Methodology in Plain English

The authors start with a ROM that already exists: an autoencoder squeezes each 64-bin droplet-size distribution into a 4-dimensional latent state (3 variables from the PSD encoding plus 1 for total liquid mass), and a SINDy module—a bias-free single-layer network using a polynomial library up to second order—describes how that latent state changes in time. They then add uncertainty quantification as a separate, after-the-fact step, leaving the trained model untouched.

The tool they use is conformal prediction. The idea is straightforward: hold out some data the model has not seen, measure how wrong the model is on that data, and use the size of those errors to draw prediction intervals that come with a statistical guarantee—the true value falls inside the interval with probability at least 1 − α, assuming the calibration and test data are exchangeable (the paper argues this holds because the sampled initial conditions are independent). They apply this to each component separately: reconstruction, latent dynamics, and the full pipeline.

Two technical choices matter. Because droplet-size distributions are non-negative and often sit near zero, the authors build asymmetric intervals using the α/2 and 1 − α/2 empirical quantiles of signed residuals, rather than assuming errors are symmetric. Because latent variables are multivariate and correlated, coordinate-wise residuals would miss the joint error structure, so they use a scalar score based on squared Mahalanobis distance, with the residual covariance estimated from the calibration set using a Ledoit–Wolf shrinkage estimator. Intervals are recomputed at each timestep, so uncertainty is tracked in time.

The data come from large eddy simulations of warm-rain coalescence using the superdroplet method. These simulations use 3 × 10⁶ grid cells with 128 particles each—roughly 10⁸ Lagrangian particles per simulation. DSDs are averaged over a (200 m)³ cubic domain, filtered for cloud condensate, and discretized into 64 bins uniformly spaced in ln r. Each grid cell gets a 600 s coalescence-only forward integration from t = 0 to t = 600 s at Δt = 10 s intervals, yielding 61 timesteps. The dataset is split 80% training (494 samples) and 20% testing (124 samples).

Why This Matters

Impact on research. Latent-space ROMs have demonstrated efficiency on complex fluid systems, but convincing practitioners to rely on them has been hampered by the absence of a unified, robust UQ framework. This work supplies one that is distribution-free, requires no retraining or architecture changes, and decomposes uncertainty by pipeline component—letting researchers see which part of a model needs improving rather than treating the whole thing as a black box. The authors state this is a first for latent-space ROMs.

Real-world applications.

• Climate and weather modeling, where error and uncertainty in microphysics parameterizations is typically unquantified but is believed to be a dominant source of uncertainty in future climate projections.
• Precipitation forecasting and the modeling of warm-rain initiation, a process the authors highlight as a persistent challenge.
• Data compression and surrogate design in atmospheric simulation pipelines, where traditional Eulerian binned microphysics still evolves 30–100 prognostic DSD bins per cell with quadratically scaling collisional calculations.
• General scientific surrogate modeling for high-dimensional dynamical systems, since the framework is architecture-agnostic and the companion paper De Jong et al. applies it to two additional latent-space ROMs with different latent dynamical models.

Industry relevance. Any sector that depends on atmospheric simulation—weather services, insurance and reinsurance risk assessment, agriculture, renewable energy siting, and aviation—benefits from cheaper surrogates that come with defensible error bars. The computational economics are the driver: the AE–SINDy surrogate evaluates in O(1) per grid cell per timestep and yields reductions of several orders of magnitude in cost relative to the superdroplet method and at least an order of magnitude relative to bin microphysics, while the framework identifies when surrogate uncertainty becomes dominant.

Future Directions

• Improve latent-space coverage stability. The agreement across CP methods and the variability in empirical coverage were notably better for reconstruction and end-to-end predictions than for latent dynamical predictions. The authors attribute this to greater variance in prediction fidelity across times and variables, and a wider range of magnitudes, in latent predictions compared to normalized DSD predictions.
• Increase training data. The authors suggest altering the initial conditions or dynamical driver of the cloud LES to reduce latent-prediction variability, yielding more consistent CP intervals and improving test-set coverage accuracy.
• Adopt adaptive conformal methods. A stated limitation of standard CP is that interval widths are fixed across the dataset for a given output and time—only the scale relative to input variables changes. Adaptive variants exist that adjust widths to reflect varying uncertainty, but the authors do not explore them here.
• Extend beyond conformal prediction. The post hoc approach is not limited to CP and could extend to other interval- and set-valued UQ methods such as parametric prediction intervals, confidence intervals, and Bayesian credible intervals, ideally tested on other ROMs.

Target Audience

Researchers and graduate students working at the intersection of scientific machine learning, reduced-order modeling, and atmospheric science—particularly those building or evaluating surrogate models for high-dimensional physical systems and needing defensible uncertainty estimates. It is also relevant to climate and weather modelers concerned with microphysics parameterization error, and to methodologists interested in conformal prediction applied to latent-space dynamical systems. Readers will get the most from it with prior exposure to autoencoders, latent dynamics models, and the basics of conformal prediction; the detailed AE–SINDy architecture, training, and accuracy are deferred to the companion paper by De Jong et al., and code is available at https://github.com/jonaskat87/UQ_AE-SINDy.

Authors’ abstract

Reduced-order models (ROMs) can efficiently simulate high-dimensional physical systems but lack robust uncertainty quantification methods. Existing approaches are frequently architecture- or training-specific, which limits flexibility and generalization. We introduce a post hoc, model-agnostic framework for predictive uncertainty quantification in latent space ROMs that requires no modification to the underlying architecture or training procedure. Using conformal prediction, our approach estimates statistical prediction intervals for multiple components of the ROM pipeline: latent dynamics, reconstruction, and end-to-end predictions. We demonstrate the method on a latent space dynamical model for cloud microphysics, where it accurately predicts the evolution of droplet-size distributions and quantifies uncertainty across the ROM pipeline.