CloudCons: A Comprehensive End-to-End Benchmark for Cloud Resource Consolidation

AI summary

Overview

• Research area: Cloud computing / AIOps — time series forecasting and its downstream use in cloud resource consolidation (VM-to-physical-machine placement).
• Technical level: Intermediate. Readers need basic familiarity with time series forecasting metrics (MASE, CRPS) and cloud scheduling concepts, but the decision-centric framing is explained from first principles.
• Scope: A single paper introducing CloudCons, an end-to-end benchmark that evaluates forecasting models — statistical, deep learning, and time series foundation models — by their downstream consolidation decisions rather than by prediction error alone, across datasets from Huawei Cloud, Microsoft Azure, and Google Borg.

What This Paper Is About

Cloud data centers are deliberately over-provisioned to protect service reliability, which keeps global average CPU utilization at low levels of 15%-20%. The "forecast-then-optimize" paradigm attacks this by predicting future demand, then packing virtual machines onto fewer physical machines. New time series foundation models promise zero-shot generalization, but existing cloud benchmarks score them only on prediction error, so nobody knows whether their better forecasts actually produce better placement decisions. CloudCons is built to answer exactly that question end to end.

Key Contributions

1. Multi-cloud datasets and an end-to-end framework. The authors apply a standardized processing pipeline to real workload traces from Huawei Cloud, Microsoft Azure, and Google Borg, producing four datasets spanning 5-minute, 30-minute, and 1-hour granularities, and wrap them in a framework that simulates the full forecast-then-optimize workflow rather than forecasting alone.
2. Decision-centric benchmarking of foundation models. Instead of stopping at prediction error, the benchmark measures the practical utility of forecasts in driving consolidation decisions, establishing a new decision-centric baseline.
3. A five-dimension evaluation suite. Assessment spans prediction error, resource efficiency, load balance, service reliability, and uncertainty quantification — bridging the gap between forecasting accuracy and decision utility.
4. Experimental analysis and actionable guidance. The paper reports the finding that superior forecasting accuracy does not inherently translate into better decision utility, and analyzes predictive quantile selection as a lever for trading off resource efficiency against service reliability.

Main Findings

• Forecasting accuracy favors foundation models in complex scenarios. Chronos 2 records the lowest MASE in Table 3 on Huawei2025 (0.752), Azure2019 (0.690), and Borg2019-d (0.702), while TimesFM 2.5 records the lowest MASE on Borg2019-e (0.846). Even in a zero-shot setting, foundation models outperform statistical baselines and deep learning models trained from scratch on each dataset.
• The accuracy gap narrows on regular workloads. On Borg2019-e, characterized by strong seasonality and workload homogeneity, PatchTST achieves forecasting accuracy approaching the leading foundation models, suggesting traditional architectures remain viable for stable, periodic workloads.
• Forecasting accuracy does not equal decision utility. Despite predictive superiority, foundation models show no advantage over baselines in violation rates at comparable utilization levels. On Borg2019-e, most models — from AutoETS and DLinear to Moirai 2 — converge to an identical utilization rate of 0.637, yet reliability metrics diverge significantly, with AutoETS and DeepAR achieving both lower violation rate and lower violation severity than foundation models.
• Baselines can capture peaks better despite higher overall error. The Appendix C case study finds that Moirai 2 and Sundial underestimate aggregated peaks and mislead the optimizer into risky decisions, while DeepAR — with higher overall prediction error — shows superior fidelity in capturing peak loads and avoids contention.
• Huawei2025 consolidates easily; Azure2019 and Borg2019-d do not. On Huawei2025, workload heterogeneity supports efficient consolidation, with utilization rates exceeding 95% across all models and low PAR. On Azure2019, pulse-like workloads cause PatchTST and DeepAR to underestimate future peaks, leading the scheduler to over-pack VMs and trigger severe resource contention, while FlowState-9.1M balances best by minimizing PAR (1.063) and achieving the highest utilization (0.973). On Borg2019-d, low-utilization tasks and high-frequency noise cause a significant decline in utilization alongside persistently high violation rates.
• PatchTST trades efficiency for reliability on Huawei2025. It exhibits a conservative provisioning tendency, giving up marginal resource utilization for a twofold improvement in service reliability (violation rate 0.096).
• Optimization algorithms carry distinct strategic biases. Gurobi (MILP) consistently dominates in resource efficiency and load balancing but removes safety buffers, causing severe reliability degradation in bursty environments like Azure2019; heuristics such as BFD sacrifice efficiency for robustness, with resource fragmentation acting as an implicit safety margin; ACO ranks second to Gurobi in low-load scenarios such as Borg2019.
• Quantile selection is a practical control lever. Analysis of predictive quantiles (α) on Util and VR shows how calibrating quantile selection can balance resource efficiency against service reliability, with configurations marked as achieving the reliability threshold VR < 0.01.
• Uncertainty quantification matters at the PM level. Because VM aggregation introduces error superposition, the authors model PM-level predictive uncertainty via Monte Carlo simulation and evaluate it with PICP, MPIW, and Winkler Score, using a prediction interval with α set to 0.05.

Methodology in Plain English

The authors gathered publicly available production traces from three cloud providers and cleaned them with a single pipeline: they dropped traces shorter than 7 days, traces with missing values exceeding 0.125%, and traces with zero variance, then filled remaining gaps by interpolation and aggregated everything into 5-minute, 30-minute, and 1-hour granularities. The resulting four datasets are Huawei2025 (174 series, 3,352,860 points, 2 targets), Azure2019 (10,800 series, 104,516,835 points, 1 target), Borg2019-d (414 series, 3,906,696 points, 2 targets), and Borg2019-e (618 series, 5,901,510 points, 2 targets).

They then built a two-stage pipeline. In Stage I, a forecasting model reads a look-back window of CPU usage and predicts a future demand sequence. In Stage II, those predictions feed a dynamic bin packing optimizer that assigns VMs to physical machines with the objective of minimizing the number of active machines, subject to capacity constraints at every time step and a one-VM-one-PM assignment constraint. Placement is held static across the forecast horizon to limit migration overhead.

Forecasting contestants span three families: statistical (AutoARIMA, AutoETS, AutoTheta), deep learning (DeepAR, TFT, DLinear, PatchTST), and foundation models (Moirai 2, Chronos 2, TimesFM 2.5, Sundial, TOTO, FlowState-9.1M, Kairos 50M). Foundation models run zero-shot, deep learning models are trained from scratch per dataset, and statistical models fit parameters to each historical window. Optimization contestants are FFD and BFD (heuristics), ACO (meta-heuristic), and Gurobi (exact MILP solver, providing the performance upper bound). Reinforcement learning is deliberately excluded to keep the focus on the decoupled forecast-then-optimize paradigm.

Evaluation covers five dimensions: MASE and CRPS for prediction error; utility ratio for resource efficiency; peak-to-average ratio for load balance; violation rate and violation severity for service reliability; and PICP, MPIW, and Winkler Score for PM-level uncertainty, obtained by Monte Carlo sampling from per-VM predictive distributions and summing samples. Forecasting is assessed with a non-overlapping rolling window, while downstream utility uses a sliding window with stride 1.

Why This Matters

• Research impact. The paper reframes foundation-model evaluation in cloud operations from "how accurate is the forecast" to "does the decision get better," and demonstrates empirically that the two can diverge. It gives subsequent work a standardized, multi-source, decision-centric testbed, and shows that ranking models on prediction error alone can yield skewed conclusions — one reason the authors report metrics averaged across all optimization methods rather than any single algorithm.
• Cloud capacity planning: choosing how aggressively to consolidate physical machines based on calibrated quantiles rather than point forecasts, with the reliability threshold VR < 0.01 as a concrete target.
• Autoscaling and node provisioning: the heuristics benchmarked here (FFD, BFD) are described as the core packing logic in systems like Karpenter and Azure Node Autoprovisioning, so the trade-offs reported map directly onto production provisioning behavior.
• SLA risk management: the uncertainty quantification suite indicates when to trust the safety margins implied by a model, which matters when aggregated VM errors on a single PM superpose.
• Model selection for AIOps teams: the results support pragmatic choices — for highly regularized workloads such as Borg2019-e, traditional cost-efficient models can be more useful than large foundation models.
• Industry relevance. The evaluation spans three major production cloud providers and includes an observability-specialized foundation model (TOTO) trained on telemetry and internal monitoring data from Datadog, making the findings relevant to real operators rather than only to academic forecasting leaderboards.

Future Directions

• Systematic quantile calibration procedures. The paper presents quantile selection as a critical lever and provides guidelines, but the open question is how to choose quantiles automatically per workload class rather than by manual analysis.
• Closing the accuracy-to-utility gap. Since higher MASE/CRPS does not guarantee fewer violations, the field needs training objectives or model selection criteria aligned with downstream decision quality — for instance, peak fidelity rather than aggregate error.
• Extending beyond the decoupled pipeline. Reinforcement learning is explicitly excluded from this benchmark; whether end-to-end learned policies resolve the forecast-decision misalignment remains untested here.
• Broadening the resource model and data coverage. The benchmark uses CPU load as the representative demand metric and four datasets from three providers; whether the conclusions hold for other resource dimensions, other providers, and finer or coarser aggregation granularities is not established.

Target Audience

Researchers working on time series foundation models who need to know whether zero-shot accuracy buys anything downstream; cloud platform and infrastructure engineers responsible for VM placement, autoscaling, and consolidation heuristics; AIOps practitioners selecting between statistical, deep learning, and foundation models for production telemetry; and benchmark or evaluation researchers interested in decision-centric rather than error-centric assessment design.

Authors’ abstract

Driven by conservative over-provisioning to guarantee service reliability, resource utilization in cloud data centers remains at low levels. To mitigate this, the forecast-then-optimize paradigm has emerged to optimize consolidation by anticipating future demands. While emerging time series foundation models promise to enhance this paradigm through zero-shot generalization, existing benchmarks focus solely on prediction error metrics. The actual decision utility of these advanced models remains unverified, rendering their practical value for downstream tasks uncertain. To bridge this gap, we propose CloudCons, a comprehensive end-to-end benchmark designed to evaluate forecasting models within the specific context of cloud resource consolidation. We build high-quality datasets that cover diverse workloads from Huawei Cloud, Microsoft Azure, and Google Borg, capturing distinct service characteristics ranging from synchronized diurnal rhythms to stochastic, pulse-like bursts and high-frequency noise. We conduct an extensive evaluation of statistical, deep learning, and foundation models. Our experiments reveal a pivotal finding: while foundation models demonstrate superior zero-shot forecasting accuracy, this advantage does not inherently translate into better decision utility. Of practical significance, we systematically analyze how the selection of predictive quantiles acts as a critical lever. We provide actionable guidelines for calibrating these selections to balance the trade-off between resource efficiency and service reliability, offering vital insights for real-world deployment decisions.