Decision-oriented benchmarking to transform AI weather forecast access: Application to the Indian monsoon

AI summary

Overview

• Research area: AI weather prediction (AIWP), forecast verification/benchmarking, climate adaptation, and development economics, applied to Indian monsoon onset forecasting.
• Technical level: Intermediate. The paper combines machine learning model evaluation with meteorological indices and decision-science framing; no deep mathematical background is required, but familiarity with forecast verification metrics (MAE, Brier skill score, AUC, reliability) helps.
• Scope in one sentence: The paper proposes a "decision-oriented" benchmarking framework that evaluates six open AI weather models and one physics-based NWP model on an agriculturally meaningful local monsoon-onset index over India, and shows how that framework informed the delivery of AI-based forecasts to 38 million farmers in 2025.

What This Paper Is About

Most benchmarks for AI weather prediction models score them on aggregated meteorological quantities (such as 500 hPa geopotential height, often in extratropical regions) rather than on what a specific user population actually needs to decide. The authors argue this leaves a gap: agencies choosing a forecast system for farmers, heat warnings, or disaster evacuation cannot tell from existing benchmarks whether a model will help. Using the 150-year-old problem of Indian monsoon onset forecasting as a test case, they build an operational, decision-oriented benchmark around a local rainfall-based onset index relevant to rain-fed agriculture, evaluate AIWP and NWP models against Indian rain-gauge observations, and report how the results fed into a real 2025 dissemination effort to roughly 38 million farmers.

Key Contributions

1. A decision-oriented, operational benchmarking framework that connects meteorology, AI, and social sciences: it defines a stakeholder-relevant event index, a ground-truth observational dataset, a meaningful baseline, and a suite of metrics tied to real decisions (including miss rate and false alarm rate, not just average error).
2. A locally defined, agriculturally targeted monsoon onset index — a modification of a widely used agronomic index — that is compatible with operational lead times and varies spatially, unlike continental-scale wind-based indices or universal rainfall thresholds.
3. A systematic evaluation of six state-of-the-art global AIWP models plus one NWP model (ECMWF's IFS) on that index, using deterministic and probabilistic metrics, across multiple periods including hindcasts produced as far back as 1965 to work around the small number of out-of-sample onset events.
4. Demonstration of real-world impact: the framework informed model selection for a project led by India's Ministry of Agriculture and Farmers Welfare, in which two AIWP models (AIFS and NGCM) were blended into calibrated probabilistic onset forecasts disseminated to approximately 38 million farmers in 2025.

Main Findings

• Skill at medium range: Of the six AIWP models and one NWP model evaluated for 2019–2024, nearly all provided skillful deterministic forecasts of the onset index at 1–15 day lead times. Within the Core Monsoon Zone (CMZ), most models' error was around two days smaller than the climatological forecast at those lead times, and most outperformed climatology in MAE and miss rate while having comparable false alarm rate.
• Skill fades at longer leads: At 16–30 day lead times, most models' deterministic accuracy approached or fell below the climatological baseline, though IFS and GenCast retained skill out to about three weeks of lead time by all three deterministic metrics.
• No single winner: No single model consistently outperformed all others across all metrics and regions, which the authors say suggests a potential benefit from blending multiple models operationally.
• Large-scale circulation is easier to predict: Using a large-scale wind-based onset index (the Webster–Yang index), all models outperformed the baseline at a 15-day forecast, with most having average error 3–4 days smaller than baseline at 1–15 day lead times, and positive skill often preserved at 16–30 days.
• Small test samples can be addressed: Because one onset occurs per year, out-of-sample testing would normally leave at most about six onset events given AIWP training periods; the authors generated hindcasts for 1965–1978 for all open-source AIWP models, showing that national agencies can expand hindcast sets using their own historical data.
• The baseline matters: Model skill appeared higher for the extended period (1965–1978 plus 2019–2024) largely because the climatological baseline performed worse there, with baseline MAE increasing by about a day.
• Model-specific weaknesses: In the extended period, GraphCast was the only model with MAE and false alarm rate worse than baseline for all periods, and FuXi and FuXi-S2S suffered from total misses (no onset forecast in a summer) despite competitiveness on other metrics. GenCast stood out in the extended period, but its high computational cost and long runtime limited its inclusion elsewhere.
• Probabilistic skill: The IFS ensemble and two of the three probabilistic AIWP models (GenCast and NGCM) had skillful 15-day forecasts on all three probabilistic metrics examined. IFS and NGCM had positive Brier Skill Scores aggregated over 1–30 days; IFS had the higher BSS but a lower AUC than both NGCM and the baseline. FuXi-S2S had a negative BSS for its 15-day forecasts, contrasting with its skillful deterministic metrics, and had only three years of out-of-sample test data available.
• Overconfidence: Reliability diagrams showed the models were generally overconfident, with predicted probabilities higher than observed frequencies for most forecasts.
• Regional asymmetry: Probabilistic skill was generally higher in the CMZ and areas immediately south and west, and lower north and east of the CMZ — in contrast to higher deterministic skill in those northern and eastern regions.
• 2025 case study: In 2025 a very early onset in southern India was followed by over two weeks of dry conditions and a pause in the northward progression of onset. AIFS and NGCM both captured the overall anomalous spatio-temporal pattern in 35-day forecasts initialized on 20 May 2025, including the late-May heavy rainfall in southern India and the subsequent roughly 3-week dry period over central India (28 May–17 June), and both predicted the Somali jet evolution with 3–4 weeks of lead time.
• Real-time initialization is viable: Forecasts initialized with real-time analyses instead of ERA5 showed no substantial qualitative differences, although detailed analyses revealed some differences.

Methodology in Plain English

The authors first decide what a farmer actually needs to know: when continuous rains will begin locally, since planting too early risks crop failure. They adapt a standard agronomic definition of local onset — the start of the first 5-day rainy period typical for that location's wet season — and modify its long dry-period safeguard so it works within operational forecast lead times, anchoring it to the India Meteorological Department's median climatological monsoon onset date over Kerala.

They then pick models that are openly available (codes and weights public, except FuXi-S2S, for which only shared hindcasts were available), that output precipitation, and that could plausibly be run operationally. The six AIWP models are AIFS, FuXi, FuXi-S2S, GraphCast, GenCast, and NeuralGCM (NGCM); the comparison NWP model is ECMWF's IFS. They generate or retrieve 35-day forecasts, initialized twice weekly from early May, so that forecasts face realistic opportunities for false alarms and misses.

Ground truth comes from IMD rain-gauge observations rather than ERA5, which was used for model training. Skill is measured against a simple climatological baseline — historical average onset dates in 4° × 4° grid cells. Metrics include mean absolute error (MAE), miss rate (MR), false alarm rate (FAR), the Brier skill score, ranked probability score/skill score, area under the ROC curve, and reliability diagrams. To cope with the tiny sample of onset events, they evaluate four periods, including hindcasts back to 1965 and a common 2004–2021 period, explicitly tracking which years overlap with model training.

Why This Matters

• Research impact: The paper argues that existing AIWP benchmarks measure scientific validity but not operational usefulness, and shows a template for closing that gap. It also highlights that a model can have low average error while having high miss or false-alarm rates — a distinction that matters for decisions and could motivate changes to model architecture, loss functions, or post-processing.
• Real-world applications:
  • Agricultural advisories telling farmers when to plant in rain-fed systems, where early planting after a false onset risks total crop loss.
  • National meteorological agencies in low- and middle-income countries benchmarking AIWP models with their own historical data, expanding hindcast samples for rare events.
  • Government dissemination programs: the framework directly informed model choice for India's Ministry of Agriculture and Farmers Welfare effort reaching about 38 million farmers in 2025.
  • Public health and disaster management use cases the authors mention, such as heat warning systems and setting risk thresholds to activate evacuations.
• Industry relevance: Open-source AIWP models can generate multi-model ensembles at far lower computational cost than physics-based NWP, potentially letting institutions without supercomputers produce tailored, calibrated forecast products. The paper also stresses that releasing model weights publicly is what makes robust benchmarking possible — a point illustrated by the FuXi-S2S case.

Future Directions

• Improving physical representation: The authors note that monsoon onset is influenced by incompletely understood couplings with remote ocean temperatures, distant snow and ice cover, and land surface processes, which current models represent inadequately or not at all — a clear target for AIWP development.
• Calibration and post-processing: Since all ensemble models were overconfident, the paper points toward probability calibration schemes, ideally applied within multi-model ensembles, as a needed step before dissemination.
• Designing dissemination itself: Moving from benchmarking to at-scale delivery requires messaging systems and message content that appropriately guide decisions in local contexts, alongside bias correction.
• Extending the framework: The authors frame this as one component of a broader blueprint for using AIWP to help large vulnerable populations adapt, suggesting the approach could be adapted to other regions, hazards, and decision contexts.

Target Audience

This paper is most useful to researchers and practitioners working at the intersection of AI weather prediction and societal benefit: meteorologists and climate scientists evaluating forecast models, AI researchers building or benchmarking weather models, development economists and policy analysts designing forecast dissemination programs, and staff at national meteorological and agricultural agencies in low- and middle-income countries who must choose which models to operationalize. It is also relevant to funders and program designers interested in how open AI models could expand access to high-quality forecasts.

Authors’ abstract

Artificial intelligence weather prediction (AIWP) models now often outperform traditional physics-based models on common metrics while requiring orders-of-magnitude less computing resources and time. Open-access AIWP models thus hold promise as transformational tools for helping low- and middle-income populations make decisions in the face of high-impact weather shocks. Yet, current approaches to evaluating AIWP models focus mainly on aggregated meteorological metrics without considering local stakeholders' needs in decision-oriented, operational frameworks. Here, we introduce such a framework that connects meteorology, AI, and social sciences. As an example, we apply it to the 150-year-old problem of Indian monsoon forecasting, focusing on benefits to rain-fed agriculture, which is highly susceptible to climate change. AIWP models skillfully predict an agriculturally relevant onset index at regional scales weeks in advance when evaluated out-of-sample using deterministic and probabilistic metrics. This framework informed a government-led effort in 2025 to send 38 million Indian farmers AI-based monsoon onset forecasts, which captured an unusual weeks-long pause in monsoon progression. This decision-oriented benchmarking framework provides a key component of a blueprint for harnessing the power of AIWP models to help large vulnerable populations adapt to weather shocks in the face of climate variability and change.