Cloud4D: Estimating Cloud Properties at a High Spatial and Temporal Resolution

AI summary

Overview

Research area: 3D computer vision applied to atmospheric science — specifically multi-view reconstruction of volumetric cloud properties and wind from ground-based camera arrays.

Technical level: Intermediate to Advanced. The paper assumes familiarity with multi-view geometry (homographies, camera intrinsics/extrinsics), vision transformers, sparse voxel representations, and basic atmospheric variables such as liquid water content (LWC) and liquid water path (LWP).

Scope: Cloud4D is a learning-based pipeline that turns images from six synchronized ground-based cameras into a four-dimensional cloud state — 3D liquid water content plus height-varying horizontal winds — at 25 m spatial and 5 s temporal resolution, validated against radar, satellite, ERA5, and a wind profiler over a two-month deployment.

What This Paper Is About

Weather and climate models operate at kilometer scales, so individual shallow cumulus clouds — which span less than a kilometer, form below 2000 m, and can cover as much as 40% of Earth's surface (Tselioudis et al., 2021) — cannot be resolved directly and must be approximated by hand-crafted parameterizations. Better observations of individual clouds over their full lifetime are needed, but satellites revisit a given location only every few days, while radar and aircraft give detail over only a small portion of a cloud.

Cloud4D aims to close this observational gap by reconstructing a physically consistent, four-dimensional cloud state using only synchronized ground-based cameras, which are cheap and widely deployable.

Key Contributions

1. The first method to jointly estimate cloud physical properties — liquid water content, height, and thickness — at high spatial and temporal resolution from multi-view images.

2. A homography-guided 2D-to-3D transformer model that ingests images and predicts 3D cloud properties on a high-resolution grid, exploiting the fact that clouds form in spatially structured layers by explicitly aligning a homography with cloud layers in world space.

3. A sparse-transformer refinement stage that lifts 2.5D predictions (LWP, cloud base height, cloud geometric thickness) into a full 3D LWC field while preserving the predicted LWP in each height column.

4. A demonstration on a two-month deployment with six cameras showing an order-of-magnitude improvement in temporal resolution over space-borne products, with less than 10% relative error against collocated radar retrievals, plus height-varying horizontal wind profiles derived by tracking the reconstructed clouds over time.

Main Findings

• Accuracy against radar: Over 12 cumulus days (17 hours of camera data), Cloud4D achieves an F1 of 0.70 for cloud occupancy, mean absolute error of 0.03 g m⁻³ for LWC, 0.06 kg m⁻² for LWP, 189.58 m for cloud base height (CBH), and 295.77 m for cloud top height (CTH). The satellite-based baseline VIP-CT (Ronen et al., 2022) scores 0.40, 0.13 g m⁻³, 0.39 kg m⁻², 791.23 m, and 1021.49 m respectively.

• Relative error below 10%: The reported relative error against radar is 8.9%, computed as MAE(LWC of clouds) / mean(LWC of clouds from radar GT) = 0.029 g m⁻³ / 0.321 g m⁻³.

• Resolution gain over satellites: Cloud4D estimates volumetric cloud properties every 5 s on a 25 m grid, whereas the paper states Sentinel-2 takes one image every five days on average and MODIS once per day.

• VIP-CT fails on ground-based imagery: The authors note that VIP-CT is unable to learn cloud property estimation from ground-based cameras, which is not unexpected since it is designed for satellite imagery, relies on unobstructed orthographic views, and implicitly learns multi-view geometry — substantially harder with ground-based views.

• Sparse transformer helps: Adding the sparse transformer to the 2D-CNN-only model improves LWC (0.030 to 0.029 g m⁻³), CBH (202.04 m to 189.58 m), and CTH (299.04 m to 295.77 m), while occupancy (0.70) and LWP (0.063 kg m⁻²) remain unchanged because the transformer preserves the original LWP.

• View count matters for geometry, less for water content: Dropping two camera pairs changes occupancy to 0.76, LWC to 0.038 g m⁻³, LWP to 0.070 kg m⁻², CBH to 316.28 m, and CTH to 371.71 m. Dropping four pairs gives occupancy 0.68, LWC 0.037 g m⁻³, LWP 0.072 kg m⁻², CBH 325.03 m, and CTH 383.16 m — CBH and CTH degrade more than LWC and LWP.

• Wind profiles: Retrieved horizontal wind vectors have similar magnitudes and directions to a collocated wind profiler (Figure 5). Wind retrieval is performed over five-minute buckets with 15 s between tracked frames, giving a coarser temporal resolution than the LWC estimates.

• Performance across cloud conditions: For cloud coverage 0.20–0.45 the scores are occupancy 0.65, LWC 0.034 g m⁻³, LWP 0.077 kg m⁻², CBH 149.36 m, CTH 287.12 m; for coverage 0.75–0.90 they are 0.75, 0.023 g m⁻³, 0.048 kg m⁻², 230.92 m, and 304.67 m. The paper states performance is slightly worse for higher cloud bases and geometrically thicker clouds, but not to a significant extent.

• Qualitative validation on unseen LES: On a BOMEX configuration (Siebesma et al., 2003) unseen during training, predictions capture correct cloud base and top heights and correct plume shapes.

Methodology in Plain English

The system takes images from several cameras whose positions and lenses are known, and asks what the distribution of liquid water in the air above them must have been to produce those images.

The central idea is that clouds sit in roughly flat layers at particular altitudes. For any candidate altitude, the authors use a homography — a standard geometric mapping — to project each camera's image onto a plane at that height, and then average across cameras. When the assumed height matches the real cloud layer, the projected images line up and the averaged result is clean; when it is wrong, the projections disagree and artifacts appear. The model samples 18 such heights, from 400 m to 3800 m in 200 m steps, producing a stack of candidate planes.

Rather than working directly on pixel colors, the model extracts features with DINOv2, spatially upsampled with LoftUp and projected down to 16 channels. Because cloud layers are vertically thin relative to their horizontal extent, the stacked feature planes are flattened into a single 2D feature map with many channels, and a 2D CNN predicts three 2.5D properties: liquid water path, cloud base height, and cloud geometric thickness. Each height plane is conditioned on its sampled height using adaptive layer normalization.

Those 2.5D maps are then lifted into a coarse initial 3D LWC field, distributing the LWP along each column with a linear increase toward cloud top. Empty voxels are discarded, leaving a sparse set of voxels that a sparse transformer refines using backprojected image features and sinusoidal positional encoding. The transformer's output is normalized along height with a softmax and rescaled so the column's original LWP is preserved.

Wind is recovered by slicing the 3D LWC at different heights, summing five consecutive slices to reduce noise, rendering each slice as a grayscale image, and running the off-the-shelf point tracker CoTracker3 over sequences of 20 frames spaced 15 s apart (five minutes total). Occluded points are discarded and only the tracks with the highest 95% pixel displacements are kept, since failed tracks tend to drift slowly over empty space. Wind speed comes from each track's displacement converted to meters per second, with the median taken within each five-minute bucket.

Training data is synthetic: large eddy simulations from MicroHH (ARM, Cabauw, and RICO cases) and photorealistic Terragen volumes, rendered into images via Monte Carlo path tracing in Blender's Cycles engine, using six views with intrinsics and extrinsics matching the real deployment. Each camera contributes 1500 MicroHH images and 1000 Terragen images, 15000 images total. Terragen data is used only for pre-training because it is not physically realistic. Training runs in stages — 20000 steps on Terragen, 40000 steps on MicroHH for the 2D CNN, and 30000 steps for the sparse transformer — taking three days on 4x H100 80GB GPUs.

The real deployment uses three inward-looking stereo camera pairs in a triangle, with baselines from 190 m to 350 m and 5000 m to 8000 m between pairs, covering a 5 km x 5 km area. Cameras are GPS-synchronized to take an image every five seconds and were calibrated with real-time kinematic positioning accurate to within a centimeter, with night-time star images used to optimize rotation and focal length.

Why This Matters

Impact on research. The paper argues that both physics-based and neural weather and climate models inherit biases from coarse reanalysis data, and that the absence of fine-scale cloud observations blocks progress on subgrid phenomena — extreme precipitation, wind gusts, turbulence, and surface irradiance. Cloud4D provides an observational dataset at a resolution regime that radar, aircraft, and satellites cannot simultaneously cover, which could support evaluation of fine-scale models and training of data-driven surrogates.

Real-world applications.

• Improving short-range forecasting of localized convection, wind gusts, and turbulence near the surface.
• Solar energy operations, where cloud timing and coverage directly determine surface irradiance and power output.
• Aviation routing and safety, given the relevance of cloud structure and wind profiles to flight planning.
• Climate model development and evaluation, through better constraints on how shallow cumulus populations evolve and aggregate.

Industry relevance. The reliance on low-cost, widely available cameras rather than radar or satellite infrastructure makes dense, high-frequency cloud monitoring potentially scalable. The paper also releases code, synthetic training data, and a 17-hour real-world benchmark, which lowers the barrier for others to build on the method.

Future Directions

1. Extending to other cloud types. Training focused on cumulus clouds; the authors note other cloud types are also significant atmospheric drivers and are an interesting avenue for future work.

2. Multi-layer retrieval. The method currently handles a single cloud layer, which the authors justify because higher layers are generally occluded from the ground. Handling multiple layers would increase robustness and broaden applications.

3. Handling optically thick and adverse-weather regimes. The paper explicitly lists multilayer and optically thick cloud regimes as future work, noting that ground-based cameras are susceptible to occlusion from rain, fog, and snow.

4. Embedding physical constraints and coupling to simulators. The authors propose embedding radiative-transfer constraints and coupling the retrievals with differentiable simulators to increase physical fidelity.

Target Audience

This paper is most valuable to researchers working at the intersection of computer vision and Earth observation — particularly those interested in multi-view 3D reconstruction, transformer-based volumetric estimation, and vision foundation models applied outside standard graphics benchmarks. It is also relevant to atmospheric scientists and weather/climate modelers seeking high-resolution observational data for evaluating subgrid cloud processes, and to engineers building cost-effective ground-based sensing systems for renewable energy, aviation, or meteorology. Readers without a background in camera geometry or atmospheric variables will find the motivation accessible but the method sections demanding.

Authors’ abstract

There has been great progress in improving numerical weather prediction and climate models using machine learning. However, most global models act at a kilometer-scale, making it challenging to model individual clouds and factors such as extreme precipitation, wind gusts, turbulence, and surface irradiance. Therefore, there is a need to move towards higher-resolution models, which in turn require high-resolution real-world observations that current instruments struggle to obtain. We present Cloud4D, the first learning-based framework that reconstructs a physically consistent, four-dimensional cloud state using only synchronized ground-based cameras. Leveraging a homography-guided 2D-to-3D transformer, Cloud4D infers the full 3D distribution of liquid water content at 25 m spatial and 5 s temporal resolution. By tracking the 3D liquid water content retrievals over time, Cloud4D additionally estimates horizontal wind vectors. Across a two-month deployment comprising six skyward cameras, our system delivers an order-of-magnitude improvement in space-time resolution relative to state-of-the-art satellite measurements, while retaining single-digit relative error ($<10\%$) against collocated radar measurements. Code and data are available on our project page https://cloud4d.jacob-lin.com/.