Plug-and-Play Benchmarking of Reinforcement Learning Algorithms for Large-Scale Flow Control

AI summary

Overview

Research area: Reinforcement learning for active flow control (AFC); benchmark design for machine learning on partial differential equations.

Technical level: Intermediate. The paper is readable for anyone familiar with basic RL concepts, but full appreciation of the environment design (differentiable CFD, MARL interfaces, performance profiles) benefits from some background in fluid dynamics or simulation-based learning.

Scope: This paper introduces FluidGym, a standalone, fully differentiable, PyTorch-native benchmark suite covering 13 single- and multi-agent flow control tasks in 2D and 3D, together with baseline results for four control algorithms.

What This Paper Is About

Research on reinforcement learning for flow control is fragmented: studies use different sensors, actuators, simulation setups, and evaluation protocols, which makes it nearly impossible to say which algorithm actually works best. Existing benchmarks partially address this but require external CFD solvers (OpenFOAM, Fluent, FEniCS), are not differentiable, and rarely support 3D or multi-agent control. FluidGym is a plug-and-play alternative that puts the entire simulation and control interface into a single Python package, so researchers can pip install it and immediately run standard RL libraries against high-fidelity flow simulations.

Key Contributions

1. A standalone, fully differentiable benchmark. FluidGym runs entirely in PyTorch on top of the GPU-accelerated PICT solver, with no external CFD dependencies and no coupling layers. Every environment supports backpropagation through the simulation, enabling gradient-based control alongside classical RL.

2. A standardized environment collection. Thirteen environments across four flow scenarios (cylinder wake, Rayleigh–Bénard convection, airfoil, turbulent channel flow), spanning three difficulty levels, 2D and 3D geometries, and both single-agent and multi-agent settings with consistent observation, action, and reward definitions.

3. A unified training and evaluation protocol. Predefined train/validation/test splits with ten randomly generated initial domains each, automatically downloaded and cached, plus randomized perturbations on reset to avoid the common pitfall of reusing training initial conditions at test time.

4. An extensive baseline study. More than 25,000 GPU hours of experiments covering PPO, SAC, TD-MPC, and Differentiable Predictive Control (DPC), including transfer-learning evaluations across dimensions and domain sizes, with all environments, datasets, and trained models released publicly.

Main Findings

• SAC consistently beats PPO on single-agent tasks. Across all difficulty levels and environment categories, SAC achieves the highest normalized improvement over the uncontrolled baseflow. PPO's weaker profile is attributed to slower convergence rather than insufficient model capacity (verified by a network-size ablation).

• Multi-agent variants are more evenly matched. MA-PPO and MA-SAC show similar performance profiles, likely because parallel actors increase the effective sample count and erode SAC's usual sample-efficiency advantage. MA-PPO performs slightly better on turbulent channel flow.

• Differentiable control is dramatically sample-efficient. DPC, which learns purely from reward gradients through the differentiable simulation, reaches useful policies on the 2D cylinder task roughly one to two orders of magnitude faster than PPO and SAC. It achieves approximately 7.2% drag reduction versus roughly 8% for SAC.

• DPC remains competitive in harder regimes. On the turbulent 2D Rayleigh–Bénard convection task, DPC matches TD-MPC and SAC and clearly outperforms PPO.

• Environments with similar flow structure behave similarly. Cylinder and airfoil tasks show comparable learning dynamics despite different reward definitions, suggesting flow physics rather than reward shaping drives difficulty.

• Multi-agent policies discover coordinated spatial structure. On 3D Rayleigh–Bénard convection, MA-PPO agents learn to form two stable convection rolls, consistent with prior findings and indicating spatially invariant control strategies emerge from decentralized training.

• Transfer from 2D to 3D is robust. A 2D policy applied independently to the eight 3D actuators outperforms 3D-trained baselines on the easy and hard cylinder tasks and is on par on the medium task.

• Policies transfer across domain sizes. In turbulent channel flow, policies trained on a small domain match an opposition-control baseline and substantially outperform policies trained directly on the large domain, supporting the hypothesis that MARL learns translation-equivariant strategies.

• Gradient-based control costs more wall-clock time. DPC requires 1.5–2x the training time of standard RL due to the extra backward pass, highlighting the trade-off between sample efficiency and compute per update. TD-MPC has the lowest training runtimes among the baselines.

• Runtime varies enormously across environments. Per-step costs range from 0.29 seconds (small turbulent channel flow) to 52.89 seconds (3D airfoil) on a single NVIDIA A100 GPU.

Methodology in Plain English

The core engineering move is to eliminate every layer of glue code that usually sits between an RL agent and a fluid simulation. The authors build their solver (PICT) directly in PyTorch with custom CUDA kernels, then wrap it in a FluidEnv abstraction that exposes the same observation, action, and reward interface that standard RL libraries expect. Because the whole stack is one autograd graph, the same environment that PPO or SAC interacts with can also be differentiated through end-to-end for gradient-based methods like DPC.

Each environment is defined by domain configuration, a control mapping (how raw agent outputs become physical boundary conditions such as heater temperatures or jet velocities after normalization, clipping, and smoothing), and a reward function tied to a physical objective — drag coefficient reduction, Nusselt number reduction, lift-to-drag ratio, or wall shear stress reduction. Difficulty is set by physical parameters: Reynolds number for cylinder and airfoil flows, Rayleigh number for convection.

For evaluation, the authors train each algorithm with five random seeds and test on ten held-out episodes, using standard library hyperparameters to keep comparisons fair and reproducible. Results are aggregated using performance profiles and interquartile mean scores following best practice from the RL benchmarking literature, with stratified bootstrap confidence intervals.

Why This Matters

Impact on research: The field of RL for flow control has suffered from a reproducibility crisis — inconsistent setups, single-seed runs, test episodes reusing training initial conditions, and heavy reliance on PPO despite SAC's known advantages on continuous control. FluidGym removes the infrastructure excuse for these practices by making rigorous, standardized comparison the path of least resistance. Its differentiability also opens a research avenue the field has barely explored: gradient-based and differentiable RL methods that existing benchmarks could not support.

Real-world applications:

• Aerodynamics and transport: Drag reduction on cars and aircraft. The paper notes Europe could save over 20 million tonnes of CO2 annually through AFC-based drag reduction.
• Energy systems: Improving heat transfer in industrial processes, and control of plasma in Tokamak fusion reactors.
• Wind energy: Energy harvesting and turbine efficiency.
• Industrial cooling and thermal management: Rayleigh–Bénard convection environments directly target heat transfer enhancement.

Industry relevance: Companies working on autonomous control of fluid systems — automotive, aerospace, energy, and HVAC — benefit from a lower barrier to entry: no CFD expertise is required to run experiments, and policies pretrained on high-fidelity simulations can be tested for transfer before deployment. The released trained models and datasets also let practitioners skip expensive training runs for common tasks.

Future Directions

• Statistical robustness. The current study uses only five random seeds due to computational cost; scaling up seeds would strengthen algorithm comparisons that currently fall within overlapping confidence intervals.
• Gradient-based methods in 3D. DPC results are currently limited to two 2D environments. Extending them to complex 3D flows would test whether reward gradients remain informative at higher Reynolds and Rayleigh numbers.
• Differentiable RL. Combining gradient-based control with classical RL (via differentiable RL methods) is a natural next step that the benchmark is uniquely positioned to support.
• Beyond incompressible flows. The authors plan to extend FluidGym to magnetohydrodynamic flows for fusion-relevant control, and to keep adding harder environments as control methods advance.
• Hardware accessibility. Removing the CUDA-only requirement would broaden adoption, since CPU-only execution is not currently supported.

Target Audience

Researchers and graduate students working at the intersection of reinforcement learning, control theory, and computational fluid dynamics. It is particularly valuable for those designing new RL or differentiable control algorithms who need a credible, non-toy testbed, and for practitioners in aerospace, energy, or automotive control who want to evaluate learned controllers against high-fidelity simulations without maintaining a CFD toolchain. Benchmark designers in adjacent simulation domains (climate, plasma physics, structural mechanics) may also find the architecture instructive as a template for standalone, differentiable, multi-agent benchmarks.

Authors’ abstract

Reinforcement learning (RL) has shown promising results in active flow control (AFC), yet progress in the field remains difficult to assess as existing studies rely on heterogeneous observation and actuation schemes, numerical setups, and evaluation protocols. Current AFC benchmarks attempt to address these issues but heavily rely on external computational fluid dynamics (CFD) solvers, are not fully differentiable, and provide limited 3D and multi-agent support. To overcome these limitations, we introduce FluidGym, the first standalone, fully differentiable benchmark suite for RL in AFC. Built entirely in PyTorch on top of the GPU-accelerated PICT solver, FluidGym runs in a single Python stack, requires no external CFD software, and provides standardized evaluation protocols. We present baseline results with PPO, SAC, DPC, and TD-MPC, and release all environments, datasets, and trained models as public resources. FluidGym enables systematic comparison of control methods, establishes a scalable foundation for future research in learning-based flow control, and is available at github.com/safe-autonomous-systems/fluidgym.