Benchmarking Machine Learning for Chaotic Three-Body Dynamics in Astrophysics

AI summary

Overview

Research area: Machine learning for physical dynamics — specifically benchmarking ML against Newtonian three-body dynamics in astrophysics, at the chaotic stability boundary of hierarchical triple-star systems.

Technical level: Intermediate. Readers should be comfortable with standard ML benchmarks (classification, regression, time-series forecasting) and basic orbital mechanics vocabulary; the paper's own machinery is largely empirical rather than theoretical.

Scope: A workshop benchmark paper (Sim2Science: ML with Imperfect Scientific Models; arXiv:2611.08634v1 [cs.LG], 05 Nov 2026) that builds a 160,000-system simulated dataset of chaotic hierarchical triples and evaluates classical ML, pretrained time-series foundation models, and astrophysical stability criteria on a graded hierarchy of stability and trajectory prediction tasks.

Authors: Sheng Wan (École Polytechnique, IP Paris), Pavan Vynatheya (CITA, University of Toronto), Duo Xu (CITA, University of Toronto), Li Kevin Wenliang (Google DeepMind), Ye Zhu (LIX, CNRS, École Polytechnique, IP Paris). Code and dataset are public at https://github.com/Simonwan03/ThreeBodyMLBench.

What This Paper Is About

The three-body problem — three masses moving under their own gravity — is fully deterministic but chaotic: nearly identical starting conditions can end with one star ejected or with the system bound forever. Near the critical boundary between stable and unstable configurations, this sensitivity is extreme, making long-term prediction genuinely hard. The paper asks a benchmarking question rather than a modelling one: how well do current ML methods, including pretrained time-series foundation models, actually do across a ladder of prediction tasks derived from the same chaotic dynamics, compared with the analytical criteria astrophysicists already use?

Key Contributions

1. A controlled perturbation-ensemble dataset. 5,000 base triple systems placed near the critical stability boundary (as predicted by the MLP model of Vynatheya et al.), each expanded into 32 variations, for 160,000 individual triple systems integrated with REBOUND using the 15th-order IAS15 integrator. Each trajectory is sampled every 5 outer orbits over 10,000 outer orbits, yielding 2,001 time points. Roughly 40% of the base systems are predicted stable by that MLP model.

2. A hierarchy of tasks spanning difficulty levels. Three stability sub-tasks — prediction from initial physical conditions (Task A), from a short trajectory prefix (Task B), and multi-horizon survival prediction within an ensemble (Task C) — plus a trajectory forecasting task over a 512-step context with a one-shot 128-step forecast and a 512-step open-loop rollout made of four consecutive 128-step blocks.

3. A broad, unified comparison. Domain-specific astrophysical criteria (Mardling–Aarseth and the Vynatheya et al. criterion), task-specific supervised models (XGBoost, MLP, CNN, TCN, attention-based models), pretrained time-series foundation models with fine-tuned variants, and a scale-invariant "orbital15" feature representation shared across tasks.

4. A characterization of where ML succeeds and fails. The paper establishes a clear ordering of task difficulty in chaotic three-body dynamics: early dynamics give strong stability signals, while precise long-horizon trajectory forecasting remains hard.

Main Findings

• Initial conditions carry real but incomplete information. For binary stability classification, learned models beat the analytical criteria: XGBoost reaches 0.805 accuracy and 0.535 F1, versus 0.619 accuracy and 0.441 F1 for the Mardling–Aarseth criterion. The MLP classifier from Vynatheya et al. gives the strongest threshold-independent discrimination, with AUROC 0.909 and AUPRC 0.718. For stable-fraction regression, XGBoost obtains the lowest MSE (0.088) and highest R² (0.444), while the Vynatheya et al. MLP has a slightly lower MAE (0.218 vs. 0.233). Table 1 reports the XGBoost initial-condition reference point as AUROC 0.8525 and fraction MAE 0.2330.

• Early dynamics are highly informative. Using dynamical summary statistics, XGBoost achieves AUROCs of 0.965 and 0.985 with 96- and 512-step prefixes, consistently outperforming models trained on raw sequences. Frozen PANDA and Chronos-2 representations are competitive, with Chronos-2 at AUROC 0.979 and AUPRC 0.955 at 512 steps.

• Pretrained encoders win the harder survival task. On multi-horizon survival prediction, the best Chronos-2 model reaches overall MAE 0.0376 and IBS 0.0416, against 0.310 and 0.132 for the train-mean baseline. Table 1 lists the prefix-based XGBoost result as AUROC 0.9851 and fraction MAE 0.0398.

• The best representation depends on the objective. Physically motivated dynamical summaries excel at binary classification, whereas pretrained sequence representations are particularly effective for multi-horizon survival prediction.

• Zero-shot foundation models are competitive at short-horizon forecasting. PANDA obtains MSE 0.222, close to the supervised iTransformer at 0.212, while Chronos-2 achieves the lowest MAE (0.228) and unit-circle error (0.227) despite no task-specific training.

• Long-horizon rollout fails through error accumulation. Prediction errors grow consistently during the 512-step open-loop rollout across all models, which the authors identify as a key limitation that is especially severe in chaotic systems.

Methodology in Plain English

The authors do not invent a new model. They build a testbed and run a horse race.

They start by choosing initial conditions deliberately close to the knife-edge where a triple system may or may not stay bound, because that is where prediction is hardest and where existing analytical criteria are least reliable. To measure how sensitive each configuration is, they do not simulate one system per set of initial conditions — they simulate 32 near-identical siblings that differ only in orbital phases or slight semi-major-axis perturbations. That group of 32 is treated as the fundamental unit, so the data split is by ensemble rather than by individual trajectory; otherwise the 32 near-copies would leak across train and test.

They integrate everything with a high-order gravitational N-body code, record each trajectory at 2,001 time points, and convert the raw positions and velocities into 15 dimensionless orbital descriptors (semi-major-axis ratio, mass ratios, eccentricities, inclinations, arguments of periapsis, longitudes of ascending node). This keeps the problem focused on orbital dynamics rather than arbitrary coordinate choices, and gives every model family the same input representation.

They then pose graded questions. Given only the starting conditions, will this system survive to a distant horizon? Given the first 96 or 512 steps of a trajectory, will this particular system survive? Given an entire family of 32 short prefixes, what fraction of them will still be bound at each of several future horizons (framed as conditional hazards so the predicted survival curve is monotonic)? And, hardest of all, can a model forecast the actual future orbital state, both in a single 128-step shot and when its own predictions are fed back to it four times over?

Finally they score all of this against the criteria working astrophysicists already use, and against simple baselines such as predicting the training mean, to see how much the ML methods actually add.

Why This Matters

The paper's value is diagnostic. It shows that "ML for physics" is not one difficulty level: a benchmark that only asked "stable or not?" would suggest the problem is essentially solved, while a benchmark that asked for a faithful long-horizon trajectory would show it is not. That distinction matters for how results in physics-ML are reported and compared, and the authors frame the three-body problem as a simple, well-controlled stress test that complements existing physics benchmarks such as PDEBench, LagrangeBench, The Well, Physion, Physics-IQ and WorldModelBench.

Stated or directly implied applications:

• Rapid screening of triple systems. Task A is explicitly motivated as a way to predict stability from initial conditions without performing a long-term numerical integration.
• Replacing or accelerating numerical integration. The forecasting task is framed as a test of whether pretrained time-series models can act as data-driven predictors of chaotic dynamics, which is conventionally studied through numerical integration.
• Understanding stellar system evolution. The authors state that long-term stability and orbital evolution of triples are relevant to a broad range of astrophysical phenomena.
• Choosing representations for chaotic time series. The finding that summary statistics beat raw sequences for classification, while pretrained sequence encoders win on survival prediction, is transferable guidance for other chaotic forecasting problems.

Industry relevance: Most directly for time-series foundation model developers (Google DeepMind's Li Kevin Wenliang is a co-author) who need hard, non-language, non-financial testbeds where pretrained representations can be stress-tested and where failure modes like rollout error accumulation are visible. Secondarily for scientific software and simulation groups evaluating surrogate models where chaotic amplification limits achievable accuracy. The paper does not discuss any commercial deployment.

Future Directions

• Taming rollout error accumulation. All evaluated models degrade over the 512-step open-loop rollout; whether architecture, training objective, or physics-informed constraints can slow this in chaotic regimes is left open.
• Statistical rigor. The authors state in the checklist that they report point estimates on a fixed train–validation–test split without error bars, confidence intervals, or significance tests, and plan to quantify variability across repeated runs in an extended version.
• Which representation for which target. The observed split — physical summaries for classification, pretrained encoders for survival curves — invites a systematic study of representation choice as a function of task and horizon.
• Adapting foundation models to chaotic dynamics. The competitive zero-shot forecasting results, and the fine-tuned variants the authors already evaluate, suggest further task-specific adaptation of pretrained time-series models as a route to better long-horizon prediction.

Target Audience

Machine learning researchers working on physics-informed models, world models, and time-series foundation models who want a hard, well-controlled chaotic benchmark rather than another smooth dynamical system. Also astrophysicists studying hierarchical triple stability who want to know whether learned models offer anything beyond Mardling–Aarseth-style criteria near the stability boundary. The paper is less suited to readers seeking new architectures or theoretical results — the authors explicitly state there are no new theoretical contributions — but is well matched to benchmark designers and to anyone who needs to argue that a physics-ML result must specify its prediction horizon.

Authors’ abstract

Machine learning (ML) and generative AI have recently made impressive progress in understanding physics. We explore a hierarchy of prediction problems stemming from chaotic three-body systems and benchmark existing ML approaches, including general-purpose prediction algorithms and those that build on top of foundational physics models. We first simulate the trajectories of triple systems near the critical boundary of stable/unstable initial conditions. We then design a hierarchy of learning tasks, focusing mainly on two broad problems: stability prediction and trajectory prediction. The former classifies whether a given triple system has diverged or not up to a specific horizon, while the latter predicts the exact trajectory, both given suitable input features. We benchmark a range of ML approaches, from simple regression methods to state-of-the-art foundational physics models, across these tasks, and compare them against astrophysical methods such as the Mardling-Aarseth stability criterion. Our experiments reveal a clear order of task difficulties in chaotic three-body dynamics: early dynamics enable strong stability prediction, while precise long-horizon trajectory forecasting remains difficult. Code and dataset are publicly available at https://github.com/Simonwan03/ThreeBodyMLBench.