Can We Predict Before Executing Machine Learning Agents?

AI summary

Overview

• Research area: Autonomous machine learning agents, LLM reasoning, and "world model" style predictive evaluation, published under the Natural Language Processing (cs.CL) category.
• Technical level: Intermediate. Readers need basic familiarity with LLM agents, ML pipelines, and benchmarks such as MLE-bench, but the paper explains its core ideas in accessible terms.
• Scope (one sentence): The paper asks whether large language models can predict which of two machine learning solutions will perform better before either is executed, and then builds an agent around that predictive ability.

What This Paper Is About

Most autonomous machine learning agents follow a "Generate-Execute-Feedback" loop: they write code, actually run it, and refine based on runtime output. This creates an Execution Bottleneck, since physical execution is expensive and slow — the paper cites benchmarks like MLE-Bench where a single run can consume up to 9 hours. The authors formalize a task called Data-centric Solution Preference, in which a model must look at a task, a data analysis report, and two candidate solutions, and then predict which solution is superior — using only reasoning, with no physical execution.

Key Contributions

1. A new task and corpus. The authors define the task of Data-centric Solution Preference and build a corpus of 18,438 pairwise comparisons drawn from real agent trajectories, spanning 26 distinct tasks across 3 domains (CV, NLP, and Data Science).
2. Evidence that LLMs can predict. They show that LLMs exhibit significant predictive capabilities when primed with a Verified Data Analysis Report, with DeepSeek-V3.2-Thinking achieving 61.5% accuracy and robust confidence calibration.
3. An agent that exploits the prediction. They instantiate the framework in ForeAgent, which uses a Predict-then-Verify loop to decouple exploration from execution, expanding the search space by 3.2×, achieving a 6× acceleration in convergence, and surpassing execution-based baselines by +6%.
4. An open-source dataset. They release a large-scale dataset of verified execution trajectories intended as a foundation for training scalable Reward Models to accelerate reinforcement learning rollouts and optimization across agent frameworks.

Main Findings

• LLMs beat both baselines. DeepSeek-V3.2-Thinking achieved 61.5% pairwise accuracy and GPT-5.1 achieved 58.8%, both above random guessing (50.0%) and a complexity-based heuristic (50.8%). The gap over the heuristic is described as greater than 10%.
• Semantic data reports matter more than raw numbers. In the input-modality study, performance rose from the Heuristic Baseline (50.8%) and Code Only (56.7%) to Numerical Stats (59.0%), peaking with Verbal Reports (61.3%). A Context Mismatch control reached 56.8%, only slightly above Code Only, indicating that predictive success depends on strict semantic alignment between code and data.
• Reasoning is the engine, but it has boundaries. Thinking Mode (chain-of-thought) reached 61.3% versus 55.9% for Direct Answering, and remained stable across temperatures in the range [0, 1.5]. Performance stratified sharply by task and solution type: NLP tasks (66.9%) and Easy tasks (63.9%) scored highest, while complex code incurred a "Complexity Tax" (59.6%), and models were best at broad Cross-Algo contrasts (62.8%).
• Global ranking is much weaker than pairwise comparison. In listwise ranking, Accuracy@1 dropped from the pairwise baseline of 61.3% to 31.1%, with Spearman correlation hovering around ρ ≈ 0.23 — evidence the model lacks global discrimination capability beyond binary comparisons.
• Scaling parameters does not help. Across five Qwen checkpoints (4B, 30B, 235B, 480B, and 1T), performance saturated after the initial 30B threshold and plateaued even at the 1T scale, suggesting predictive power comes more from reasoning-centric architecture than raw parameter count.
• Confidence is well calibrated. Self-reported confidence correlated with accuracy, which the authors say makes the confidence value reliable enough to be used as a gating threshold in downstream agent decisions.
• Execution-based validation has its own ceiling. Using internal validation metrics (M_val) as a proxy for test performance yields only 72.2% accuracy. The authors call this the Validation-Test Gap and describe 72.2% as an "Indirect Epistemic Bound" caused by distributional shifts and validation overfitting. Their LLM predictor costs roughly seconds versus hours for execution, at 61.5% accuracy (Table 4: Random Guess 50.0, Exec/M_val ≈ Hours 72.2, LLM ≈ Seconds 61.5).
• The agent speeds up and improves. ForeAgent achieves an average 6× speedup, explores 3.2× more nodes within 1/6 of the time budget, and improves Beat Ratio by +6% over the AIDE baseline, generalizing to unseen tasks. It also boosted the Test Improve Rate by 23%.
• The model can override human complexity bias. In a case study on the Google Quest Challenge, the model correctly favored a simple LightGBM while humans favored a complex Deep Neural Network, detecting small-sample overfitting risks that humans missed.

Methodology in Plain English

The authors built a large comparison dataset from the real execution traces of two existing agents, AIDE and AutoMind, running on MLE-bench with DeepSeek-V3.1 and o3-mini. From 1,329 valid solutions across 26 tasks, an expert-in-the-loop pipeline pruned the raw traces down to 895 high-quality instances through deduplication, taxonomy tagging, and expert sampling. Every pair of solutions was then combined into a comparison, with ambiguous pairs discarded and the ground-truth winner's position balanced to avoid position bias.

Each comparison was augmented with a Verified Data Analysis Report produced by a "Profile-Verify-Verbalize" pipeline: GPT-5.1 writes a Python script to profile the raw data with labels masked; the script runs in a sandbox and a human expert performs a strict pass/fail validity check; then GPT-5.1 translates the raw execution log into a semantic insight (for example, turning a raw "Target Distribution: 0: 0.915, 1: 0.085" into an imbalance warning with an implication about metric choice). The model then receives the task, the report, and two candidate solutions, and must output a reasoning trace, a predicted winner, and a confidence score.

For the agent, the authors kept AIDE as the backbone but replaced its improvement stage with a Predict-then-Verify loop: generate m = 10 candidates in parallel without executing, use a confidence gate of c = 0.7 to decide which pairs to trust, then physically verify only the Top-k (k = 1) candidate. Evaluation used 5 AI4Science tasks from MLE-bench, 12-hour limits, three independent runs per task, and the Beat Ratio metric (the percentage of human leaderboard contestants outperformed), with two of the five tasks marked "Unseen."

Why This Matters

The paper argues that hours of physical execution can potentially be compressed into seconds of logical inference, which changes the economics of autonomous ML research. Instead of treating execution as the only source of truth, agents can use internal prediction as a fast filter and reserve expensive runtime for the candidates that survive.

Real-world applications (drawn from the paper's domains):

• Medical and biological discovery, where the agent was tested on a diagnostic cancer detection task (Histo. Cancer Detect) and a Stanford Covid Vaccine task.
• Physical and Earth sciences, via the Ventilator Pressure (Physics) and Statoil Iceberg (Geoscience) tasks.
• Ecology, through the Aerial Cactus Identification task, showing the method can transfer to unseen scientific problems.
• Reinforcement learning and agent optimization, through the proposed use as a scalable Reward Model that provides dense, execution-free feedback to accelerate RL rollouts.

Industry relevance: MLE-bench is a competitive, leaderboard-style benchmark, so a Beat Ratio improvement is framed as expert-level competitiveness. Any organization running expensive ML training loops — where a failed run costs hours of GPU time — could benefit from pruning candidates before execution. The predictive model also acts as a "semantic safeguard" against overfitting to noisy internal validation metrics.

Future Directions

• Move beyond static prediction toward interactive simulation. The authors argue that current models are fundamentally "World-Blind Learners" because static training corpora pair code with only trivial inputs or abstract descriptions. They suggest future scaling should pivot from static ingestion to interactive simulation with genuine causal feedback loops.
• Integrate symbolic regression. The paper proposes that distilling intricate logic directly from data via symbolic regression could fundamentally resolve the gap between numeric and semantic spaces.
• Extend beyond inference to training as a Reward Model. The authors reserve this for future work, noting that dense execution-free feedback could accelerate reinforcement learning rollouts.
• Improve the agent architecture and deepen data profiling. The current ForeAgent uses a deliberately conservative Predict-then-Verify loop with unexplored hyperparameters, so the reported performance is described as a likely lower bound. The Verified Data Report also currently relies on metadata for unstructured domains like CV and NLP, leaving multimodal data analysis agents as an open direction.

Target Audience

Researchers and engineers working on LLM-based autonomous agents, automated machine learning (AutoML), and agentic ML benchmarks such as MLE-bench will benefit most. The paper is also relevant to practitioners interested in world models, execution-free code evaluation, and confidence-calibrated LLM reasoning, as well as to teams seeking to reduce compute costs in iterative model development pipelines.

Authors’ abstract

Autonomous machine learning agents have revolutionized scientific discovery, yet they remain constrained by a Generate-Execute-Feedback paradigm. Previous approaches suffer from a severe Execution Bottleneck, as hypothesis evaluation relies strictly on expensive physical execution. To bypass these physical constraints, we internalize execution priors to substitute costly runtime checks with instantaneous predictive reasoning, drawing inspiration from World Models. In this work, we formalize the task of Data-centric Solution Preference and construct a comprehensive corpus of 18,438 pairwise comparisons. We demonstrate that LLMs exhibit significant predictive capabilities when primed with a Verified Data Analysis Report, achieving 61.5% accuracy and robust confidence calibration. Finally, we instantiate this framework in FOREAGENT, an agent that employs a Predict-then-Verify loop, achieving a 6x acceleration in convergence while surpassing execution-based baselines by +6%. Our code and dataset are publicly available at https://github.com/zjunlp/predict-before-execute.