Training Numerical Intelligence via Auto-Diagnosis and Skill Discovery

AI summary

Overview

• Research area: Automated scientific computing / AI agents for numerical algorithm improvement (numerical intelligence).
• Technical level: Advanced.
• Scope: The paper introduces ADSD (Auto-Diagnosis and Skill Discovery), a framework that lets language-model agents diagnose why a numerical solver performs poorly and then discover, verify, and package literature-grounded numerical methods as reusable "solver skills," evaluated on four numerical domains across three base models.

What This Paper Is About

AI agents can now write scientific code fluently, but writing code that runs is not the same as writing a solver that is accurate, robust, and fast. Execution feedback tells an agent that something is wrong but rarely explains the underlying numerical cause — for example ill-conditioning, stiffness, or repeated matrix factorizations — and different causes can produce the same visible symptom. ADSD addresses this gap by forcing a diagnosis-first loop: observe the frozen baseline solver's behavior, verify a numerical condition that explains it, retrieve the matching numerical method from the literature, and implement it as a reusable solver skill.

Key Contributions

1. A diagnosis-first paradigm for numerical intelligence. ADSD separates what current workflows blur together: (i) observable behavior (residual histories, convergence logs), (ii) diagnostic knowledge (testable links between symptoms and numerical conditions), and (iii) executable solver knowledge (concrete algorithms correctly implemented and packaged for integration).
2. A concrete algorithmic pipeline. Auto-diagnosis (Algorithm 1) builds diagnostic skills from a literature database, proposes hypotheses per diagnosis round, and verifies them on a held-out verification pool, keeping hypotheses that pass over 90% of it. Skill discovery then converts verified hypotheses into executable, frozen solver skills.
3. A weight-free, general self-improvement framework. ADSD updates no model weights; it improves solvers by supplying the original baseline solver N_0 plus skill packets to a fresh agent under the same model, reasoning budget, runtime, and prompt, producing improved solver B.
4. Consistent gains across four numerical domains and three model families. Power flow, AC optimal power flow, stiff ODE integration, and heterogeneous diffusion PDEs, using GPT-5.5, Claude-Sonnet-5, and Qwen3.8-2.4T, including transfer to unseen topologies and harder feasibility regimes.

Main Findings

• Large power-flow error reduction. On GOC-500 power flow, ADSD reduces Qwen3.8-2.4T's mean solver error by nearly 71×, from 1.0 × 10^-9 ± 2.1 × 10^-9 to 1.4 × 10^-11 ± 2.4 × 10^-10, while maintaining comparable solver speed. The most significant gain across base models was observed for the Qwen3.8 baseline on GOC-500.
• In-domain improvements across all three models on IEEE-118 and GOC-500. GPT-5.5 on IEEE-118 goes from 9.1 × 10^-10 ± 2.0 × 10^-9 to 2.7 × 10^-10 ± 9.9 × 10^-10; on GOC-500 from 9.3 × 10^-10 ± 2.2 × 10^-9 to 1.6 × 10^-10 ± 7.4 × 10^-10. Claude-Sonnet-5 on IEEE-118 improves from 1.1 × 10^-9 ± 2.1 × 10^-9 to 7.4 × 10^-10 ± 1.8 × 10^-9; on GOC-500 from 7.9 × 10^-10 ± 1.9 × 10^-9 to 1.4 × 10^-10 ± 8.4 × 10^-10. Qwen3.8-2.4T on IEEE-118 goes from 9.9 × 10^-10 ± 2.1 × 10^-9 to 6.5 × 10^-10 ± 1.6 × 10^-9. Runtime is comparable or slightly lower in each case.
• Skill transfer to unseen grid topologies. Skills learned only from N2 topology cases (grids with two fewer components) generalize without further adaptation to N1 and full N cases, with significant improvement across all base models and grid benchmarks. Skills can be transferred to grid setups outside the feasible regime as well.
• Transfer to close-to-infeasible cases with limits acknowledged. Skills learned only from feasible (F) cases still improve close-to-infeasible (IF) solvers, though the paper notes the room for improvement is naturally more limited because grid constraints must be satisfied near the voltage-stability boundary.
• Pass-rate gains on AC-OPF edge cases. On 5,000 AC-OPF hidden evaluation cases drawn from near-maximum-loadability IEEE-57 instances, Qwen3.8-2.4T rises from a 14.42% pass rate to 98.36% (the paper describes this as roughly an 84% pass-rate improvement), with mean error falling from 2.7 × 10^-6 ± 8.6 × 10^-7 to 1.2 × 10^-6 ± 8.2 × 10^-7 and runtime from 0.26 ± 0.04 to 0.18 ± 0.05. GPT-5.5 moves from 97.82% to 99.26% pass rate, and Claude-Sonnet-5 from 78.78% to 81.42% with runtime dropping from 0.62 ± 0.11 to 0.42 ± 0.09.
• ODE integrators become faster and more accurate. On held-out stiff-ODE cases, GPT-5.5 goes from a 70.1% pass rate and median error 9.39 × 10^-6 to 90.6% and 3.93 × 10^-7, with per-case time dropping from 4.713 ± 9.570 to 3.451 ± 6.743. Claude-Sonnet-5 reaches a 100.0% pass rate (from 94.0%) with median error 5.04 × 10^-8 (from 4.43 × 10^-7). Qwen3.8-2.4T improves from 98.3% to 99.2% pass rate and from 1.30 × 10^-8 to 5.95 × 10^-9 median error.
• PDE error and operator cost drop. On 2,400 held-out ConDiff cases, GPT-5.5's ℓ2 error falls from 1.69 × 10^-9 ± 0.86 × 10^-9 to 5.27 × 10^-10 ± 2.04 × 10^-10, wall time from 7.29 ± 1.32 to 2.78 ± 0.58, and operator calls from 215.49 ± 40.35 to 71.22 ± 17.58. Qwen3.8-2.4T's operator calls fall from 86.81 ± 29.68 to 17.57 ± 10.03. Claude-Sonnet-5 improves more modestly, from 1.12 × 10^-11 ± 0.54 × 10^-11 to 8.15 × 10^-12 ± 5.41 × 10^-12.
• Plain self-improvement does not substitute for ADSD. In an ablation without diagnosed hypotheses or discovered skills, direct self-improvement reduced error in four comparisons but increased it in the remaining five. Extending self-improvement to five recursive iterations did not produce sustained error reduction: the GPT baseline initially worsened, partially recovered, and plateaued above its original N_0 solver, while Claude-Sonnet-5 remained nearly unchanged.

Methodology in Plain English

The pipeline starts by having the agent write a baseline solver N_0 from its own knowledge, then freezing it. The baseline is run once over a public test set and its behavior recorded — traces, residual histories, summary statistics. Test cases are split into public (usable for development) and private (hidden, reserved for final evaluation), and a literature database across power-grid and dynamic-system solvers is pre-built. The agent is barred from using off-the-shelf solvers throughout, so the comparison reflects only the base model's own numerical capability.

For diagnosis, papers are first screened using a short preview (metadata, abstract, introduction, conclusion) to keep those likely to contain concrete symptom–condition relationships. Shortlisted papers are read in full and each relationship is recorded as a diagnostic claim with exact page-level evidence. Claims sharing a common diagnostic computation are clustered into diagnostic skills. The public observations are split into non-overlapping diagnosis rounds plus a verification pool — on IEEE-118, 5,000 public observations become 1,000 diagnosis cases across five rounds of 200 cases each, with 4,000 verification cases. In each round the agent applies every skill, proposes hypotheses connecting symptoms to suspected numerical conditions, and converts each hypothesis into a deterministic detector with explicit support/falsification criteria. Hypotheses passing over 90% of the verification pool join the verified set and are labeled common "illnesses."

For skill discovery, each verified hypothesis drives a targeted literature search. Candidate papers are read in full for mechanism relevance and compatibility with N_0; a separate theory audit rechecks derivations and assumptions. Supported methods become theoretical claim dossiers, are implemented empirically, checked for reproduction of claimed results, and packaged as frozen skills containing code, usage instructions, and the dossier. Finally, the agent receives N_0 plus the skill packets — under the same model, reasoning budget, runtime, and original solver-generation prompt — and integrates them into the improved solver B.

Benchmarks: IEEE-118 (5,000 public cases, 10,000 hidden validation cases, N2 topology, feasible regime) and GOC-500 from PFΔ; AC-OPF uses 7,000 filtered edge cases near maximum loadability from 300,000 IEEE-57 cases (2,000 training, 5,000 hidden evaluation); ODE uses 111 system families from the systems-biology collection of Städter et al., split into 39 training and 72 evaluation, with the largest system having 77 state variables over 101 time points (a 101×77 trajectory matrix) and a 10^-4 trajectory-error threshold; PDE uses the 128×128 ConDiff subset with 12 coefficient configurations, 12,000 training cases, 2,400 held-out evaluation cases (200 per configuration), 16,384 unknowns per case, and 7,200 runs per solver from three repetitions per case.

Why This Matters

Impact on research. The paper reframes solver improvement as diagnosis plus knowledge acquisition rather than trial-and-error editing. It shows that giving a model more revision attempts is not the same as giving it the right diagnosis, which is a sharp challenge to recursive self-improvement narratives in scientific agents. It also proposes a clean separation — observable behavior, diagnostic knowledge, executable solver knowledge — that other agentic scientific systems could adopt, and it does so without updating model weights.

Real-world applications:

• Power-grid operations, where engineers routinely handle multiple network topologies and need skills learned on one configuration to remain reusable on others.
• Stressed grid conditions such as high demand with transmission congestion, where AC-OPF constraint satisfaction and objective quality matter most near the operating boundary.
• Systems-biology simulation, where stiff reaction networks demand stable integration and reliable error control.
• Subsurface, materials, or thermal modeling with heterogeneous, high-contrast diffusion coefficients, where ill-conditioned linear systems slow iterative convergence.

Industry relevance. Utility and grid-planning teams, scientific-software vendors, and any organization whose workflows depend on custom numerical solvers rather than off-the-shelf libraries can benefit. The framework converts expensive expert numerical-analysis knowledge into reusable, versioned artifacts, and its efficiency gains — for example, cutting operator calls on the PDE benchmark from 215.49 ± 40.35 to 71.22 ± 17.58 for GPT-5.5 — translate directly into compute cost.

Future Directions

1. Diagnostic skill coverage. How well does auto-diagnosis scale to numerical conditions not represented in the pre-constructed literature database, and can the database be extended automatically?
2. Transfer across solver families. The paper deliberately blocks off-the-shelf solvers and shows transfer across grid topologies and feasibility regimes; whether skills transfer across entirely different problem formulations, or across solvers within the same family, remains open.
3. Costs and limits of the pipeline. The paper reports solver runtime and error but does not report the pipeline's own token, retrieval, or human-review cost; whether the diagnosis-first loop is economical at larger scale is not reported.
4. Weaker baselines and harder regimes. Gains on strong baselines are smaller (for example, Claude-Sonnet-5 on ConDiff), so understanding where ADSD helps most — and combining diagnosis with reinforcement-learning-based numerical reasoning that the paper cites — is a natural next question.

Target Audience

Researchers and practitioners in AI-for-science, agentic code generation, and automated algorithm discovery; numerical analysts and scientific-computing engineers working on power systems, differential equations, or PDE solvers; and machine-learning engineers building self-improving agent pipelines who want a worked example of diagnosis-guided skill acquisition without weight updates. Readers should be comfortable with numerical-method terminology such as ill-conditioning, stiffness, preconditioning, and convergence control.

Authors’ abstract

AI agents are becoming increasingly capable of generating scientific code, but generating code is not the same as improving the algorithms behind it. For numerical solvers, execution feedback can expose poor performance, but rarely reveals its underlying cause and how to address it. We introduce Auto-Diagnosis and Skill Discovery (ADSD), a framework that links numerical diagnosis to reusable solver self-improvement. ADSD follows a diagnosis-first paradigm that first explains why a solver performs poorly, then uses this diagnosis to guide the discovery of appropriate numerical methods. The resulting knowledge is packaged into reusable solver skills, turning solver improvement from trial-and-error editing into a structured process of diagnosis, discovery, and implementation. Across four challenging numerical domains--power flow equation, AC optimal power flow control, stiff ordinary differential equations, and heterogeneous diffusion PDEs--ADSD consistently improves solver accuracy, robustness, and efficiency. On GOC-500 power flow, for example, ADSD reduces mean solver error by nearly $71\times$, with improvements further transferring to unseen grid topologies and operating regimes.