PoTRE: Test-Time Reasoning inspired by Cognitive Heterogeneity

AI summary

Overview

• Research area: Test-time reasoning for large language models, specifically heterogeneous multi-agent orchestration for complex inference.
• Technical level: Advanced (assumes familiarity with LLM prompting, ensembles, and reasoning benchmarks such as ARC-AGI-2 and Humanity's Last Exam).
• Scope: The paper introduces PoTRE, a four-agent heterogeneous reasoning pipeline that is evaluated against single-stream and self-consistency baselines on HLE, ARC-AGI-2, and PRBench Finance Hard.

What This Paper Is About

Large language models are good at fluent text generation but remain brittle on tasks that demand long-horizon planning, novel abstraction, or strict domain constraints. Standard single-stream prompting and homogeneous ensembles fail on such tasks because agents that share the same probabilistic priors make correlated errors, a failure the authors call "topological mode collapse." The goal of PoTRE is to fix this by running several structurally different reasoning strategies in parallel and then reconciling their answers with a task-adaptive aggregation layer.

Key Contributions

1. A heterogeneous reasoning architecture: PoTRE (Poly-Topological Reasoning Ensembles) splits inference across four agents with genuinely different reasoning topologies — an Adversarial Refinement Agent, a Hierarchical Strategic Planning Agent, a Spectrum Search Agent, and a Direct Chain Agent — rather than replicating one topology many times.
2. Task-adaptive aggregation: A final Synthesis Agent selects among the candidate answers using one of three protocols matched to the task type: final candidate selection for constrained outputs, qualitative synthesis for open-ended generation, and neuro-symbolic (logic-consistency) verification for rule-based tasks.
3. Empirical evidence that architectural diversity beats homogeneous scaling: PoTRE outperforms self-consistency ensembles at N=8 and N=16 in nearly all tested settings, and lets the smaller Gemini-3-Flash-Preview model exceed the un-augmented Gemini-3-Pro-Preview baseline.
4. A cost-performance argument: The authors show that strategically pruning sub-agents can reduce token consumption by up to 85% while surprisingly improving accuracy by mitigating synthesis interference.

Main Findings

• State of the art on Humanity's Last Exam: PoTRE with Gemini-3.1-Pro-Preview reached 49.92% accuracy, which the authors report as a new state-of-the-art on the official leaderboards, versus a 42.15% prior-work-prompt baseline — a margin of 7.77 percentage points.
• Scaffolding lift on ARC-AGI-2: The Flash model went from a 19.16% baseline to 38.33% with PoTRE, an improvement of 19.17 percentage points, beating the N=16 self-consistency ensemble at 36.67%. The Pro model improved from 21.66% to 33.33% (+11.67).
• One exception to the pattern: On ARC-AGI-2 with Gemini-3.1-Pro-Preview, the N=16 self-consistency ensemble scored 80.00%, marginally above PoTRE's 78.33%. This is the only experimental setting where PoTRE did not beat the N=16 baseline.
• PRBench Finance Hard: PoTRE Final Synthesis produced an Average Clipped score of 0.3486 for Gemini-3-Flash-Preview, 0.3319 for Gemini-3-Pro-Preview, and 0.3931 for Gemini-3.1-Pro-Preview, each above the corresponding prior-work baseline. The introduction also claims a state-of-the-art average clipped score of 0.5196 on this benchmark, which is higher than any score shown in the paper's own results tables.
• Flash with PoTRE beats the bigger bare model: On HLE the PoTRE-augmented Flash model reached 39.80%, outperforming the standard Gemini-3-Pro-Preview baseline; on PRBench the PoTRE Flash score (0.3486) exceeded the PoTRE Pro score (0.3319).
• Different model tiers prefer different agents: On HLE, Gemini-3-Flash-Preview peaked with Adversarial Refinement (37.92%), while Gemini-3-Pro-Preview and Gemini-3.1-Pro-Preview peaked with Spectrum Search (41.92% and 48.40%).
• Synthesis always beats the best single agent: On HLE, final synthesis scored 39.80%, 44.00%, and 49.92% across the three models, above every individual agent in each case. The same held on PRBench.
• The Oracle gap remains: At least one HLE agent produced a correct answer for up to 55.96% of the 2,500 questions (Oracle upper bound for Gemini-3.1-Pro-Preview), while realized accuracy was 49.92%, meaning candidate routing is the remaining bottleneck.
• Specialists matter on spatial reasoning: In the ARC-AGI-2 analysis, the Spectrum Search Agent uniquely solved 14% of tasks missed by all other agents; with Gemini-3-Flash-Preview it produced 35 standalone solves, 17 of which were entirely exclusive.
• Synthesis corrects, not just selects: On ARC-AGI-2, neuro-symbolic synthesis lifted Gemini-3-Flash-Preview from 35 to 46 solved tasks and Gemini-3.1-Pro-Preview from 89 to 94.
• Cost reduction with pruning: Pruning sub-agents tailored to a target domain cut token use by up to 85% while improving accuracy.
• Open-book HLE comparison: On HLE Open-Book with Gemini-3-Flash-Preview, PoTRE is reported at 55.28% versus 51.7%/52.2%.

Methodology in Plain English

For every problem, PoTRE runs four agents in parallel, each with its own system prompt and its own style of thinking:

• The Adversarial Refinement Agent pairs a Proposer with a Verifier. They debate for up to five turns; the answer is accepted only when the Verifier emits a "STATUS: APPROVED" token, otherwise the agent abstains and passes nothing forward.
• The Hierarchical Strategic Planning Agent works as a role-based team — Planner, Executor, Verifier, and an Overseer. The Planner breaks the query into 2–3 sub-goals or abstracts one universal transformation rule; if the workers get stuck in a loop, the Overseer rejects the current hypothesis and issues a fresh natural-language directive to force re-planning.
• The Spectrum Search Agent spawns N independent workers at non-zero temperature to generate diverse candidates. Selection uses either constraint-guided filtering (hypothesis verification, execution pruning, then majority voting) or an LLM-as-a-Judge when no execution environment exists.
• The Direct Chain Agent is a plain chain-of-thought anchor, using zero-shot CoT for open-ended reasoning and few-shot CoT for inductive tasks.

A final Synthesis Agent receives the original problem plus the candidate set and applies a task-specific protocol: final candidate selection for constrained answers (HLE-style), qualitative synthesis for open-ended professional judgment (PRBench-style), and neuro-symbolic verification against training examples for rule-based/visual tasks (ARC-style). Operational limits are enforced per agent: T=5 turns for adversarial refinement, one turn for hierarchical planning on HLE and PRBench but T=10 on ARC-AGI-2, N=8 candidates for spectrum search, and a single answer from the direct chain. On ARC-AGI-2 all four agents emit Python code that is executed in a sandbox, with errors fed back to the agents.

Evaluation uses three benchmarks: HLE (full set, N=2,500), ARC-AGI-2 (public evaluation set of 120 tasks), and PRBench Finance Hard (N=300). Baselines are Gemini-3-Flash-Preview, Gemini-3-Pro-Preview, and Gemini-3.1-Pro-Preview with prior-work prompts, plus self-consistency at N=8 and N=16. HLE grading uses a blind LLM-as-a-Judge (Gemini-3-Flash-Preview) with the official prompt; ARC-AGI-2 uses strictly binary pixel-perfect programmatic validation; PRBench uses the official evaluation script and rubric-weighted scoring.

Why This Matters

• Impact on research: The paper argues that reasoning gains need not come from parameter scale or raw sampling volume. Orchestrating topologically distinct reasoners produced larger gains than scaling homogeneous self-consistency in nearly every setting, offering an alternative direction to sequential-depth and homogeneous-refinement "Deep Research" systems.
• Real-world applications:
  • High-stakes professional judgment in finance and law, where PRBench measures adherence to expert-curated rubrics.
  • Expert-level cross-domain question answering, as tested by HLE's multidisciplinary problems.
  • Abstract visual or program-induction problems, where agents generate and execute code under constraint verification.
  • Cost-sensitive deployment, since pruned agent configurations cut token use by up to 85% while improving accuracy.
• Industry relevance: The framework runs on publicly released Gemini 3 models, requires no weight modification or specialized training pipeline (unlike RL-trained orchestrators or data-centric SFT recipes), and demonstrates that lightweight models with scaffolding can outperform heavier frontier baselines under comparable or fewer tokens — directly relevant to serving cost and latency budgets.

Future Directions

• Closing the Oracle gap: Even the best configuration reached 49.92% against a 55.96% oracle ceiling on HLE, so better candidate routing and selection remains an open problem.
• Explaining the ARC-AGI-2 exception: Why large self-consistency ensembles edge out PoTRE for Gemini-3.1-Pro-Preview on ARC-AGI-2, and whether a different aggregation rule would fix it, is unresolved.
• Systematic pruning policies: The paper shows domain-tailored pruning can cut tokens by up to 85% and improve accuracy, but how to choose which agents to drop for a new domain is not reduced to a general procedure.
• Generalizing beyond the three benchmarks: Whether heterogeneous topologies transfer to other modalities and task families, and whether synthesis interference can be predicted in advance, are natural extensions.

Target Audience

Researchers and engineers working on LLM inference-time reasoning, test-time compute allocation, and multi-agent orchestration; practitioners building cost-constrained reasoning systems on frontier models; and benchmark-focused evaluators interested in HLE, ARC-AGI-2, and PRBench results. Readers unfamiliar with LLM prompting and benchmark protocols will need background reading first.

Authors’ abstract

While Large Language Models (LLMs) excel at many tasks, they frequently struggle with complex reasoning that requires long-horizon planning and iterative error correction. Furthermore, standard single-stream prompting proves brittle when models encounter novel abstractions or rigorous domain constraints. We introduce PoTRE (Poly-Topological Reasoning Ensembles), a heterogeneous framework that decouples inference into four agents: (1) Adversarial Refinement Agent, (2) Hierarchical strategic Planning Agent, (3) Spectrum Search Agent, and (4) Direct Chain Agent. A final Task-Adaptive Aggregation Layer dynamically reconciles these perspectives -- via final candidate selection, semantic synthesis, or neuro-symbolic verification -- to produce a robust global solution. We evaluate PoTRE on three frontier benchmarks: ARC-AGI-2, Humanity's Last Exam (HLE), and PRBench Finance. PoTRE achieves state-of-the-art accuracy of 49.92% on HLE, surpassing the previous best official score. We demonstrate that this architectural heterogeneity achieves improved reasoning performance using similar or fewer inference tokens compared to heavily scaled homogeneous baselines.