NeoHorse-1: Towards Recursive Self-Improvement via Agentic Post-Training with Routing Harness

AI summary

Overview

• Research area: Natural Language Processing — specifically agentic LLM post-training, LLM routing, curriculum learning, on-policy distillation, and recursive self-improvement (RSI).
• Technical level: Advanced. The paper assumes familiarity with supervised fine-tuning, loss masking, distillation objectives, and agent harnesses.
• Scope (one sentence): The paper presents NeoHorse-1, a family of agent-native language models at 4B and 9B scale whose post-training pipeline reuses the routing signals and interaction records left behind by a deployed agent harness as both training material and as a feedback mechanism for choosing what to train on next.

Note on completeness: the supplied paper content is truncated partway through Table 2 (the 9B comparison table), so per-benchmark 9B results, the Muse-Glimmer-30B baseline row, and the NeoHorse-1-9B per-benchmark row are not available in the content provided. Where a figure is not reported in the available content, this summary says so rather than supplying a value.

What This Paper Is About

Recursive self-improvement (RSI) is the idea that an AI system can take part in improving itself, so that each training round contributes to producing the next. The paper argues that such a loop needs a concrete mechanism by which a system can observe its own capabilities and turn that evidence into the next round of learning.

The authors' key claim is that a deployed routing harness already supplies that mechanism. Because agentic interaction leaves behind execution trajectories alongside observable evidence of what the model can and cannot yet do, the harness itself can serve as the sensor for self-improvement. NeoHorse-1 is a prototype that closes this loop: capability feedback from routing and evaluation reshapes the next training mixture, so that what the system learns to do shapes what it learns from next.

Key Contributions

1. A routing-harness data pipeline that treats deployment records as structured training evidence. The system couples a heterogeneous model pool with intelligent routing and records, for every user turn, the capability demand predicted, the service tier selected, and the interaction that followed. These records become user-turn training examples that preserve interleaved reasoning, tool calls, and harness context, admitted through structural validation, six-dimensional semantic evaluation, and subscene-level labeling.

2. Routing-guided curriculum learning (R-CL) for supervised fine-tuning. Routing-derived capability-demand scores organize SFT examples into a three-stage curriculum that progressively introduces higher-scored examples while reserving some lower-scored examples for later stages — addressing the fact that the model actually served is an imperfect proxy for required capability, since the executed route can also reflect user overrides, service availability, and deployment policy.

3. Routing-guided on-policy distillation (R-OPD). The same three-stage routing progression is extended to on-policy distillation, where a fixed teacher supervises student-generated responses under the staged progression, using a response-normalized reverse-KL objective over top-K candidate tokens plus one aggregated bin. This addresses the distribution gap between SFT on recorded responses and deployment on the student's own generated prefixes.

4. A capability-guided allocation step that closes an evaluation–selection–update loop. Evaluation feedback over a stratified suite is aggregated into a model-deficiency profile that shifts the next training mixture toward underperforming regions while preserving broad coverage — while verified successful trajectories provide positive supervision and informative failures identify regions needing rebalanced coverage.

Main Findings

• Post-training lifts macro-average scores at both scales. Across ten benchmarks spanning harness-based agents, tool use, coding, and instruction following, agentic post-training raises the macro-average score of the 4B model from 58.94 to 64.87, and of the 9B model from 65.60 to 69.04.
• Scale narrowing: the post-trained 4B model substantially narrows the aggregate gap between itself and the 9B base model.
• Gains concentrate on execution-heavy tasks: the paper reports that the largest gains appear on harness-based and execution-intensive evaluations, while additional benefits from model scale are concentrated primarily on tasks involving multi-step interaction, execution feedback, and challenging code generation.
• Both base Qwen3.5 models are compared directly. Qwen3.5-4B scores a macro-average of 58.94 and Qwen3.5-9B scores 65.60 in the same evaluation pipeline — the figures the post-trained models improve upon.
• 4B per-benchmark results (Table 1, NeoHorse-1-4B vs. its base): BFCL v4 61.79 vs. 61.02; VitaBench 32.00 vs. 21.50; τ²-Bench 88.46 vs. 84.29; PinchBench 77.33 vs. 71.19; WorkBuddy Bench 34.41 vs. 24.62; QwenClawBench 44.68 vs. 38.47; HumanEval 96.95 vs. 87.20; LiveCodeBench v6 59.43 vs. 53.71; IFBench 65.33 vs. 60.33; IFEval 88.35 vs. 87.06.
• 4B comparisons are not uniformly won. On the 4B track, other open-weight baselines score higher than NeoHorse-1-4B on some individual columns — for example Nanbeige-4.2-3B reports 67.28 on BFCL v4 and 98.78 on HumanEval, and Agents-A1-4B reports 39.25 on VitaBench, 75.07 on PinchBench, and 33.37 on WorkBuddy Bench. NeoHorse-1-4B leads on the macro-average.
• 9B per-benchmark results are not reported in the available content because the paper text provided is truncated mid-table. The only 9B figure available here is the aggregate 69.04.
• Three benchmarks use repeated runs. QwenClawBench, WorkBuddy Bench, and τ²-Bench are each evaluated with three independent runs and the arithmetic mean is reported; PinchBench and VitaBench are each evaluated with a single run.
• Six agentic benchmarks are visualized in Figure 1 for both the 4B and 9B tracks, with NeoHorse-1-4B and NeoHorse-1-9B shown against comparison models whose sizes are indicated in the legends.

Methodology in Plain English

Where the data comes from. The system runs a routing harness backed by a pool of several different models across diverse real-world tasks. Every interaction is recorded: the user request, the model's reasoning, tool calls, environment observations, recovery attempts, and the final outcome. The primary corpus is on the order of 10^5 to 10^6 harness-generated trajectories, supplemented with public instruction, reasoning, tool-use, code, agent, and preference data. Corpus scale is tracked by number of trajectories and tokens after unified serialization, deduplication, and tokenizer freezing.

Three levels of organization. A trajectory is one complete harness interaction. A user turn begins at a user request and ends at the next user request or task termination — it is the unit that gets serialized into a training example. A subscene groups adjacent user turns that share a local goal and is the unit used for semantic characterization.

How data is filtered and described. Trajectories pass through structural validation, which reconstructs requests, responses, tool calls and results, and checks payload readability, message structure, causal event order, and tool-call/result closure. Outputs land in three buckets: internally complete, partially recoverable, or quarantined. Usable trajectories are then scored on six quality dimensions — goal attainment, instruction adherence, tool use, evidence consistency, error recovery, and termination — each marked PASS, WARN, FAIL, or NOT_EVALUATED, with missing evidence never converted into a positive verdict. Separately, each subscene is labeled along three axes: Scene (what the user is doing), Goal (what they expect and how success is judged), and Outcome (the verifiable result).

What routing contributes. The harness router estimates capability demand per user turn from the current request, recent dialogue, prior routing decisions, and available execution state, and assigns the turn to one of four service tiers: C0 (bounded, low-risk), C1 (general-purpose default), C2 (multi-step reasoning and execution), C3 (maximum capability or reliability). The corpus keeps three fields separate — the router's raw prediction, the policy-adjusted decision, and the tier actually served — yielding a prediction–action–outcome record. The predicted-demand estimate, not the tier actually served, is what gets used to order training examples.

How training is scheduled. SFT examples get a routing score (hard: the assigned tier index; soft: the score-weighted mean tier index). These scores drive a three-stage curriculum, each stage containing roughly one third of the examples, with some lower-scored examples deliberately reserved for later stages. The same masked SFT objective is used throughout, without resetting the optimizer or restarting the learning-rate schedule between stages. The loss mask is one only for tokens in the retained assistant target spans of the current turn — current-turn reasoning when present, serialized tool calls and arguments, visible responses, and end-of-response tokens. System instructions, tool specs, user messages, and tool results receive no prediction loss. Each user-turn example is serialized with the Qwen3.5 chat template and tool-call format; reasoning from earlier turns is omitted.

How distillation is added. Because SFT imitates recorded responses while deployment unfolds on the model's own output, the second stage of training uses on-policy distillation: the student generates a response from each recorded starting context, and a fixed teacher supplies next-token distributions at each position conditioned on the context and the student's preceding tokens. The loss is the response-normalized reverse KL over a coarsened distribution — the rollout student's top-K candidate tokens plus one bin holding the remaining probability mass, with both models using the same candidate set. The rollout checkpoint is refreshed as training proceeds so later contexts get teacher supervision on more recent student behavior, while the rollout parameters themselves stay fixed during optimization on the collected responses.

How the loop closes. At each iteration the current checkpoint is evaluated on a stratified suite kept disjoint from training by decontamination screening. Results are aggregated across attributes, quality dimensions, outcome states, and routing tiers into a model-deficiency profile, which shifts the next mixture toward underperforming regions. Between iterations the harness keeps adding trajectories processed by the same pipeline, and existing and new data are reallocated together so the corpus follows changes in usage patterns and capability.

Evaluation setup. Ten benchmarks are used: QwenClawBench, WorkBuddy Bench, PinchBench, and VitaBench for end-to-end agent execution; BFCL V4 and τ²-Bench (Airline, Retail, Telecom) for tool use and interactive task completion; HumanEval and LiveCodeBench v6 for coding; IFEval and IFBench for instruction following. Evaluation is text-only even for multimodal-capable models. Within each benchmark, models evaluated by the authors use the same harness, tool interfaces, context limits, and interaction budgets. Results taken from an official blog post or technical report are marked with an asterisk and included as reference rather than as measurements under the authors' pipeline. NeoHorse-1 and the Qwen3.5 baselines are evaluated in thinking mode with enable_thinking: true and force_nonempty_content: true, using Qwen3.5 recommendations (temperature 1.0, top-p 0.95, top-k 20, min-p 0.0, presence penalty 1.5, repetition penalty 1.0). All NeoHorse-1 variants are deployed using SGLang v0.5.17. Maximum output length is 51200 tokens for IFEval, IFBench, HumanEval, and LiveCodeBench v6, and 32768 tokens for all other benchmarks. QwenClawBench and PinchBench use OpenSquilla as the agent harness; WorkBuddy Bench uses its official native harness; VitaBench uses its official framework with DeepSeek-V4-Flash as both user-simulator and judge model because the originally recommended models are no longer available.

Why This Matters

The paper's central argument is that RSI does not require a separate self-reflection apparatus bolted onto a model — a routing harness already produces the observations that a self-improvement loop needs, because routing decisions encode an estimate of what capability an interaction demanded, and outcomes encode whether that demand was met. That reframes routing from a serving-time cost/quality mechanism into a source of training signal, and it lets multi-model experience feed the continued improvement of individual models.

Impact on research: The work sits at the intersection of several active threads — trajectory-based agentic post-training, on-policy distillation, LLM routing, and curriculum learning — and contributes a specific way to derive difficulty ordering without hand-labeled difficulty or dataset-specific heuristics. It also separates the router's prediction, the policy decision, and the executed action, which lets later work isolate the routing estimate used for ordering from the tier actually served. The paper frames itself explicitly as an initial prototype and outlines how sustaining the loop across iterations can move harness-mediated RSI from design to practice.

Real-world applications (as implied by the task domains used):

• Workplace and office software agents — WorkBuddy Bench is described as covering multi-domain workplace scenarios.
• Standardized desktop/claw-style workflows — QwenClawBench targets realistic OpenClaw tasks and PinchBench focuses on standardized OpenClaw workflows.
• Everyday service interactions — VitaBench examines multi-turn interactions in daily-life service scenarios, and τ²-Bench covers Airline, Retail, and Telecom domains.
• Tool-calling assistants and coding assistants — via BFCL V4, HumanEval, and LiveCodeBench v6.

Industry relevance: The pipeline is built on signals that production routing systems already collect, so the barrier to adopting it as a data strategy is relatively low for anyone operating a multi-model serving stack. The reported outcome that a post-trained 4B model substantially narrows the aggregate gap to a 9B base model is directly relevant to cost-sensitive deployment, since it implies a smaller served model can absorb capability that would otherwise require a larger one.

Future Directions

1. Close the loop across multiple iterations. The paper's next stated step is to extend the feedback-driven process across successive iterations and broader task settings, and to study whether its gains can be sustained as model capabilities evolve. NeoHorse-1 is described as an initial prototype of this process, not a completed demonstration.
2. Test whether the routing signal remains a valid difficulty proxy as models change. Routing-derived scores depend on a router whose tier semantics are versioned because models, pricing, and inference configurations change across deployments. As the served model improves, the predicted capability demand it faces may drift, and it is an open question how the curriculum ordering should be refreshed or recalibrated.
3. Establish how much of the gain comes from each mechanism. The pipeline combines structural validation, six-dimensional semantic evaluation, subscene labeling, routing-guided curriculum, routing-guided on-policy distillation, and capability-guided reallocation. The paper does not report ablations isolating the contribution of each in the content provided here.
4. Extend beyond text. The evaluation explicitly focuses on text-based tasks, and for multimodal-capable models only the text interfaces are evaluated. Whether the same routing-derived feedback mechanism transfers to multimodal agentic settings is not addressed.

Target Audience

Researchers and engineers working on agentic model post-training, LLM routing and model-pool serving systems, and on-policy distillation. The paper will be most useful to practitioners who already operate a harness or router and want to turn its logs into a training mixture, and to researchers interested in curriculum learning where difficulty labels are unavailable and a model-selection signal must substitute for them. Readers seeking a fully validated RSI demonstration will find this an explicitly partial prototype; readers seeking a concrete, implementable pipeline for routing-informed agentic post-training are the intended audience.

Authors’ abstract

Recursive self-improvement (RSI) requires a concrete mechanism through which an AI system observes its capabilities and converts that evidence into the next round of learning. We present NeoHorse-1, a family of agent-native models developed to explore this path through agentic post-training. Our system combines a heterogeneous model pool with intelligent routing, recording the predicted capability demand, selected service tier, and subsequent interaction for each user turn. These records are converted into training examples that preserve interleaved reasoning, tool calls, and harness context, and are admitted through structural validation, six-dimensional semantic evaluation, and subscene-level labeling. Routing signals organize supervised fine-tuning into a three-stage curriculum and extend to routing-guided on-policy distillation, where a teacher supervises student-generated responses under the same progression. Capability-guided allocation then converts evaluation feedback into the next training mixture, closing an evaluation-selection-update loop in which what the system learns to do shapes what it learns from next. Across eleven benchmarks covering harness-based agents, tool use, coding, and instruction following, post-training raises the macro-average from 58.94 to 64.87 at 4B and from 65.60 to 69.04 at 9B, substantially narrowing the aggregate gap between the post-trained 4B model and the 9B base model. NeoHorse-1 provides an initial prototype of this feedback-driven process and a path toward harness-mediated RSI across successive iterations.