QuantCode Model: Specializing Language Models for Executable Algorithmic Trading Code

AI summary

Overview

Research area: Natural Language Processing — domain specialization of large language models for executable code generation, specifically algorithmic-trading strategies.

Technical level: Intermediate. The paper assumes familiarity with continued pretraining, supervised fine-tuning, agentic tool use, and executable benchmark evaluation, but its core arguments are accessible without deep mathematical background.

Scope: The paper studies how continued pretraining on algorithmic-trading framework code and supervised fine-tuning on validated request-to-code pairs change a model's ability to produce executable, semantically faithful Backtrader strategies on the 400-task QuantCode-Bench benchmark.

What This Paper Is About

General-purpose language models can write Python, but writing an executable algorithmic-trading strategy is harder: the code must run in a specialized framework, actually produce trades on historical data, and still implement the strategy the user asked for. This paper asks whether targeted training — continued pretraining on trading-framework source code, then supervised fine-tuning on validated request-to-code pairs — can improve performance on that execution-centered benchmark. It also reports a deployment failure in which the same specialization broke structured tool calling, and tests two ways of recovering it.

Key Contributions

1. A staged training study within directly comparable model lineages. The paper traces Qwen3.6-35B-A3B through base → continued pretraining → SFT, and Qwen3.5-397B-A17B through base → continued pretraining, so each intervention's effect is measured against its own predecessor checkpoint.
2. A construction and validation pipeline for trading-strategy SFT data. Approximately 800 validated request-to-code pairs were derived from Reddit, StackExchange, and GitHub material, rewritten into realistic user queries, with quantcode-Bench tasks excluded from the pool, and code targets generated and checked in an agentic environment with framework context available.
3. Agentic and repository-level evaluation beyond single-turn generation. Agentic QuantCode-Bench measures cumulative Judge Pass at turn budgets T1 through T10 (up to 10 retries with structured feedback), and a SWE-bench-like track of roughly 200 verified algorithmic-trading repository tasks measures long-horizon software engineering.
4. Identification and partial mitigation of a capability-retention failure. Domain specialization degraded parser-conformant structured tool calling. A 0.65/0.35 linear weight merge with the original checkpoint did not restore it; a recovery SFT stage on agentic trajectories restored formatting but left downstream repository performance below the base checkpoint.

Main Findings

• Continued pretraining gives a moderate, consistent single-turn gain. Judge Pass rose from 41.5% to 47.5% for Qwen3.5-397B-A17B (+6.0 points) and from 27.8% to 33.0% for Qwen3.6-35B-A3B (+5.2 points). Backtest success rose from 65.5% to 71.5% and from 46.0% to 53.8%, respectively; trade generation rose from 42.8% to 47.5% and from 28.5% to 36.8%.

• SFT produces the largest single-turn improvement. The final Qwen3.6-35B-A3B checkpoint reached 83.5% Backtest, 60.0% Trades, and 58.2% Judge Pass. Against the domain-pretrained checkpoint that is +29.7, +23.2, and +25.2 points; cumulatively against the original base checkpoint it is +37.5, +31.5, and +30.4 points. The largest change occurs at the executable backtest stage, not the judge.

• The specialized model is competitive with much larger general-purpose systems but below frontier models. On the single-turn leaderboard, Claude Opus 4.6 scores 98.2/77.2/75.8, GPT-5.4 scores 95.5/72.0/70.2, Claude Sonnet 4.5 scores 91.5/71.2/69.8, GPT-5.2-Codex scores 94.5/74.5/67.5, GLM-5 scores 92.4/70.3/65.4, and Gemini-3-Flash scores 76.0/63.2/59.8 (Backtest/Trades/Judge).

• SFT improves both the first attempt and the outcome after repair. For Qwen3.6-35B-A3B, SFT raised first-turn success (T1) from 32.3% to 58.3% and T10 success from 32.5% to 79.5% relative to the domain-pretrained checkpoint. Against the base checkpoint, the cumulative change is 22.3% to 58.3% at T1 and 47.5% to 79.5% at T10.

• Continued pretraining alone raises first-turn success but lowers post-repair success. The domain-pretrained checkpoint scored 32.3% at T1 (versus 22.3% for base) but only 32.5% at T10, ending 15.0 points below the base checkpoint's 47.5%. The authors interpret this as degraded instruction following and processing of conversational feedback, a hypothesis they state is not tested by a dedicated ablation.

• Iterative feedback helps general-purpose models substantially. Claude Opus 4.6 rises from 75.8% at T1 to 97.5% at T10, GPT-5.4 from 70.2% to 95.0%, Kimi-K2.5 from 64.8% to 93.5%, and Qwen3-14B from 25.2% to 62.7%.

• Domain specialization broke structured tool calling. After continued pretraining, the Qwen3.6-35B-A3B checkpoint frequently emitted unclosed tool-call tags that the serving engine's parser could not recover, stalling the agent loop. The same defect appeared in the strategy-SFT checkpoint, and the domain-specialized checkpoints could not be evaluated on the repository track for this reason. QuantCode-Bench does not directly measure parser-conformant tool calling.

• Repository-level tasks are much harder than isolated strategy generation. GPT-5.4 solved 38.6% of tasks, Qwen3.5-397B-A17B 33.2%, Kimi-K2.5 29.0%, Qwen3.5-35B-A3B 25.6%, and Qwen3.5-35B-A3B without reasoning 20.9%. The 4.7-point reasoning gap indicates these tasks benefit from explicit multi-step reasoning.

• Weight merging did not restore tool calling; recovery SFT restored formatting but not performance. The 0.65/0.35 linear merge (MergeKit) with the original checkpoint continued to produce malformed tool-call tags. Recovery SFT from the merged checkpoint restored well-formed calls in qualitative deployment checks, but repository-level performance fell below base: 23.7% for the base checkpoint versus 16.3% for the 16k-token recovery variant and 13.0% for the 32k-token variant, with reasoning disabled.

Methodology in Plain English

The researchers took existing Qwen-family checkpoints and specialized them in two ways.

First, they collected GitHub repositories related to algorithmic trading and backtesting by keyword search, filtered them by minimum star count and a most-recent-commit date threshold, and removed near-duplicates (forks, vendored copies, near-identical strategy templates) using MinHash-based detection. This produced roughly 5–6 billion tokens of framework-native code, which the models were trained to continue. Training used full fine-tuning in bf16 with FSDP2 and 8-way expert parallelism, AdamW (β₁=0.9, β₂=0.95, no weight decay), peak learning rate 1×10⁻⁴ cosine-decayed to 1×10⁻⁶, global batch of 512 sequences of 8192 tokens, and 2,250 optimizer steps.

Second, they built a supervised fine-tuning set of about 800 request-to-code pairs. Requests came from Reddit, StackExchange, and GitHub, ranked by a model-based judge and rewritten into realistic user queries where needed. Code targets were generated and checked in an agentic environment that exposed the Backtrader framework, so targets used native indicators, lifecycle callbacks, broker methods, order objects, and data lines. Validation rejected malformed imports, invalid API calls, initialization failures, runtime exceptions, and incorrect framework-object access. SFT used AdamW (β₁=0.9, β₂=0.95, weight decay 0.1), constant learning rate 1×10⁻⁵, batch size 128, and 3 epochs without sequence packing, with a standard conditional negative log-likelihood objective.

Evaluation used QuantCode-Bench's 400 tasks under greedy decoding (temperature 0) with GPT-5.4 as the semantic judge. Success is defined as passing three nested conditions: the strategy executes in the backtest environment, produces at least one trade, and passes the semantic judge. Profitability is deliberately excluded from the success criterion and from target selection, so the problem stays code generation rather than strategy discovery. The agentic setting allows up to 10 retries with structured feedback; the repository track packages roughly 200 verified algorithmic-trading tasks with reproducible automated tests and requires the model to read the issue, edit code, run tests, and repair from feedback. Recovery experiments used trajectories from the OpenHands agent framework with Qwen3.5-122B-A10B as the trajectory-generating policy, split equally between software-engineering trajectories and algorithmic-strategy trajectories, trained with an assistant-only loss mask on every assistant turn, at 16k and 32k maximum sequence lengths, with reasoning disabled.

Why This Matters

Impact on research. The paper argues that domain specialization is not one mechanism but several acting on different parts of a pipeline: continued pretraining improves framework familiarity and single-turn domain coding, while SFT improves instruction-to-code alignment and both initial and post-repair agentic success. It also shows that benchmark gains can coexist with deployment-critical regressions invisible to the benchmark, and separates two quantities agentic evaluations tend to conflate — the probability of emitting a parser-valid call and the utility of the action that call represents.

Real-world applications:

• Building developer-facing coding plugins that generate and repair trading strategies for frameworks such as Backtrader.
• Automating issue resolution, patch generation, and test-driven repair inside algorithmic-trading codebases.
• Reducing the compute and latency cost of interactive coding assistants, since higher first-turn accuracy cuts the number of backtests and repair iterations needed.
• Designing capability-retention checks for any domain-specialized model that must emit structured tool calls in production serving engines.

Industry relevance. The paper's central operational warning is that specialization toward a target benchmark can silently break the serving-format behavior that a production agent harness depends on. Weight merging, a cheap and commonly used recovery tactic, was insufficient in this setting, and trajectory distillation restored syntax without restoring the base model's downstream agent capability.

Future Directions

• On-policy capability recovery. The authors argue their observations motivate, rather than establish, a dedicated recovery stage using on-policy trajectory collection, since restoring tool-call syntax is not the same as restoring a tool-use policy.
• Controlled ablations on trajectory length. Three competing explanations for the 16k-versus-32k recovery gap are left untested: length acting as an implicit filter against failure-dominated trajectories, disproportionate token weighting from a few long trajectories, and off-policy mismatch in later teacher-generated states. Disentangling them requires equalizing training-token budgets, filtering by outcome, and comparing full versus truncated versions of the same trajectories.
• Repository-level trajectories as an SFT source. Real issue resolution, search, patch generation, test execution, and repair inside algorithmic-trading codebases is proposed as a natural next source of training data.
• Untested recovery design choices. The recovery SFT was initialized from the merged checkpoint on the assumption that proximity to the original weights helps; neither the unmerged SFT checkpoint nor other merge ratios were tried, so the merge's contribution is unknown. Other merging methods and coefficients also remain untested as ways to recover the lost capability.

Target Audience

Researchers and engineers working on domain-specific code generation, LLM fine-tuning pipelines, and agentic software-engineering systems will benefit most, particularly those deploying specialized models behind structured tool-calling harnesses. Practitioners building algorithmic-trading assistants will find the execution-centered evaluation framing and the capability-retention warning directly applicable. Readers interested in benchmark design will find the paper's distinction between executability, trading behavior, and semantic alignment useful, as will anyone evaluating whether a specialization gain on one benchmark transfers to production.

Authors’ abstract

Large language models are strong general-purpose code generators, but executable algorithmic trading remains a demanding specialization target: a model must translate a natural-language strategy specification into correct program logic for a specialized trading framework, execute on historical data, produce trades, and remain semantically faithful to the request. We study two complementary mechanisms for specializing language models for this setting: continued pretraining on algorithmic-trading framework code and supervised fine-tuning (SFT) on agent-validated request-to-code pairs. Evaluation is centered on QuantCode-Bench, our 400-task benchmark for Backtrader strategy generation, together with a repository-level SWE-bench-like track. Continued pretraining improves single-turn Judge Pass from 41.5% to 47.5% for Qwen3.5-397B-A17B and from 27.8% to 33.0% for Qwen3.6-35B-A3B. SFT applied after continued pretraining yields a larger gain for Qwen3.6-35B-A3B, reaching 58.2% Judge Pass and 83.5% successful backtests; in agentic evaluation it raises first-turn success from 22.3% to 58.3% and final success after up to 10 turns from 47.5% to 79.5%. Continued pretraining alone improves first-turn agentic success but lowers final success after repair from 47.5% to 32.5%, consistent with degraded instruction following, whereas SFT improves both. We also identify a capability-retention failure: domain specialization degrades parser-conformant structured tool calling, and targeted recovery SFT restores tool-call formatting but not the base checkpoint's repository-level agent performance. The results show that framework-oriented pretraining, validated SFT, and explicit capability-retention evaluation address distinct failure modes in domain-specific executable code generation.