ZGCM-1: A Fully Open and Extremely Efficient Foundation Model for Math and Agentic Search

AI summary

Overview

Research area: Efficient large language model pre-training, mathematical reasoning, and tool-augmented (agentic) search.

Technical level: Intermediate. The paper is written for readers comfortable with transformer architecture, attention variants, mixed-precision training, and post-training pipelines, but the central arguments can be followed without deep systems expertise.

Scope: The paper documents the design, training recipe, evaluation, and full open-source release of ZGCM-1, a 7.39B-parameter dense language model built to reach competitive math and agentic-search performance on academic-scale compute.

What This Paper Is About

Frontier reasoning and agentic-search models are increasingly the product of hundred-billion-parameter training runs with undisclosed recipes, which excludes most academic labs from both building and studying them. ZGCM-1 attacks this from the opposite direction: a 7B dense model trained from scratch with aggressively optimized architecture, numerics, and data curricula, plus an explicit bet that a small model can compensate for limited parametric knowledge by thinking longer and actively calling tools (web search, terminals, binary inspection). The paper's goal is to show that this combination is feasible on modest hardware and to release every artifact needed for others to reproduce and extend it.

Key Contributions

1. A fully open 7B foundation model and its entire lifecycle. The authors release pre-training, mid-training, and post-training weights, intermediate checkpoints, training code, per-stage data and data recipes, W&B logs, and evaluation harnesses — not just final weights, which is the common industry practice.

2. A hybrid attention architecture tuned for long context. Gated 128-token sliding-window attention is interleaved with full global attention at a 5:1 ratio across 32 layers. This cuts per-token KV cache from 128 KiB to 20 KiB and yields roughly 6.4x lower KV memory and 3.94x higher throughput at 256K context versus full attention.

3. A ~4.2x pre-training efficiency gain from system-algorithm co-design. The speedup in 16K time-to-loss over a BF16/AdamW baseline is decomposed as 1.4x from hybrid sliding-window attention, 1.5x from FP8 mixed precision with TWEO outlier regularization, 1.8x from the Muon optimizer, and 1.1x from Pre-LN data efficiency.

4. MDP-style mid-training with progressive context scaling. Interaction traces are reformulated as Markov Decision Process state-action transitions to provide dense step-level supervision, while context is extended from 16K to 64K to 256K over a 600.51B-token schedule.

Main Findings

• The core thesis holds at 7B scale. ZGCM-1-7B ranks first on average across 14 reasoning benchmarks among 7B–8B models, scoring 75.0% on AIME 2026, 97.1% on MATH-500, and 70.4% on HMMT 2025 — while remaining competitive with models orders of magnitude larger on several agentic suites (63.1% WebWalkerQA, 19.4% BrowseComp, 62.0% Binary Function Search).

• Hybrid SWA 5:1 is the best quality-throughput point tested. In head-to-head 10B-token runs, SWA 5:1 reached 9,566 tokens/s/GPU, matching the full-attention tail loss of 1.93; SWA 3:1 hit 1.92 loss at slightly lower throughput; MLA cost throughput (7,645 tokens/s/GPU) without a loss benefit.

• Lexical complexity is a cheap but narrow curriculum signal. Ordering general-language documents by surface complexity helped code and math BPB substantially (coding 1.99 to 0.81, math 0.97 to 0.94) but raised general-domain BPB by 0.03–0.09, and the ordering was judged unreliable for code and math, so those domains were interleaved independently.

• FP8 training requires activation-outlier control. Combining delayed FP8 scaling with TWEO regularization prevented numerical divergence and sustained roughly 60% BF16-equivalent MFU (about 585 model TFLOP/s/GPU on H100s).

• SFT data quality beats data volume. Pruning about half of the raw SFT candidate pool improved the six-benchmark mean from 67.78 to 68.83, with gains concentrated in BBH (+10.10) and losses confined to a subset of tasks.

• Long chain-of-thought supervision has non-monotonic value. Oversaturating the SFT mixture with extended reasoning traces introduces verbosity bias and degrades instruction following; calibrated mixture ratios preserve both.

• Agent training needs general data alongside agentic data. A joint schedule interleaving general-instruction and agentic examples outperformed a sequential general-then-agentic schedule, and agent-only fine-tuning degraded interaction fidelity.

• Long-context mid-training lets SFT stay short. The 256K foundation established during mid-training allowed long-context reasoning and agent behavior to be elicited with moderate-length SFT data, avoiding expensive 256K-long reasoning trajectories in post-training.

• Staged context extension beats a direct jump. Extending 64K then 256K (10B then 20B tokens) produced slightly lower final loss than a direct 30B-token 256K run.

• AI-native R&D has uneven autonomy. Across nine core contributors, experimentation, monitoring, and deployment received higher autonomy ratings from agent swarms, while architecture and learning-algorithm design remained more dependent on human judgment.

• A systems-level fix preserved throughput. Monitored throughput decayed from 585 to 554 model TFLOP/s/GPU across matched 80-step windows; periodic CUDA allocator cache clearing and garbage collection every 100 iterations restored stable throughput.

Methodology in Plain English

The authors treat a small model's capacity limit as something to be routed around rather than eliminated. Instead of trying to compress all world knowledge into 7B parameters, they train the model to reason at length internally and to fetch what it is missing externally through tools.

Architecturally, they avoid the cost of full attention at long context by letting most layers look only at a local 128-token window, with a small number of global layers providing long-range information flow. A learned sigmoid gate modulates the local attention output, which the authors selected after comparing full attention, MLA, and 3:1 and 5:1 sliding-window schedules under matched budgets.

For training efficiency, they combine three independent levers: FP8 matrix multiplication with delayed scaling, an activation regularizer (TWEO) that suppresses the extreme intermediate values that destabilize FP8, and the Muon optimizer instead of AdamW for matrix parameters. They estimate each lever's contribution separately and multiply them.

The data pipeline runs in two broad phases. General pre-training (0.99T tokens then 3.20T tokens) builds broad language, code, and math ability, with the first stage ordered by lexical complexity. Mid-training then spends 600.51B tokens on denser reasoning, instruction, and agentic data while stepping context from 16K to 64K to 256K. During these stages, agent trajectories are reformulated as state-action prediction problems so the model receives supervision at each decision step, not only at the final answer.

Post-training is supervised fine-tuning followed by mixed reinforcement learning. The SFT corpus contains roughly 4.9M examples, of which 96.46% are general and 3.54% agentic, spanning deep research, software engineering, and terminal interaction. Each agentic branch is aligned to the exact tool schema used at inference, validated for call-observation pairing, and filtered by deterministic rules plus model-based quality scoring. The corpus is deliberately mixed between "think" examples with explicit reasoning traces and "no-think" examples with direct answers, so one set of weights can toggle between deliberative and fast modes. Decontamination uses an 8-gram sliding window, purging any sample with over 50% overlap against evaluation sets. Reinforcement learning then uses domain-appropriate rewards: binary answer correctness for math and executable test pass rate for code.

Why This Matters

Impact on research. The dominant narrative in frontier AI is that capability scales with parameters and compute, and that training recipes are trade secrets. This paper provides a counterexample with an unusually complete release: not only weights but intermediate checkpoints, data recipes, and logs. That lets other researchers study training dynamics directly rather than inferring them from final artifacts, and it gives compute-constrained groups a documented path to competitive reasoning performance at 7B scale. The efficiency decomposition — attributing speedup to specific architectural and numerical choices — is also methodologically useful independent of the model itself.

Real-world applications:

• Long-document and research assistance: the 256K context with reduced KV footprint makes retrieval-heavy question answering and multi-document synthesis cheaper to serve.
• Software engineering agents: the execution-grounded SFT branch targets repository understanding, file localization, patch planning, and terminal use, which map directly to automated code maintenance workflows.
• Security analysis: binary function search and terminal interaction are evaluated capabilities, relevant to reverse engineering and vulnerability triage.
• Tool-using assistants on constrained hardware: the 6.4x KV cache reduction at 256K context lowers the memory cost of serving agentic workloads that repeatedly operate at long context.

Industry relevance. The paper's central engineering claims — that FP8 plus outlier regularization is stable at scale, that Muon gives roughly 1.8x step-to-loss over AdamW, and that hybrid sliding-window attention dominates full attention at long context — are directly applicable to anyone training or serving models, regardless of parameter count. The finding that SFT quality filtering beats SFT volume, and that long-CoT supervision has a saturation point, is equally relevant to teams with large post-training budgets.

Future Directions

• Does the compact-plus-tools thesis hold beyond 7B and beyond math/search? The paper demonstrates competitiveness on reasoning and agentic benchmarks but does not establish where the approach breaks down, or whether the efficiency multipliers compound at larger scales.

• How much does the MDP reformulation of agent traces actually contribute? The paper reports it as a design element and as one of its distilled findings, but the isolated effect size relative to ordinary trajectory supervision is not clearly bounded in the provided text.

• Can the AI-native R&D workflow extend to architecture and learning-algorithm design? Contributor assessments show the highest human dependence precisely in those areas, which are also where the paper's efficiency gains originate.

• How well does the released recipe transfer? Full open release invites reproduction on different hardware, tokenizers, and data pools; the sensitivity of the ~4.2x figure and of the curriculum findings to those changes is an open empirical question. The paper also states it distills eight empirical findings, of which seven are visible in the excerpted text, leaving the final finding to be recovered from the full paper.

Target Audience

This paper is most valuable to three groups. First, researchers and engineers at academic labs or smaller organizations who want to train or fine-tune competitive reasoning models without frontier-scale compute, and who will benefit most from the released checkpoints, code, and data recipes. Second, practitioners working on long-context serving and agent infrastructure, who can use the hybrid attention results and KV cache analysis directly. Third, students and researchers studying training dynamics, who gain a transparent record of capability development, curriculum effects, and failure modes — including the finding that lexical complexity is a poor difficulty proxy for code and mathematics — that is rarely visible in closed model reports.

Authors’ abstract

In this work, we present ZGCM-1, a fully open 7B dense foundation model trained from scratch with extreme data, system, and algorithmic efficiency. ZGCM-1 is founded on a core premise: compact models cannot passively memorize the open web, but can overcome parametric capacity limits by coupling deliberate internal thinking with active external tool use. To support this paradigm across a 256K context, we develop an end-to-end, high-efficiency open training recipe: Architecture & System Co-design: interleaved gated sliding-window and full attention, and a stable FP8 Muon optimizer; Progressive Curriculum & MDP Mid-Training: context scaling across 16K, 64K, and 256K, and the reformulation of interaction traces into Markov Decision Processes. Furthermore, we establish an AI-native R&D workflow where agent swarms autonomously manage cluster operations, data curation, and rapid diagnostic evaluation. Extensive evaluations show that ZGCM-1-7B is competitive across 7B model family on general benchmarks. On several challenging mathematical reasoning and agentic search suites, it remains competitive with frontier models orders of magnitude larger, such as Qwen3-235B-A22B and GLM-5.1. We also show that our pre-training design offers a ~4.2x efficiency improvement in 16K pre-training time-to-loss. Across the full development lifecycle, we distill eight actionable empirical findings-spanning architectural scaling, SFT quality pruning, long-context generalization, and agentic co-training dynamics. To facilitate community research, we open-source model weights from the pre-training, mid-training, and post-training stages, intermediate checkpoints, training code, per-stage data and data recipes, and W&B logs.