The Pulse
StepFun Details Step 5 Preview’s 92-Block Agent Design
StepFun has published technical details for Step 5 Preview, an open-weight model built for long-running agent tasks. Its design combines 92 transformer blocks, sparse attention for long context and a training system aimed at keeping tool-he
AI.info Team · Written with AI ·
“The scaling law we want is one where the horizon is unbounded while the working context stays finite.”
StepFun Team
StepFun’s technical post, published October 15, lays out how it designed Step 5 Preview for agents that work through long sequences of tool calls, code changes and feedback. The company’s argument is that models handling those tasks need more than a large context window: they need enough depth to carry dependencies through lengthy inputs, plus training systems that can learn from extended work.
The model’s weights are available in StepFun’s Hugging Face repository. The company’s post explains the architecture and training approach; it does not provide independent evidence that the model outperforms competitors.
Why StepFun chose 92 transformer blocks
Step 5 Preview uses what StepFun calls a “narrow but deep” design, with 92 transformer blocks. The company’s rationale centers on long prefilling: when an agent repeatedly calls tools, the model must process growing amounts of context, and StepFun argues that deeper networks give information more room to move across those inputs. It favors adding depth directly over recurrent or looped transformer designs.
The attention layers alternate in a 3:1 pattern: three efficient layers for every full, sparse-attention layer. For the efficient layers, StepFun selected Gated Sliding Window Attention, citing implementation simplicity and a marginal performance gap against more elaborate alternatives in its tests. The full-attention path uses Gated Grouped-Query Attention, with a sparse indexing method the company calls Sparse GQA.
Sparse GQA targets million-token context
StepFun says its model supports context lengths of up to 1 million tokens, where full attention can become a major cost in training and inference. Sparse GQA selects relevant historical blocks rather than processing the full history uniformly. To cut the cost of indexing and selection, the design groups tokens into blocks of eight; StepFun says this reduces those costs by eight times.
The company reports using a top-k setting of 4,096 with those eight-token blocks. In its comparison, token coverage reached 78.06%, against 79.09% for a reference configuration using single-token blocks and a top-k of 2,048. The figures describe StepFun’s own experiment, not an independent benchmark.
Deep training also brought stability problems, including numerical explosions and gradient spikes, according to the post. StepFun says it trains with mixed-precision FP8 while keeping hidden states and their gradients in FP32; it also uses FP32 for the RMSNorm scale. The company says those changes reduce extreme activation peaks and smooth behavior when scale values approach negative one.
Agents help build tasks and manage long rollouts
StepFun describes a data pipeline in which agents coordinate task construction under expert-written principles and quality checks. The system shares task specifications, environment settings, intermediate results and prior feedback across projects, while researchers retain responsibility for expert judgment and audits. StepFun says it has built more than a million challenging, verifiable environments across scientific reasoning, software engineering, machine-learning research and professional work.
The company also tracks training tasks by duration, depth and how well they distinguish between agent attempts. Its reinforcement-learning system records each rollout through a lifecycle ledger, a response to the risk that long, difficult samples disappear before they reach training. StepFun says the ledger and turn-level records brought the end-to-end sample drop rate below 1%.
For work that outlasts a context window, StepFun says it trains agents to compact their progress into a resumable state and continue. It also assigns penalties to specific faulty turns, such as malformed tool calls, rather than spreading the blame across an entire sequence. Those are design claims from the company; the post describes methods and engineering observations, not an independent evaluation of the resulting behavior.
The weights come with deployment and license limits
The official repository lists a 650 GB FP8 checkpoint and includes a StepFun Model License 1.0. That license permits use, copying, modification and distribution under its terms, but it also requires a separate commercial agreement for specified commercial uses, including certain hardware-device deployments and some model-as-a-service businesses above a revenue threshold. “Open weights” therefore does not mean unrestricted commercial use.
StepFun’s deployment guide also sets a high hardware bar: it gives example FP8 configurations using four B300 GPUs or eight H200 GPUs, and says actual memory needs vary with context length and concurrency. The concrete trade-off in the release is a downloadable model with a million-token context target, but operating it still calls for substantial infrastructure and careful attention to the license.