Safin-1: Safety from Within through Memory-Native State Evolution

AI summary

Overview

Research area: Machine learning — foundation model architecture, long-context memory, and safety alignment.

Technical level: Advanced.

Scope: This paper introduces Safin-1, a family of foundation models built on the Memory-Anchor Routing across Context History (MARCH) architecture, which turns a recurrent model's evolving state into an addressable memory bank that can host both context-derived state anchors and a learned, detachable Safety State.

What This Paper Is About

Recurrent and linear-attention language models compress everything they have seen into a single fixed-size state, so once an earlier association is overwritten or attenuated, it can no longer be read back out. The paper asks whether that state can instead serve as an addressable memory — and, beyond memory, as a native substrate for persistent capabilities such as safety, so that safety lives inside the model's own computation rather than depending on external guardrails, adapters, or post-hoc alignment. The authors build and test this idea at the architecture level, at 0.8B scale across several recurrent backbones, and at 4B and 35B-A3B scale.

Key Contributions

1. Safin-1 and its state-native architecture. Safin-1 is a family of foundation models built on MARCH, which converts the transient trajectory of recurrent computation into a growing bank of addressable state anchors, enabling selective access to earlier model states without altering the underlying recurrent update.

2. Persistent capability states for Safety from Within. The memory-state bank is extended with learned persistent capability states and instantiated as a detachable Safety State. This specializes a frozen Safin-1 backbone through its native state-routing pathway, making safety a model-native and selectively invoked capability.

3. Controlled and scaled empirical validation. The memory-routing architecture is validated through controlled 0.8B pretraining studies across multiple recurrent backbones, then scaled to 4B and 35B-A3B through continual pretraining and supervised fine-tuning. Results span general capability, long-context, retrieval, safety, and efficiency evaluations.

4. Efficient producer–reader implementation. Fused dense and sparse routing implementations are developed and evaluated up to 128K tokens, where sparse state routing acts as a state-level analogue of sparse attention by selecting among compressed recurrent memories rather than token-level keys.

Main Findings

• General language modeling at 0.8B: MARCH achieves the best score on every one of the eight zero-shot commonsense benchmarks among recurrent models, raising the average to 41.5, compared with 40.1 for Gated DeltaNet and 40.0 for its Log-Linear variant. It also slightly outperforms both full-attention reference models on average (41.3 for the 21-layer Transformer and 41.4 for the 24-layer parameter-matched model).

• Length extrapolation and associative recall: Across the 24 task–length combinations on RULER NIAH, MARCH outperforms the strongest recurrent baseline in 19 settings and matches it in the remaining five, with the largest gains on multi-needle tasks and at 32K — beyond the 16K maximum training length.

• LongBench: MARCH is the strongest recurrent model on every one of the twelve LongBench tasks, improving the overall average to 14.9 versus 11.9 for Gated DeltaNet and 12.5 for Log-Linear Attention — a relative gain of 25%.

• Real-world in-context retrieval: MARCH outperforms both recurrent baselines on every dataset, raising the best baseline average from 20.5 to 23.3 (a 14% relative gain), with per-dataset gains including 8% on SQuAD, 13% on SWDE, 17% on FDA, 23% on TriviaQA, 12% on DROP, and 20% on Natural Questions.

• Generality across recurrent update rules: MARCH improves NIAH in 11 of the 12 backbone–length settings across GDN, KDA, and GDN2, and consistently improves SWDE and FDA while largely preserving short-context performance and training perplexity. Six-task NIAH averages rise from 31.58 to 46.43 for GDN, from 31.64 to 41.38 for KDA, and from 40.52 to 44.80 for GDN2.

• RoPE and length extrapolation: In hybrid models with a 3:1 recurrent-to-attention ratio trained at 16K, removing RoPE improves extrapolation for both MARCH and GDN. For GDN this costs retrieval performance within 4K–16K, but MARCH benefits across all context lengths, reaching an NIAH average of 68.26 without RoPE versus 46.79 with it.

• Anchor interval: With training and inference resolutions matched, C = 512 gives the best overall trade-off. The C = 512 checkpoint can use a denser bank at inference, where C = 256 achieves the strongest aggregate performance; the Fenwick-tree variant remains competitive despite an organization unseen during training.

• Routing design: The default is dense routing with d_r = 64 and the null option included (Commonsense 41.48, LongBench 14.87, Retrieval 23.31, NIAH average 51.33). Raising d_r to 192 improves in-context retrieval but degrades other aggregates; Top-4 routing largely preserves general understanding and retrieval but underperforms on NIAH; removing the null option consistently degrades all aggregates.

• Efficiency: At 128K tokens, Top-4 routing more than doubles end-to-end training throughput relative to dense MARCH and reduces core runtime by roughly an order of magnitude. It also exceeds FlashAttention-2 in throughput at this length, though vanilla Gated DeltaNet remains faster because it performs no historical retrieval.

• Scaling to foundation models: Safin-1 variants raise the macro-average over ten capability benchmarks from 66.79 to 69.20 at 4B and from 76.25 to 78.35 at 35B-A3B, with gains most pronounced on challenging reasoning and competition-level mathematics.

• Safety State: Starting from the Safin-1 SFT checkpoints with the language-model backbone frozen, the persistent Safety State reduces average jailbreak attack success rate by 42.3% at 4B and 52.3% at 35B-A3B, while producing substantially less over-refusal than a training-matched rank-8 LoRA control.

• Stated limitation: The authors emphasize that this work represents only an initial architectural exploration of Safety from Within, and that substantial further work is needed to realize the broader vision.

Methodology in Plain English

The MARCH backbone keeps the underlying recurrent computation intact and adds a way to remember and re-read earlier versions of its own state.

• Anchor placement: A learned anchor embedding is inserted after each text boundary, placed every C = 512 tokens by default. Text positions evolve the recurrent state normally; anchor positions do not modify the state but instead read the checkpoint made at that boundary and produce compact routing metadata.

• Checkpointing: Because the recurrence is never reset at boundaries, each snapshot is a cumulative prefix, not just the most recent segment. The ordered bank of these snapshots is what the authors call state anchors, and it preserves the model's own state trajectory as causally, end-to-end trainable memory.

• Content-conditioned addressing: Anchor positions share an input embedding, but their representations — and therefore their routing keys — become state-dependent after the first layer. The router thus addresses anchors by retained content rather than by temporal index.

• Routing and fusion: For each token, a routing query scores all causally visible anchors plus a learned null option whose payload is fixed to zero, and the scores are jointly normalized with a single softmax. Retrieved state readouts are added residually to the current-state readout, preserving the original recurrent path.

• Sparse variant: Aggregation can be restricted to the Top-4 highest-scoring visible anchors, which keeps dense token-level processing while limiting the expensive matrix-valued state readout.

• Persistent capability states: The same bank can host J learnable persistent states with the same matrix-valued structure as dynamic anchors, available from the first text token without consuming input positions. For safety, the layer-wise collection of these states is the Safety State, optimized while the shared language-model backbone stays frozen, so it can be attached or removed without changing backbone parameters.

• Implementation: A recurrent producer inherits the chunkwise kernel of the underlying mixer and emits checkpoints; a fused state reader tiles queries and candidate states, reuses candidate tiles across query blocks, and fuses scoring, normalization, and weighted readout into a streaming reduction so that neither the dense routing matrix nor per-candidate matrix readouts are materialized.

• Validation plan: First, controlled 0.8B studies pretrain matched models on 50B tokens of Long-Data-Collections at 16K sequence length, with MARCH applied to GDN, KDA, and GDN2, plus full-attention references. Second, Safin-1 is scaled from dense Qwen3.5-4B and Qwen3.5-35B-A3B checkpoints, equipping 16 selected GDN layers with MARCH, then running matched continual pretraining for 50B tokens on the Intern-S2 pre-training mixture, followed by supervised fine-tuning for a 30B-token budget on Nemotron-Cascade-2-SFT-Data, both at 32K maximum sequence length.

Why This Matters

The work reframes model memory from a passive record of prior context into an active substrate for maintaining and evolving behavior over time, and it offers a concrete path toward safety as a state-native, adaptively maintainable capability rather than a constraint bolted on from outside. It connects to the R² AI perspective that scalable safety should evolve alongside model capability.

Real-world applications:

• Long-horizon agents and assistants that must accumulate information and maintain internal states across extended interactions without losing earlier associations.
• Retrieval-heavy enterprise workloads such as document question answering, where routed state anchors improved accuracy on SQuAD, SWDE, FDA, TriviaQA, DROP, and Natural Questions.
• Deployable safety specialization, where a single shared general-purpose backbone can have a safety capability attached or removed without rewriting its weights.
• Long-context serving, where the same checkpoint can trade retrieval quality against state-storage and routing cost by changing only the inference-time bank construction.

Industry relevance: The dense 4B and mixture-of-experts 35B-A3B results indicate the benefits persist across model scales and backbone organizations, and the stated 128K-token efficiency profile (more than double the throughput of dense routing, core runtime reduced by roughly an order of magnitude) targets the cost constraints that govern real long-context deployments.

Future Directions

• Extending beyond the initial architectural exploration. The authors explicitly frame the work as a first step and call for substantial further work to realize Safety from Within, including alternative and complementary architectures that make safety native to model computation.
• Broadening what persistent capability states can hold. Safety is studied as the first instance of a persistent state; the same routed interface could plausibly host other reusable specializations.
• Refining the efficiency–accuracy trade-off. Top-4 routing underperforms dense routing on NIAH despite preserving general understanding and retrieval, leaving room to improve sparse routing quality.
• Understanding inference-time bank construction. The finding that a C = 512 checkpoint performs best with a C = 256 bank, and that a Fenwick-tree layout transfers despite being unseen in training, raises open questions about how far a learned state reader can generalize across bank organizations.

Target Audience

Researchers and engineers working on foundation model architecture, linear-attention and state-space models, long-context memory, and safety alignment — particularly those interested in alternatives to external safeguards and adapter-based alignment. Readers need familiarity with recurrent state updates, attention, and standard benchmark methodology to follow the ablation tables.

Authors’ abstract

Long-horizon complex tasks require foundation models to accumulate information, maintain internal states, and adapt over extended interactions. Safety should be an intrinsic property of the model itself, rather than a behavioral constraint relying solely on external safeguards or post-hoc alignment such as supervised fine-tuning. This motivates Safety from Within, where safety-relevant capabilities are represented and invoked through the model's native computation. We present Safin-1, a family of foundation models realizing this principle through memory routing and state evolution. Safin-1 is built on Memory-Anchor Routing across Context History (MARCH), a network architecture that maintains structured memory states and selectively retrieves relevant historical information through content-conditioned routing. It supports test-time adaptation of persistent capability states without repeatedly modifying the backbone, enabling controlled specialization over a shared foundation. We investigate this interface on downstream safety tasks through a Safety State, demonstrating effective state-based adaptation with substantial safety improvements. More broadly, the routed-state interface unifies contextual memory and persistent capability adaptation within the model's native computation, reframing memory from a passive record of prior context into an active substrate for maintaining and evolving model behavior. Evaluations across general capabilities, long-context understanding, retrieval, and efficiency further validate Safin-1. These findings provide a path toward safety as a state-native and adaptively maintainable capability. This work is only an initial architectural exploration of Safety from Within, and substantial further work is needed to realize this broader vision.