LatentPort: Beyond KV Cache - Cross-Model Transfer of Recurrent Memory in Hybrid Language Models: A 4B-to-9B Hybrid-State Handoff Without Target Prefix Replay

AI summary

Overview

• Research area: Natural language processing — specifically cross-model transfer of persistent inference state in attention–recurrent hybrid large language models.
• Technical level: Advanced. The paper assumes familiarity with KV caching, Gated DeltaNet recurrence, negative log-likelihood, and bootstrap confidence intervals.
• Scope: One directed handoff of persistent hybrid inference state from a Qwen3.5-4B-Base source to a Qwen3.5-9B-Base receiver, evaluated on teacher-forced continuation with zero historical target prefix tokens.

What This Paper Is About

Normally, when one language model takes over from another, the receiver must reread the entire historical prefix and rebuild its internal inference state from scratch. Prior work has shown that attention key/value (KV) caches can be translated across models to avoid this repetition, but hybrid models keep more than KV: their Gated DeltaNet layers also carry recurrent matrices and convolution history.

This paper asks whether that additional persistent memory can cross a model boundary at all. The authors transfer KV, recurrent matrices, and convolution history from a 4B hybrid model into a differently sized 9B sibling, and test behaviorally whether the larger receiver can continue the text well without ever seeing the original prefix.

Key Contributions

1. Beyond-KV information. A controlled intervention holds translated KV fixed and adds the full Gated DeltaNet (GDN) persistent-state package, establishing a substantial contribution from recurrent and convolution state beyond attention KV.
2. Functional compatibility. Directly reusing recurrent and convolution state outperforms the tested learned GDN translation maps across differently sized siblings, which the authors read as evidence of partially shared functional coordinates for persistent recurrent memory.
3. A constructive handoff. Component selection plus a compact rank-4 correction (434,176 trainable parameters) produces a 9B handoff that significantly beats continued 4B inference while processing zero historical prefix tokens.
4. Controlled evidence. Paired-document uncertainty, wrong-donor shuffle controls, frozen splits, and same-model restoration checks support the result, with failures reported explicitly (the near-native gate was missed and the 16K branch was not run).

Main Findings

• KV alone is nearly useless. In Experiment 1 on 64 PG19 documents, KV-only NLL is 3.1153, close to empty 9B's 3.2457 and far above native 9B's 2.1609. Adding the translated GDN persistent-state package lowers NLL by 0.7473 nats/token (95% paired document bootstrap CI [0.6921, 0.8047]), improving all 64 documents.
• The package removes most of the KV-only gap. The mean per-document reduction of KV-only excess NLL is 79.7% (median 80.2%). The contrast bundles recurrent matrices, convolution history, and initialization semantics, so their individual shares are not isolated.
• Direct reuse beats learned translation. Replacing recurrent and convolution translation with direct copying reduces Experiment 1 NLL from 2.3679 to 2.2913, a 0.077-nat improvement. In Experiment 2's 32-document factorial, translating KV lowers NLL by 0.0705 nats/token (CI [−0.1009, −0.0409]), translating recurrent state instead raises NLL by 0.0280 (CI [0.0104, 0.0475]), and convolution translation's estimate is −0.0004 (CI [−0.0026, 0.0018]), an interval that does not establish equivalence.
• The tie rule selected fewer translations. TDT had the lowest validation excess NLL (0.132226) versus TDD (0.133377), but the difference fell inside the frozen 0.01-nat tie tolerance, so TDD was selected. On the separate 64-document test set, TDD reached excess NLL 0.105 versus 0.134 for TTT (paired CI [0.0170, 0.0418]).
• A small correction beats continued 4B. On 64 fresh FineWeb-Edu documents at 4K, corrected 9B NLL is 1.989 versus 2.042 for continued 4B, a paired difference of −0.0521 nats/token (CI [−0.0843, −0.0185]).
• Near-native metrics. The corrected handoff's continuation loss is 0.076 nats/token above native 9B, JS divergence is 0.022, and native context recovery (NCR) is 0.918 (recovering 91.8% of native 9B's prefix-derived benefit relative to empty 9B — a context-benefit ratio, not accuracy). Top-1 agreement with native 9B rises from 0.846 to 0.864.
• The correction is tiny. It lowers base NLL from 2.018 to 1.989 and removes 27.5% of the remaining native gap (ratio CI [22.4%, 33.9%]). The median layer-relative residual norm is 0.0264 and the maximum 0.0440; the correction occupies 1.90 MB serialized.
• The near-native gate failed. Excess NLL 0.076 exceeds 0.05, NCR 0.918 is below 0.95, and top-1 agreement 0.864 is below 0.90. The conditional 16K branch was therefore not run.
• Wrong donors destroy the advantage. Correct full state beats shuffled state by 0.948 nats/token in Experiment 1 (CI [0.878, 1.022]); in Experiment 2, corrected minus shuffled is −1.188 (CI [−1.296, −1.085]), with shuffled NLL 3.178 — worse than empty 9B's 2.842.
• Lower reconstruction error is not better behavior. The learned recurrent mapper reduces normalized recurrent-state reconstruction error from 0.6732 (direct) to 0.5037, yet performs worse behaviorally than direct reuse.
• No interaction replicated. The validation KV–recurrent interaction is 0.0284 nats/token; its test estimate of 0.0151 falls below the 0.02 materiality threshold.

Methodology in Plain English

The authors took two official frozen Base checkpoints — Qwen3.5-4B-Base and Qwen3.5-9B-Base — that differ in parameter count (4.206B vs 8.954B) and hidden width (2,560 vs 4,096) but share exactly matched persistent-state geometry. Both have 32 layers in a repeating three-GDN-layer/one-attention-layer pattern (24 GDN, 8 attention), identical KV head structure (4 heads of width 256), recurrent state of shape 1×32×128×128 per GDN layer, and convolution state of shape 1×8192×4 per GDN layer. Before any cross-model work, they verified that state captured and restored within the same model reproduces native behavior exactly across 16 contexts per model at four lengths (100% top-1 agreement, zero maximum absolute logit difference, minimum next-logit cosine 0.99999988).

They then ran a real prefix (4,096 tokens) through the 4B source, captured its full hybrid state, and installed target-shaped copies into the 9B runtime using fresh cache objects. The receiver's first input is the real next document token — a bridge, not a learned prompt — followed by 64 teacher-forced targets. No historical prefix tokens reach the receiver.

Attention KV was translated with per-layer, per-head, per-role ridge maps (keys de-rotated before mapping and re-rotated for the target); GDN state was either copied directly or mapped. Component choices were selected with a frozen 0.01-nat tie rule that prefers fewer translated components. A final identity-anchored rank-4 residual correction, trained through the frozen receiver against native 9B's output distribution (full-vocabulary KL over nine output positions, plus an identity penalty), was fitted on 128 calibration documents and validated on 32, then frozen. Uncertainty uses 10,000 paired document bootstrap resamples with the document as the statistical unit.

Why This Matters

Impact on research. The paper makes a qualified first demonstration: to the authors' knowledge, no prior work has shown cross-model transfer of built-in persistent recurrent inference state between differently sized attention–recurrent hybrid LLMs without target prefix replay. It also delivers a negative result — lower tensor reconstruction error does not predict better cross-model continuation — and shows that direct reuse of recurrent state can beat a learned mapper. The result is explicitly bounded: one direction, one geometry-matched Base-model pair, 4K prefixes, teacher-forced scoring.

Real-world applications (as motivated by the paper, not demonstrated in it):

• Model switching or upgrading mid-session without reprocessing a long conversation prefix.
• Serving systems that need to move an active session between model sizes for cost or capacity reasons.
• Hybrid-state management in long-context inference, where rebuilding recurrent state is the expensive step.
• A possible state interface (the authors call it a speculative "state ABI") letting model families designed together hand off persistent state across sizes.

Industry relevance. If compatible persistent-state coordinates could be engineered rather than stumbled into, model families could be trained deliberately so that smaller and larger siblings share transferable memory — relevant to inference infrastructure, session migration, and tiered serving. The paper is explicit that prototype timings do not establish serving speedup and that the experiment establishes no general interface.

Future Directions

1. Unconstrained free generation after handoff. The paper names this as one of the two most decisive next tests, since it would reveal whether small state errors compound once ground-truth continuation tokens stop stabilizing the receiver.
2. Replication on an independent model pair. Compatibility may be peculiar to these Qwen siblings; cross-family transfer is untested, and shared pretraining or training recipe may contribute.
3. Mismatched geometry. The paper calls this a subsequent, harder test: whether the observed usefulness survives altered head/layer organization or different learned update dynamics.
4. Isolating components and extending context. The original contrast does not separate recurrent matrices from convolution history and initialization semantics, and the 16K branch was not run, so longer-context quality and per-component contributions remain open.

Target Audience

Researchers and engineers working on LLM inference systems, KV-cache reuse, and hybrid attention–recurrent architectures will get the most from this paper, particularly those interested in cache translation, model switching, and long-context serving. It is also relevant to anyone studying what is and is not portable between separately trained models. Readers without background in recurrent state, KV caching, and log-loss evaluation will find it demanding.

Authors’ abstract

Can one language model hand its live memory to another without the receiver rereading the context? We demonstrate useful persistent hybrid-state transfer across one architecture-matched Qwen3.5 4B-to-9B sibling pair. To our knowledge, this is the first demonstrated cross-model handoff of persistent recurrent inference state between differently sized hybrid language models without target prefix replay. Translated attention KV alone leaves a large gap; adding the Gated DeltaNet (GDN) persistent-state package lowers teacher-forced negative log-likelihood (NLL), the average next-token log-loss, by 0.747 nats/token (95% paired document bootstrap CI [0.6921, 0.8047]), improving all 64 PG19 documents. Direct recurrent and convolution reuse outperforms the tested learned GDN maps, consistent with partial functional compatibility of persistent-state coordinates. A fresh component factorial selects translated KV with direct recurrent and convolution state. An additional 434,176-parameter correction improves that base on 64 fresh web documents: continuation loss is 0.076 nats/token above native 9B (excess NLL), Jensen-Shannon (JS) divergence is 0.022, and native context recovery (NCR) is 0.918. Corrected 9B significantly beats continued 4B inference while processing zero historical prefix tokens. Evidence covers one direction, one geometry-matched Base-model pair, and 4K teacher-forced continuation; the near-native gate failed, the 16K branch was not run, and free-generation equivalence and a general state interface remain unproven.