On the Opportunities and Challenges of Prefill-to-Autoregressive Cross-Model Transfer

AI summary

Overview

Research area: Natural Language Processing — specifically cross-model latent communication and efficient LLM inference, with reinforcement-learning-based training of lightweight modules.

Technical level: Advanced. The paper assumes familiarity with transformer internals (hidden states, residual streams, KV caches), reinforcement learning for LLMs (GRPO), parameter-efficient fine-tuning (prefix tuning, adapters, LoRA), and evaluation methodology for ablation-style controls.

Scope: The paper introduces PLatour, an open-source framework for prefill-only latent communication from a large sender model to a small receiver model, and PACT-Check, a four-part control protocol for testing whether the resulting accuracy gains actually depend on information from the sender.

What This Paper Is About

Large models answer hard questions well but are expensive to run step-by-step, while small models are cheap but weaker. Prefill-to-Autoregressive Cross-model Transfer (PACT) splits the work: the large "sender" model reads the prompt once, hands its internal representations to a small "receiver," and the receiver does all the token-by-token generation alone. This paper builds an open, reproducible PACT system and then asks a harder question the field has largely skipped — whether measured improvements really come from the sender's information, or merely from the trainable glue modules adapting the receiver.

Key Contributions

1. PLatour, an open-source PACT framework. It connects frozen sender and receiver models through lightweight, zero-initialized linear bridges injected into the receiver's residual stream during prefilling, trained directly with GRPO task rewards — no alignment or distillation warm-up — and it supports hybrid-attention architectures by transferring hidden states rather than KV caches.

2. Strong empirical baselines. PLatour outperforms open-source communication methods on generation tasks across configurations with sender–receiver parameter ratios exceeding 100:1 and generation budgets up to 16K tokens, and it exceeds Mostik's reported results on shared mathematics and coding benchmarks using a 35B sender versus Mostik's 753B sender.

3. A demonstration that benchmark gains alone are not evidence of transfer. Controlled experiments show that comparable gains can arise from uninformative sender signals, self-communication, or receiver-only training, meaning trainable bridges can function as parameter-efficient adaptation mechanisms without exploiting useful sender information.

4. PACT-Check and PACT-Score. An evaluation protocol with four sanity checks — Dummy Transfer (DT), Irrelevant Transfer (IT), Self-Communication (SC), and Trained Receiver (TR) — plus a scalar metric measuring the accuracy margin over the strongest applicable control.

Main Findings

• PLatour surpasses the standalone receiver on every generation benchmark tested. With a Qwen3-235B sender and Qwen3-1.7B receiver, PLatour averages 78.95 on generation tasks (Chemistry 69.29, GSM8K 89.35, MATH500 78.2) versus 65.23 for Receiver-Alone; with a Qwen3-8B sender the average is 77.60.

• Existing open-source methods degrade on long-form generation. C2C and KV-Lingo both underperform Receiver-Alone on all three Qwen3 generation benchmarks; KV-Lingo drops to a 23.32 average with the 235B sender, which the authors attribute partly to direct KV-cache replacement being hard to align across a large capacity gap.

• A much smaller sender beats Mostik's reported numbers. With a Qwen3.5-4B receiver and a 35B sender, PLatour reaches 75.63 AIME-80, 64.58 HMMT, a 63.32 math average, and a 79.73 coding average, versus Mostik's (753B sender) reported 70.74 AIME-135, 54.14 LiveCodeBench v6, 86.13 HumanEval, 82.00 MBPP, and 74.09 coding average. The authors flag this as not fully controlled, since Mostik's training data are undisclosed and PLatour trains separate bridges per domain.

• Bigger sender did not mean better results. The 397B-sender PLatour variant underperformed the 35B variant (math average 60.97 vs 63.32). A single pronounced spike in clipping during training was followed by a substantial and persistent performance decline with limited recovery.

• PACT-Check reveals task-dependent attribution. On Chemistry, PLatour substantially outperforms all four sanity checks, giving evidence of meaningful information transfer; on GSM8K, all four checks achieve comparable performance, so the large gain there cannot be attributed to task-specific sender information.

• Strategy matters for sensitivity. PLatour-SFT is generally more sensitive to Dummy Transfer and Irrelevant Transfer, while PLatour (GRPO-trained) is more robust; the appendix reports that GRPO-trained bridges have substantially smaller parameter norms than their SFT counterparts.

• PACT-Scores shrink the apparent advantages. With a Qwen3-1.7B receiver, PLatour-SFT keeps positive PACT-Scores on ARC-C (+14.50) and MMLU-Redux (+21.36) but negative ones on Chemistry generation (−8.15), GSM8K (−2.90), and MATH500 (−10.15). PLatour's PACT-Scores are +3.34 on Chemistry, +0.23 on GSM8K, and −4.05 on MATH500.

• Receiver-only training is a very strong control. TR generally outperforms Self-Communication and is the strongest control in most settings; neither Qwen3.5-based PACT system surpasses its compared TR baseline in domain-averaged accuracy, and neither C2C nor KV-Lingo reaches a positive PACT-Score on any generation benchmark.

• Outperforming the sender proves nothing about transfer. On answer-only Chemistry, PLatour-SFT's gain exceeds the standalone sender's gain, yet its PACT-Score is near zero (−0.95) because controls match it.

• Efficiency is favorable for long outputs. PLatour pays extra prefill latency from running both models, but that fixed cost is amortized as decoding dominates; at longer output lengths (input fixed at 4K tokens) total inference time approaches the standalone receiver's.

Methodology in Plain English

The researchers keep both the big sender model and the small receiver model completely frozen and train only a set of small linear "bridges." For each receiver layer, a matrix maps a sender layer's hidden state into the receiver's residual stream and adds it in. Which sender layer pairs with which receiver layer is decided by matching relative depth, with an extra rule for hybrid models: full-attention receiver layers may only draw from full-attention sender layers, while linear-attention receiver layers can take either. The bridges start at zero, so before training the system behaves exactly like the standalone receiver.

Instead of making the bridge copy the sender's outputs or align representations, they train it with GRPO against task rewards — correctness and checkable outputs — so the bridge learns whatever use of the sender's states improves generated answers. They then build an evaluation harness: two inference-time interventions remove or corrupt the sender signal (a fixed template signal, and representations from an unrelated question), and two retrained controls remove the sender entirely (the receiver talking to itself, which reduces to a linear residual adapter, and plain LoRA or full fine-tuning of the receiver). Comparing against the strongest of these yields PACT-Score, a signed number where positive means the system beat every control.

Experiments run in two settings: Qwen3 senders into a Qwen3-1.7B receiver on answer-only tasks (ARC-C, MMLU-Redux, Chemistry L3) and generation tasks (Chemistry, GSM8K, MATH500), and Qwen3.5 senders into a Qwen3.5-4B receiver on harder mathematics (trained on DAPO-17k, evaluated on AIME, HMMT 2025, IMO-AnswerBench) and coding (trained on a deduplicated DeepCoder pool of 18,873 training and 201 validation examples, evaluated on LiveCodeBench v6, HumanEval, MBPP).

Why This Matters

Impact on research: The paper's main service to the field is methodological. Latent-communication papers commonly report gains over a standalone small model as evidence of successful transfer; this work shows that gains of that size can appear from prefix-tuning-like conditioning, self-communication adapters, or ordinary receiver fine-tuning. PACT-Check and PACT-Score give other researchers a way to separate real transfer from disguised fine-tuning, and the open-source release makes the setting reproducible in a way the closed-source Mostik system is not.

Real-world applications:

• Cost-aware LLM serving: routing prompts through a large model's prefill while generating with a small model can approach large-model quality at small-model decoding cost, with total latency close to the standalone receiver for longer outputs.
• Prefill–decode disaggregated infrastructure: because the sender is used only at prefilling, the design fits existing systems that already separate prefill and decode machines.
• Deployment on hybrid-attention models: transferring hidden states rather than KV caches lets the method work with models that mix full and linear attention and carry recurrent states.
• Model-catalogue reuse: organizations holding several pretrained model sizes could bridge a large and a small checkpoint of the same or different families rather than training a new mid-size model.

Industry relevance: The paper directly engages with a commercial closed-source result (Mostik, 753B sender to 4B receiver) and matches or beats it with a 35B sender, which is a practical claim about serving economics. It also warns practitioners that a bridge's measured uplift may not survive comparison with cheap LoRA or full fine-tuning of the small model — a caution with direct bearing on production model-selection decisions.

Future Directions

• Extending PACT-Check beyond PACT. The four controls are designed for systems with trainable bridges; applying them to KV-translation and cache-reuse methods, which are evaluated by fidelity to the sender's cache rather than by task reward, is an open problem.

• Understanding task dependence. Chemistry shows transfer that survives all controls while GSM8K does not. Identifying in advance which task types admit genuine prefill-to-autoregressive transfer, and why, remains unresolved.

• Training stability at extreme scale. The 397B-sender run's clipping spike and persistent decline suggest that scaling the sender further is not simply a matter of more capacity; stabilizing RL training of bridges across very large capacity gaps is an open engineering question.

• Better bridge objectives and architectures. The authors deliberately avoid alignment and distillation objectives, partly because unconstrained two-sided alignment admits a collapse-to-zero solution; whether alternative objectives can produce bridges that depend on sender information more consistently — rather than drifting into adaptation — is left open.

Target Audience

Researchers and engineers working on LLM inference efficiency, model collaboration, and latent or KV-cache communication between heterogeneous models will get the most from this paper, particularly those who need an open baseline to build on or compare against. It is also valuable for evaluation-methodology researchers interested in how ablation controls can be gamed, and for practitioners deciding between cross-model bridging and ordinary parameter-efficient fine-tuning of a small model. Readers need a solid grounding in transformer internals and RL-based post-training to follow the training details, though the PACT-Check protocol itself is legible to anyone who runs benchmark comparisons.

Authors’ abstract

Prefill-to-Autoregressive Cross-model Transfer (PACT), a form of latent communication between language models, combines a larger model's prompt representations with a smaller model's efficient generation, offering a promising way to balance capability and inference cost. However, existing systems provide limited support for independently evaluating large-small model collaboration, and improved task performance alone does not establish whether the receiver benefits from the sender's information. We introduce PLatour, an open-source PACT framework that connects frozen models through lightweight, zero-initialized linear bridges trained directly with task rewards and supports hybrid-attention architectures. PLatour significantly outperforms open-source baselines on generation tasks and surpasses Mostik's reported results on shared mathematics and coding benchmarks using a substantially smaller sender. However, controlled experiments show that the gains on some tasks can be reproduced by uninformative sender signals, self-communication, or receiver-only training. We find that trainable bridges can act as parameter-efficient adaptation mechanisms without exploiting useful sender information. We therefore propose PACT-Check to quantify how much of a trained PACT system's gain genuinely comes from communication. Finally, we discuss the opportunities and challenges of PACT systems. Code is available at https://github.com/zhenyi4/platour.