The Pulse
TokenGate Reports Up to 11.3× Faster Long-Context Prefill
A research team introduces TokenGate, a learned token-pruning method for speeding up language-model prefill. The paper reports up to 11.3× faster time-to-first-token and lower peak memory across five tasks, with results depending on the mod
AI.info Team · Written with AI · fact-checked ·
Across diverse tasks, TokenGate achieves 1.4–11.3× faster time-to-first token (TTFT) and 1.5–3.4× lower peak memory at no accuracy degradation for prefill in transformers.
Liana Patel, Rishabh Ranjan, Ion Stoica, Matei Zaharia and Carlos Guestrin, TokenGate
The authors’ October 19 paper describes TokenGate as a way to let language models skip less useful parts of long prompts during prefill—the initial processing that happens before a model starts generating a response. The headline result is a range, not a single speedup: the reported time-to-first-token improvement spans 1.4× to 11.3×, alongside a 1.5× to 3.4× reduction in peak prefill memory.
TokenGate learns which tokens to keep
Most transformer models process every input token through every layer. TokenGate adds a small scoring component at an intermediate layer: it ranks tokens, keeps a selected number, and sends only those token representations through the layers above. Earlier layers still process the full prompt.
Learning that selection presents a technical problem: choosing the top K tokens is not differentiable, so the operation does not by itself provide a standard gradient for training the selector. The paper’s token-gated attention addresses this by using token-importance scores to reweight attention logits, allowing the model and gate to be trained together. The authors say training uses a next-token-prediction objective and takes a total time comparable to standard parameter-efficient fine-tuning.
The 11.3× result comes from a specific test
The study evaluates code completion, multi-hop question answering, multi-document summarization, biomedical drug-reaction extraction and needle-in-a-haystack retrieval. Its headline speed and memory ranges compare TokenGate with standard full-context prefill at matched accuracy; they are not a claim that every task or deployment will see the maximum gain.
The paper’s appendix ties the 11.3× time-to-first-token improvement to a 16,000-token needle-in-a-haystack test using Qwen3-1.7B. In the same setting, the authors report a 3.42× reduction in peak memory. They also test models trained at that context length on longer needle-in-a-haystack inputs, reporting 12.8× faster prefill at 32,000 tokens and 13× at both 64,000 and 128,000 tokens, without accuracy degradation against full-context prefill in those tests.
Against the best-performing prior token-pruning baselines, the paper reports up to 6.4× faster prefill and up to 2× lower peak memory. That comparison is separate from the 1.4×–11.3× range against full-context prefill. The evaluation includes methods such as FastKV, SlimInfer, DASH and GemFilter, as well as the sparse-attention approaches FlexPrefill and MInference.
Hybrid models widen where pruning can happen
TokenGate also runs on Qwen3.5-2B, a hybrid model that combines full-attention layers with Gated-DeltaNet layers. The authors report that placing TokenGate at layer three matches the full-context accuracy on their 32,000-token needle-in-a-haystack test while reducing time-to-first-token by 3.8×. At that matched accuracy, they report 2.1× lower latency than the best FastKV run.
FastKV applies pruning at full-attention layers, while TokenGate can also be placed at Gated-DeltaNet layers, according to the paper. Moving the gate to layers one or two raises the reported speedup to as much as 7.8×, but accuracy falls within 6% of the full-context result rather than matching it. The trade-off makes placement a practical part of the method, not just an implementation detail.
Early computation still sets a memory limit
TokenGate requires model adaptation before deployment; it is not simply a no-training inference switch. The paper reports that each training step takes about 1.3× as long as the LoRA baseline, though its runs converge in fewer steps on four of the five tasks and total training time varies by task.
Its memory gains also have a built-in ceiling: layers below the gate still process the complete prompt, and the authors say that full-context computation below the gate dominates peak memory. TokenGate’s reported results therefore depend on where pruning begins, how many tokens the gate retains and which task is being measured—not just on the maximum speedup in the abstract.