The Pulse
Raincheck Cuts First-Token Latency While Restoring Pruned Context
Raincheck delays processing some multimodal tokens rather than discarding them, allowing a model to begin answering sooner while retaining context for later turns. Tests found lower first-token latency on MultiVerse and MT-Video-Bench, with
AI.info Team · Written with AI · fact-checked ·
Hard-pruning visual tokens can make a multimodal model answer sooner, but it permanently removes input that a follow-up question may need. A paper submitted to arXiv on October 29 proposes Raincheck as a compromise: process selected tokens first, then work through the rest while the model generates its answer or waits between turns.
The authors tested Raincheck on Qwen3-VL-8B and Qwen3.5-9B. At 30% initial multimodal-token retention, their SGLang implementation cut mean first-turn time to first token by as much as 24.1% against full prefill on the MultiVerse benchmark, with a mean increase in time per output token below 0.5%.
Tokens are delayed, not discarded
Multimodal models often spend time processing large image or video inputs before producing a first token. Pruning methods reduce that wait by selecting a subset of visual tokens, but the omitted material is unavailable if a later question points to it.
Raincheck keeps the unselected tokens and prefills them in their original order, in chunks, during decoding. The model can also continue that work after returning an answer, using available scheduler gaps; a later request must wait if the context is not ready. The approach therefore changes when the tokens are processed, not whether they have to be processed.
Supporting that update after generation starts requires more than adding tokens to the end of a sequence. The paper describes attention controls for inserting deferred input before already generated output, plus state mapping for Qwen3.5’s recurrent layers. The implementation also fuses deferred prefill with decoding to share model-weight reads.
Quality returns, but not uniformly
At 30% initial retention, Raincheck recovered an average 77.7% of the quality lost to matched hard pruning on MultiVerse and 60.4% on ConvBench, across two models and four pruning methods. Those are recovery rates against the difference between full prefill and the corresponding hard-pruned run, not claims that every answer matches a model given the full input. ConvBench is a three-turn image-dialogue benchmark; the paper’s video benchmark is MT-Video-Bench.
For example, with MMTok on MultiVerse, the Qwen3-VL-8B aggregate score rose from 48.77 under hard pruning to 55.22 with Raincheck, close to 55.43 with full prefill. On Qwen3.5-9B, the corresponding scores were 37.53, 43.12 and 44.90.
“On MultiVerse, the SGLang implementation reduces mean first-turn TTFT by up to 24.1% relative to full prefill, with less than 0.5% increase in mean time per output token (TPOT).”
Wenjie Huang et al., the authors of the paper, write in “Raincheck: Deferred Multimodal Prefill for Efficient Multi-Turn MLLM Inference”.
Video tests expose a latency tradeoff
The speed result depends on the input. On MultiVerse, first-turn time to first token fell 24.1% for Qwen3-VL-8B and 17.9% for Qwen3.5-9B against full prefill, with mean time per output token rising 0.13% and 0.47%. On MT-Video-Bench, first-token latency fell about 13% and 12%, but mean time per output token rose 19.93% and 14.92%, respectively.
Even with that slower decoding, mean end-to-end latency on the video tests was lower than full prefill by 4.83% for Qwen3-VL-8B and 6.56% for Qwen3.5-9B. The paper notes that hard pruning still completes answers sooner than Raincheck, because it does not spend decode capacity restoring tokens.
The test leaves deployment questions open
The reported systems measurements used one running request on a single NVIDIA H20 GPU per worker, with at most 31 deferred tokens processed alongside each decode token. The authors say the measurements establish the co-located execution tradeoff at batch size one; batched serving, speculative decoding and other recurrent architectures require separate evaluation.
The paper also says Raincheck leaves modality encoding unchanged. An early follow-up can encounter a wait while the remaining context is completed. The paper’s central result is narrower: on the tested models and workloads, a system can reduce first-response delay without permanently throwing away the visual tokens that later questions may depend on.