Periscope: Extending Frozen Language Models Beyond Their Context Window

AI summary

Overview

Research area: Natural Language Processing — long-context inference and inference-time methods for frozen language models.

Technical level: Intermediate. The method itself is conceptually simple (bounded probes plus a max), but the paper's cost analysis involves FLOPs, key-value cache accounting, and a crossover calculation.

Scope: The paper introduces Periscope, a training-free inference method that lets a frozen model served at a window of W tokens answer questions over texts up to W²/c tokens by reading the text as 2K independent local and strided probes and composing their scores.

What This Paper Is About

Language models read long text in a single forward pass that grows quadratically in cost, stops at the context window, and loses accuracy before it reaches that window. The authors ask whether that read can be factorized when the read is a decision over a finite set — which document is relevant, which option is supported, which passage is the evidence — since such a decision only needs one supporting span. Periscope answers by covering the text with short, independent probes of a frozen model and composing their one-token answers with a maximum, so the model never reads the whole text at once.

Key Contributions

1. Factorized reading. A decomposition that captures the context of a whole text through short, independent local and strided probes, so a model with a window of W tokens reads texts of W²/c tokens without ever reading them at once.
2. The evidence map. The probes' answers combine into a map of where the evidence sits in the text, at no extra cost beyond the probes themselves.
3. Readouts of the map. The same map ranks documents, answers questions directly, locates the evidence, and selects a short read (K chunks, about 9k tokens) that replaces reading the whole window.

Main Findings

• A read of K chunks matches the best window read on LongBench v2. For Qwen3.5-4B, reading only the K chunks the map ranks highest (9k tokens) scores 41.9 across all 503 questions, the same as the best window read at 131k tokens; every other window scores lower (32k: 36.4, 64k: 38.2, 262k: 41.6, 1M with YaRN: 39.2).
• The split falls where the cost model predicts. Against the 131k window read, the window is ahead by 3.9 points on the short split (where every context fits), the map is ahead by 3.3 on the medium split (where 44% of contexts exceed 131k tokens), and the two tie on the long split (where all of them do).
• A longer window does not help. The 4B scores 38.2 at 64k, 41.9 at 131k, 41.6 at 262k, and 39.2 at 1M with YaRN, where 93% of contexts fit whole; the 1M read drops even on the short split, whose contexts already fit at 131k.
• The gain comes from the map, not the budget. At the same budget of K chunks, the map's chunks score 6 points above the chunks most similar to the question, 9 above random chunks, and 11 above the first chunks. On the 298 contexts that fit the 131k window, reading the map's chunks gives the same answer as reading the whole context on 71% of questions, against 64% for embedding selection and 58% for random chunks.
• A stronger model gains past the window. The 27B's best window read at 262k scores 55.9 on all questions, 3.2 points above either Periscope readout (52.7), but on the long split the direct readout scores 57.4, the same as the 262k read and 7.4 points above the 131k read (50.0). On the 33 questions past 1M tokens, the 262k read scores 51.5, the map's read 57.6, and the direct readout 60.6.
• A small model can choose what a large model reads. A map chosen by the 4B and read by the 27B leads embedding selection by 4 points (49.1 against 45.5).
• The result carries to a different architecture. With Gemma-4-26B-A4B-it at its native 262k, the selected read leads the window read 43.7 against 42.9, ties it on the medium split, and leads by 5.6 points on the long split, with a fifth of the cache.
• InfiniteBench En.MC. With 229 questions and a median context of 150k tokens, the map's read scores 82.1, 5 points above the best window read at 262k (77.3) and 8 above the read at 131k (74.2). On the 79 novels that fit 131k the 131k read is ahead (82.3 against 78.5); on the 150 that do not, the map's read leads by 14 points (84.0 against 70.0).
• BRIGHT retrieval. The probes alone, with no read, rank the full corpora with the best NDCG@10 (.493) and MRR (.491) of six methods, ahead of Qwen3-Embed-4B (.461 / .450), the same embedder with chunk-max (.422 / .401), the window read at 16k (.453 / .453), FirstP (.222 / .240), and BM25 (.219 / .233). Recall@1 is .266 against the embedder's .213; Recall@10 is .666 against .671.
• Memory. Each call caches only one probe, so a 27B model reads 4.5M-token contexts on one 80GB GPU, where a single pass would need 296GB of cache (148GB for the 4B). At the longest context, one probe holds 47.6k tokens, 3.1 GB for the 27B and 1.6 GB for the 4B.
• Latency. On one A100, a batch of the 4B's probes takes 26 s at 262k tokens against 40 s for the single read, and 108 s at 1M against 457 s. The probes cost more time up to 131k (1.1 times there) and less from 262k on (0.65 times at 262k, 0.24 at 1M).
• Ablations. Removing the strided probes drops BRIGHT NDCG@10 from .637 to .551, and reading each chunk alone (MaxP) also gives .551, so finer local pieces add nothing; on LongBench v2 the strided probes add 0.8 on all questions (39.4 to 40.2). Chain of Agents, with the same model and the same K local spans, scores 32.0 against the selected read's 41.9 at roughly three times the latency (207 s against 63 s).

Methodology in Plain English

The text is split into N chunks of c tokens, and K is set to the ceiling of the square root of N. Those chunks are laid out on a K × K grid. The model is then asked the same question about two kinds of spans: K "local" spans, each K consecutive chunks in reading order, and K "strided" spans, each taking every K-th chunk so that it samples the whole text. Every chunk therefore appears in exactly one local and one strided span, and is read twice across 2K independent calls, each about √(sc) tokens.

Each call is one forward pass of the frozen model, read at a single answer token, returning a log-odds score per answer against an abstain option. For each answer, the method takes the best local score plus the best strided score, so an answer scores high only when some span of each kind supports it. Because every chunk sits in exactly one local and one strided span, the two scores of its spans give it a cell value — a map of where the evidence sits, obtained at no extra cost.

The same map then serves four tasks through fixed readouts: ranking documents by the Yes score, answering directly with the argmax, localizing the answer at the map's peak, or selecting the K chunks with the highest cells and reading them in original order in one more call (so the two stages cost 2K+1 calls). Nothing is generated during the probes, no state passes between them, and the model, prompt, and readout are unchanged from a single read.

The reach follows from the probe size: a probe of K chunks of c tokens fits a window W when Kc ≤ W, giving K ≈ W/c and a maximum readable text of K²c ≈ W²/c, which is W/c times the window. Cost grows as s^1.5 instead of s², with a crossover where the two are equal.

Why This Matters

Impact on research. The paper argues that long-context reading can be factorized when the read is a decision over a finite answer set, without training, window extension, retrieval training, or agent orchestration. It shows a frozen model's accuracy can hold past its window because every call stays in the short range where the model is reliable, and it separates the memory a long read needs from the memory the text would need.

Real-world applications:

• Question answering over whole novels, books, or multi-hundred-thousand-token documents on a single GPU.
• Long-document retrieval and reranking in BRIGHT-style domains where relevance takes reasoning rather than word overlap.
• Locating the evidence passage behind an answer, since the map's peak names the chunk that produced the decision.
• Multi-choice and option-based evaluation over contexts far longer than the serving window, when the answer set is finite.

Industry relevance. The method is training-free, uses an off-the-shelf frozen model with the same prompt and one-token readout, and bounds the key-value cache per call to one probe. That means long-context serving can be sized by the model's weights rather than by the text: the 27B reads 4.5M-token contexts on one 80GB GPU where a single pass would need 296GB. The paper frames the shift directly: a long read needs a GPU that holds the model, not one that holds the text.

Future Directions

• Readouts for open-ended outputs. The paper notes the method covers decisions over a finite answer set, and asks for readouts that handle summaries and other generative outputs.
• Distilling probe scores and maps. The authors propose distilling the probe scores and maps into retrievers and rankers far cheaper than the model that produced them.
• Multi-document and spread evidence. The method trails the window read on multi-document QA and on domains where the evidence is spread across documents; the selected read is the only place distant spans join.
• Cost below the crossover. Below the crossover (119k tokens for the 4B and 278k for the 27B at chunk size 500) the probes cost more than a single read, leaving open how to make the method cheaper for shorter texts.

Target Audience

Researchers and engineers working on long-context inference, efficient serving, retrieval-augmented generation, and LLM-as-judge reranking, particularly those who want to extend an existing frozen model's usable context without training, architecture changes, or multi-GPU memory. Readers who benefit most are those comfortable with attention cost, key-value cache accounting, and benchmark evaluation, though the method's core idea is accessible without that background.

Authors’ abstract

A language model reads long text in one quadratic forward pass, stops at the context window, and loses accuracy with length before reaching it. We ask whether the read can be factorized when deciding over a finite set: which document is relevant, which option is supported, which passage is the evidence. Periscope, a training-free inference method, arranges the $N$ chunks of a text on a $K{\times}K$ grid with $K{=}\lceil\sqrt{N}\rceil$ and asks a frozen model the same question about $K$ local spans of consecutive chunks and $K$ strided spans that sample the whole text, reading the log-odds of every answer at one token. Each answer takes its best local and strided score, and scoring every chunk by its two spans gives an evidence map at no further cost, whose peak is the chunk behind the answer. Every probe is about $\sqrt{sc}$ tokens for a text of $s$ tokens and chunk size $c$, so a window of $W$ tokens reaches $W^{2}/c$ tokens at $s^{1.5}$ cost. The map replaces the long read. On LongBench v2, reading only the $K$ chunks the map ranks highest, 9k tokens, matches the same model's best window read across windows from 32k to 1M tokens, and on InfiniteBench, where the median context is 150k tokens, it leads the best window read by 5 points. The same map ranks BRIGHT's long-document corpora with the best NDCG@10 of six methods. Each call caches only one probe, so a 27B model reads 4.5M-token contexts on one 80GB GPU, where a single pass would need 296GB of cache. A long read then needs a GPU that holds the model, not one that holds the text.