V-LUMEN: Visual Lookup Memory for Embedding Scaling in Vision-Language Models

AI summary

Overview

Research area: Machine learning — vision-language models (VLMs), efficient scaling, and external lookup-based memory (an extension of "embedding scaling" from language models to the visual side).

Technical level: Advanced. The paper assumes familiarity with transformer internals (hidden states, residual streams, LoRA, Mixture-of-Experts), vector quantization codebooks, hashing, and logit-lens probing.

Scope (one sentence): The paper introduces V-Lumen, an external visual memory that is addressed by non-parametric hashing and fused into a frozen VLM through gated residuals, and shows on five reasoning benchmarks plus three zero-shot benchmarks that this adds capacity which grows in storage rather than in per-token compute.

What This Paper Is About

Scaling a vision-language model by making it deeper or wider costs dense computation on every forward pass. Recent language-model work (Engram, STEM, SCONE) sidesteps this by adding a huge table of learned embeddings that is addressed by a hash over discrete token IDs, so capacity grows with the table while per-token compute stays flat. The problem is that this recipe depends on text being discrete: visual features in a VLM are continuous, laid out on a 2-D patch grid, and their relevance changes with the question being asked, so a direct port is not available.

The goal of V-Lumen is to make lookup-based embedding scaling work for visual features — giving a frozen Qwen3-VL-2B backbone a sparsely read external visual memory that improves reasoning accuracy without updating any backbone weight, while keeping lookup cost constant in table size.

Key Contributions

1. A recipe for extending lookup-based embedding scaling from discrete text tokens to continuous visual features, in which every addressing mechanism is non-parametric: a frozen codebook (LaVIT, V = 16,384, d_cb = 32) supplies discrete addresses, Spatial Aggregation-based Hashing (SAH) builds region-aware keys, and Text-Conditioned Hashing (TCH, instantiated as Projected Relevance Hashing and a debiased variant) builds query-dependent keys.

2. Evidence that visual memory is an independent capacity axis. V-Lumen improves a frozen backbone, transfers zero-shot to three benchmarks whose images are disjoint from the training data, and at a matched total parameter budget matches LoRA and MoE adapters on two backbones (Qwen3-VL-2B and InternVL3.5-2B) while holding part of their capacity in an external memory rather than in adapter parameters.

3. Evidence that capacity grows in storage rather than in per-token compute. Lookup latency stays constant (total varies by less than 1%) across a 40× range of table sizes, and the memory adds a one-off prefill cost without changing per-token decode latency.

4. A mechanism for fusion: Multi-Head Conditional Gating (MHCG), which assigns an independent gate per retrieved memory cell rather than one gate for the whole retrieved vector, added as a residual at a small number of injected decoder layers (L = 2 in the main experiments).

Main Findings

• Frozen-backbone gains: Attaching V-Lumen to the frozen Qwen3-VL-2B backbone raises average accuracy across ChartMuseum, FRIEDA, VisRes, MMMU and MMMU-Pro by +17.79 ± 0.47 pp (average 2.18 → 19.97 ± 0.47), with gains on all five benchmarks and no backbone weight updated. The paper notes this setting uses a longer training schedule than the adapter settings, so it is not compared directly against LoRA-only or MoE-only.

• Matched-budget parity with adapters (not superiority): With k = 0.5, MoE + V-Lumen reaches +16.92 ± 0.91 pp versus +16.85 ± 0.95 pp for the budget-matched MoE-only (higher on ChartMuseum and VisRes, lower on the other three). LoRA + V-Lumen reaches +21.15 ± 0.02 pp versus +20.58 ± 0.85 pp for LoRA-only (+0.57 pp), higher on MMMU-Pro and ChartMuseum, on par on FRIEDA and MMMU, lower on VisRes. The authors state these differences are within about one seed standard deviation and explicitly do not claim lookup memory improves over adaptation at a fixed budget.

• Zero-shot transfer to unseen image domains: With no in-domain few-shot samples and no shared image sources with training data, V-Lumen improves the frozen backbone on all three benchmarks and in every seed: TextVQA 11.00 → 37.17 ± 3.42 (+26.17), OCR-VQA 13.80 → 47.67 ± 0.76 (+33.87), VizWiz-VQA 2.00 → 9.97 ± 1.84 (+7.97).

• Constant lookup cost: Across a 40× range in rows per cell (16,384 = 0.2× to 655,360 = 8× of the default 81,920), total lookup latency varies by less than 1% (11.16–11.23 ms) and gather time by at most 6% (0.46–0.49 ms). Key construction, which never touches the table, accounts for roughly 10.7 ms; the gather itself is under 0.5 ms. Measured for one fusion layer with 64 visual tokens in bf16 on a single RTX 6000 Ada GPU, mean over 50 runs after 10 warm-up runs.

• Table size at default setting: 81,920 rows per cell corresponds to 41,976,448 table parameters (84.0 MB); the largest tested setting (655,360 rows per cell) is 335,605,248 parameters (671.2 MB).

• Logit-lens tendency, not a firm effect: On layers 19–26, the Jensen–Shannon divergence to the model's own final prediction is lower with V-Lumen (−0.036 on VisRes, −0.004 on MMMU-Pro). Final-layer top-1 accuracy rises from 0.236 to 0.251 on VisRes and from 0.277 to 0.289 on MMMU-Pro. On VisRes the gold answer becomes top-1 earlier, with leads of +7.4 and +4.4 points at layers 19 and 20 in two of three seeds. The paper reads this as a tendency consistent with Engram's "accelerated convergence," since in each pair only two of three seeds show lower JSD.

Methodology in Plain English

• Discretize the image. Each patch's hidden state at an early decoder layer is projected to a small dimension with a frozen random projection (entries drawn from N(0, 1/D)), L2-normalized, and matched to its nearest neighbor in a frozen LaVIT codebook. This turns continuous patch features into integer "visual code" IDs — a stable address space that similar patches tend to share.

• Make addresses spatially aware. The code IDs are reshaped into an 8×8 grid (for Qwen3-VL-2B). Row and column summaries are computed non-parametrically (the mode if any code repeats in that row/column, otherwise a deterministic rolling hash of the sorted IDs) and mixed back into each cell with a weight β. A coarse 4×4 position index is then XOR-ed in as a "coordinate salt," so the same code in different regions lands at different addresses.

• Make addresses query-aware. At the same layer, a relevance score between the last text token and each patch is computed using two frozen random projections (Projected Relevance Hashing). Scores are rank-bucketed into K = 8 bins, reshaped into a bin grid, and XOR-ed into the key with a second, distinct salt prime. The same image patch therefore routes to different memory slots for different questions.

• Retrieve and fuse. The resulting integer key indexes multi-order, multi-head hash tables; a fixed number of rows is gathered per visual token. Retrieved vectors are blended into the backbone hidden state as a residual through cell-wise gates conditioned on the current hidden state, plus depthwise-convolution smoothing. Only the memory tables and fusion parameters are trained; the backbone stays frozen.

• Train in two stages. A mid-training stage on visually repetitive data (documents, charts, tables, OCR, spatial relations) forms static visual priors, followed by an SFT stage on reasoning-heavy data to align the model to use retrieved memory. Training uses answer-only masked next-token cross-entropy, AdamW with β = (0.9, 0.95), weight decay 0.01, bf16, cosine schedule with 200-step warmup, and gradient clipping at global norm 1.0. For evaluation-domain adaptation, 10% of each evaluation dataset is used as few-shot training data and the remaining 90% is held out (up to 1,000 examples per benchmark), with relaxed accuracy that accepts numeric predictions within 5% relative error.

Why This Matters

Impact on research. The paper opens a capacity axis for VLMs that is separate from adding depth, width, experts, or adapter parameters, and it does so while explicitly reporting parity rather than a win at matched budgets — a measured claim about where capacity lives (external storage) rather than how much capacity is better. It also gives a concrete non-parametric answer to three properties specific to vision (continuity, 2-D structure, query dependence) that block direct transfer of text-side lookup memories like Engram.

Real-world applications (each grounded in the benchmarks used):

• Document and form understanding — DocVQA-style layouts, tables and grids are the recurring visual motifs the training mixture deliberately targets.
• Chart and infographic question answering — ChartMuseum, ChartQA and InfographicVQA are central to the training and evaluation setup.
• OCR-heavy and scene-text queries — OCR-VQA and TextVQA show the largest zero-shot gains (+33.87 and +26.17 points).
• Assistive image question answering — VizWiz-VQA improves from 2.00 to 9.97 in the zero-shot setting.

Industry relevance. Because rows are hash-addressed and read sparsely — a few rows per visual token instead of dense matrix multiplication for every token — the memory could in principle live in host (CPU) memory or SSD-backed storage and be fetched on demand, as in the large embedding tables of industrial recommendation systems. That means visual capacity could be bought with storage and left out of GPU memory, which is directly relevant to serving small VLMs at scale. The cost profile also matters: the overhead is a one-off prefill cost, not a per-token decode cost, and the paper notes key construction is a target for optimization (about three quarters of prefill overhead in their implementation).

Future Directions

• Serving the memory outside GPU memory. Measuring latency and throughput of a host-memory or SSD-backed deployment, and designing prefetching policies. The authors note a constraint specific to their design: addresses are computed from hidden states at the first fusion layer, so fetching for later layers can overlap with computation, but fetching for the first layer cannot.

• Faster key construction. Integer hashing and indexing over the visual-token grid dominate prefill overhead; vectorizing it on the GPU while keeping the computed addresses identical should remove most of it.

• Turning memory rows into accuracy. Adding rows is free in per-token compute but does not currently improve accuracy in their setting, whereas wider memory vectors do. Understanding what the memory learns, and how training data and key design affect how much of a larger table gets used, could make the row axis deliver what width currently delivers.

• Broader settings. Testing whether the capacity axis persists on backbones beyond 2B parameters, and evaluating settings where visual codes shift systematically — for example, reusing a trained memory after the vision encoder or connector is updated.

Target Audience

Researchers and engineers working on efficient vision-language models, model scaling, and retrieval/memory-augmented architectures will get the most from this paper. It is also relevant to practitioners interested in offloading model capacity to CPU or SSD storage for serving, and to readers following the line of lookup-based embedding scaling (Engram, STEM, SCONE) who want to see how far it transfers beyond discrete text tokens. Readers without a background in transformer internals, vector quantization, and hash-based routing should expect a steep read.

Authors’ abstract

Embedding-level scaling expands the capacity of language models through lookup-based memory rather than by increasing model depth, width, or dense computation. Extending this idea to vision-language models (VLMs) is challenging because visual features are continuous, spatially structured, and query-dependent, unlike discrete text tokens that naturally support n-gram lookup. We introduce V-LUMEN, a visual lookup memory for VLMs. V-LUMEN discretizes visual features with a frozen codebook, builds spatially aware memory keys through spatial aggregation-based hashing, and modulates them with text-conditioned relevance signals; all addressing is non-parametric, and the retrieved embeddings are fused into the backbone through a gated residual pathway. On five reasoning-intensive benchmarks, V-LUMEN substantially improves a frozen Qwen3-VL-2B backbone and transfers zero-shot to three benchmarks whose images are disjoint from the training data. At a matched parameter budget, moving part of a LoRA or MoE adapter's capacity into V-LUMEN preserves accuracy on two backbones, indicating that external visual memory is an independent capacity axis rather than a stronger adaptation method. Because lookup cost is constant in table size and incurred only once at prefill, this capacity grows in storage rather than in per-token compute. Code is available at https://github.com/miso-choi/V-Lumen.