VFlash: Fast, Accurate Visual Planning via Flash Primitive Decoding

AI summary

Overview

Research area: Efficient inference and spatial reasoning for vision-language models (VLMs) used as high-level "brains" in physical AI — robotics, manipulation, and autonomous driving. It sits at the intersection of speculative/parallel decoding, multimodal tokenization, and embodied planning.

Technical level: Advanced. The paper assumes familiarity with autoregressive decoding, speculative decoding, diffusion drafters, and vision-language model architectures, though the core insight is intuitive.

One-sentence scope: VFlash is a generalist decoding framework that replaces one-coordinate-at-a-time generation of visual plans (boxes, 2D/3D trajectories, dense point motion) with single-pass parallel prediction of whole geometric primitives, and shows this is both faster and, on several trajectory benchmarks, more spatially accurate than the autoregressive target it accelerates.

What This Paper Is About

Vision-language models increasingly express their plans visually — as bounding boxes, robot trajectories, or predicted point motion — rather than only in words. But these outputs are long strings of coordinates, and autoregressive decoding emits them one token at a time, so richer spatial detail directly costs latency. The paper asks whether the known structure of a visual primitive (its type, dimensionality, number of waypoints) can be exploited so that an entire primitive is drafted in one forward pass, and whether such drafts can be accepted on geometric grounds instead of requiring exact token agreement with the target model.

Key Contributions

1. Flash Primitive Decoding (FPD): A decoding scheme where the prediction block is aligned to a complete visual primitive's boundaries, so that tens to hundreds of coordinate tokens are produced in a single drafter forward pass. The speculation budget is set by the request (via a tool call declaring block size) rather than by a fixed window.

2. Structure-based acceptance instead of token-level verification: The paper observes that spatial accuracy does not require reproducing the autoregressive model's exact token sequence. VFlash replaces greedy token-agreement verification (Eq. 2) with a schema validator — plus the spatial-confidence guard from LocateAnything for boxes — accepting structurally valid primitives in full. Text (e.g., sub-task plans) retains ordinary verified speculative decoding.

3. A generalist layout system covering both special-token and text-tokenized coordinates: Boundary alignment (one block per primitive) and position alignment (fixed role per slot: waypoint, axis, or delimiter) support boxes, variable-length 2D and 2D+3D trajectories, dual-arm robot layouts, driving trajectories, and dense 3D motion. For text-tokenized targets, coordinates are rendered into a fixed-width sign-plus-five-digit grid and converted back to the target's text format.

4. Demonstration that a lightweight drafter can beat its own autoregressive target in spatial accuracy on robot trajectory prediction, and that the resulting plans are better conditioning for downstream action models — 70.2% average success vs 62.6% baseline on MolmoSpaces — while cutting inference latency by about 80% on Jetson Thor and DGX Spark.

Main Findings

• 2D trajectory prediction improves over the AR target: On ShareRobot-Bench, VFlash lowers DFD from 0.202 to 0.194, HD from 0.175 to 0.162, and RMSE from 0.142 to 0.133; on VABench-V, DFD 0.130 → 0.126, HD 0.125 → 0.116, RMSE 0.093 → 0.089. The paper summarizes this as roughly 7% lower HD and 3–6% lower DFD and RMSE. On VABench-V this is about a 40% cut in every error relative to the best prior model, LoHo-Manip; on ShareRobot-Bench, VFlash ranks second behind RoboBrain-2.0.

• 3D robot trajectories improve substantially: Across InternA1, AgiBot, and MolmoBot, ADE falls by 18–49% with the 3D-only target and 28–61% with the 2D+3D target. Waypoint accuracy at 5 cm on AgiBot rises from 50% to 73% (3D-only) and from 40% to 74% (2D+3D). Jointly predicting 2D and 3D raises AR's 3D ADE by 15–51%, while VFlash keeps similar accuracy by drafting both segments in one pass.

• Box decoding is roughly matched in quality but much faster: With an AR fallback for boxes rejected by the spatial-confidence guard, VFlash matches AR on mean F1 while decoding 2.6–2.8× faster. Mean F1 is 47.79 vs 48.68 on COCO and 61.78 vs 60.10 on Dense200, with higher F1 at IoU 0.50 (notably 82.64 vs 76.70 on Dense200) but lower F1 at IoU 0.95 (12.11 vs 16.95 on Dense200). Boxes per second rise from 3.15 to 8.76 on COCO and 6.73 to 17.46 on Dense200. Without the fallback, mean F1 drops by about 20 points on Dense200 (39.09).

• Direct acceptance alone is not always enough: The "direct" variant (accept every schema-valid box) scores 41.32 on COCO and 39.09 on Dense200 mean F1 — a clear regression that motivates the hybrid AR-fallback path for boxes.

• Dense 3D point motion forecasting gets faster and partly more accurate: On PointMotionBench with a frozen MolmoMotion-AR target, VFlash reduces DAVIS ADE and FDE by 27% and 19% and HOT3D FDE by 8%, but WorldTrack FDE increases by 15% and PWT is 7–12% lower across the three datasets. Preliminary single-rollout measurements indicate approximately 25× faster generation.

• Better plans make better policies: On MolmoSpaces, a world action model conditioned on VFlash plans reaches 70.2% average success vs 62.6% for the baseline, and beats the listed public model PrimeR0 (60.3). Gains are largest on Pick & Place (+11.6) and Close (+12.8) vs +2.5 on Pick.

• Decoding method matters, not just the model: Substituting AR-generated 2D+3D plans for VFlash plans drops average success from 70.2% to 61.0%. Replacing trajectory states with image states drops it to 63.7%; a 3D-only plan reaches 69.3%.

• Gains grow under distribution shift in manipulation: Over the baseline, improvements are 0.8 percentage points on LIBERO, 1.9 on LIBERO-Plus, and 6.8 on LIBERO-PRO. On LIBERO-PRO, visual plans improve Spatial, Object, and Goal by 11.1, 9.4, and 6.9 points; versus subtask-only conditioning they add 2.6 (Spatial), 1.2 (Goal), and 2.0 (Long).

• Competitive driving with a frozen planning VLM: VFlash-SFT reaches 87.1 PDMS on NAVSIM (vs ReCogDrive at 86.8) and 7.79 RFS on Waymo (vs Qwen-Drive-1.0-SFT at 7.78); RL on the action head alone raises these to 90.7 PDMS and 7.86 RFS, within 0.05 RFS of the best listed Waymo result.

• Latency reduction comes almost entirely from decoding: VFlash cuts per-step latency 5.1–5.2× on MolmoSpaces on Jetson Thor and DGX Spark and 3.6× on H100; 3.4–3.6× and 2.6× on NAVSIM; and 4.5× on Waymo on H100. Under AR, decoding takes 75–87% of each step on Thor; VFlash cuts that stage 12–42×, while visual encoding, prefill, and the action model are unchanged.

• Why plain DFlash is not enough for coordinates: DFlash commits 5.5 tokens per iteration on subtask text but only 2.2 inside trajectory blocks, because a coordinate one bin off the target's choice fails token-level verification — so it only about halves decoding time. On Thor, VFlash decoding takes 72–78 ms as the driving primitive grows from 36 to 84 tokens, so denser plans add little latency over that range.

Methodology in Plain English

The starting point is a speculative-decoding setup: a small "drafter" model proposes tokens, and a larger "target" model checks them, so the expensive model runs less often. DFlash, the precursor, uses a lightweight block-diffusion drafter that guesses a block of masked tokens in one bidirectional pass, then has the target verify them one by one, accepting only tokens up to the first disagreement.

VFlash keeps the drafter but changes both what is drafted and how it is checked.

What is drafted. Before decoding, a tool call declares how big the upcoming prediction is (e.g., a block size), and the primitive type fixes the coordinate layout. That means every position in the block has a known role in advance — first coordinate of waypoint one, its second coordinate, a delimiter, and so on. The drafter therefore predicts one complete geometric object in a single pass, whether that is a 4-number box or a long variable-length trajectory. Two coordinate encodings are supported: single special tokens (one position per coordinate, from a 1,001-token vocabulary) and ordinary text digits forced into a fixed-width sign-plus-five-digit grid, which is then reformatted back into the target's native text format.

How it is trained. The autoregressive VLM is first adapted to the spatial response format with next-token supervision, then frozen. A shallow, five-layer, non-causal drafter is trained on that frozen model's internal features, minimizing a weighted cross-entropy over all positions of a block computed jointly in one forward pass, with uniform weights across positions. Critically, the labels come from annotated coordinates (or rendered grids), not from what the target model would have said — so the drafter is optimized for spatial correctness rather than imitation. Packed attention keeps separate examples and separate primitives from attending to each other.

How it is checked at inference. For visual primitives, the target model's token-by-token agreement is dropped. A decoded primitive is committed in full if it passes a schema validator — every delimiter in its declared slot, every coordinate slot holding a number token, the waypoint count matching the tool call, and, for boxes, ordered corners plus the spatial-confidence guard. Legality is enforced on logits that were already computed, so the constraints add no extra neural network evaluation. If validation fails, the first r draft slots are kept (r = 0 for trajectories) and the rest are regenerated with constrained autoregressive decoding; an accepted primitive still needs one target pass to update the cache and produce features for the action head. Ordinary text keeps the standard verified path. Because a valid primitive costs one drafter pass plus one target pass divided by the block length, adding waypoints increases the per-pass cost only mildly but never increases the number of sequential steps.

Downstream evaluation. For manipulation and driving, the VLM stays frozen and action models cross-attend to hidden states encoding the generated plans. Comparisons hold the action architecture fixed and vary only the decoder, the primitive type, or which features are fed in.

Why This Matters

The paper's central claim is conceptual as well as practical: for structured visual outputs, exact token-level fidelity to an autoregressive model is the wrong objective. Spatial predictions should be judged by geometric accuracy and usefulness for action. If a cheap one-pass drafter can actually be more accurate than the sequential model it accelerates, then autoregressive decoding is not the quality ceiling for visual outputs — which reframes a chunk of the efficiency literature that assumes lossless acceleration is the only acceptable goal.

Real-world applications:

• Robot manipulation: plan-conditioned policies over 2D+3D trajectories, with latency low enough to re-plan frequently as new camera frames arrive. Gains are largest on multi-stage tasks (Pick & Place, Close) and under perturbation (LIBERO-PRO).
• Autonomous driving: trajectory generation on NAVSIM and Waymo grids feeding a diffusion-transformer planner, at 3–5× lower per-step latency.
• On-device / embedded deployment: the evaluated hardware includes Jetson Thor (an embedded platform for robots and vehicles) and DGX Spark (desktop-class), where the ~80% latency reduction and 12–42× decoding-stage speedup matter most because AR decoding dominates each step.
• World models and motion forecasting: dense 3D point motion prediction for anticipating how a scene will evolve, at roughly 25× faster single-rollout generation.

Industry relevance: the work targets batch-size-one, latency-bound physical systems rather than server-throughput serving, and the acknowledgements note TensorRT deployment work by NVIDIA's TSE team — a signal that the method is intended for production inference stacks rather than benchmark-only gains.

Future Directions

• Generating multiple complementary primitives from shared observations: the conclusion names this explicitly as the next question — whether several plans decoded from one encoding improve control within a fixed latency budget.
• Making direct acceptance reliable for boxes: the "direct" variant loses about 20 mean F1 points on Dense200, so the current design depends on an autoregressive fallback; a better spatial-confidence criterion could remove that dependency and its cost.
• Closing the remaining quality gaps in dense motion: VFlash trails AR on HOT3D and WorldTrack ADE, raises WorldTrack FDE by 15%, and has 7–12% lower PWT — the paper does not explain why, leaving the accuracy profile of fixed-width digit layouts an open question.
• Extending beyond the tested primitive types and hardware: the layout table covers boxes, single- and dual-arm robot trajectories, and driving trajectories, but whether the approach transfers cleanly to other structured outputs and to further deployment targets is untested here.

Target Audience

Researchers and engineers working on efficient multimodal inference, embodied AI, and robot learning — particularly those deploying VLMs on latency-constrained hardware or designing action models conditioned on visual plans. It is also useful for readers interested in the broader argument that speculative decoding's lossless-verification assumption can be relaxed when outputs are structured and evaluated geometrically. The paper is advanced: readers will get the most from it with prior exposure to speculative decoding (DFlash, EAGLE, Medusa) and vision-language model tokenization.

Authors’ abstract

Models for physical AI tasks, such as robotics and autonomous driving, need not reason solely in language. They can use visual plans--points, bounding boxes, and trajectories--to represent spatial goals and anticipate motion. However, autoregressive vision-language models generate these plans one coordinate token at a time, making richer predictions increasingly costly. We introduce VFlash, a method for fast, accurate visual planning through Flash Primitive Decoding. VFlash exploits the known structure of visual primitives to predict their coordinates jointly, using a lightweight drafter that generates an entire primitive in one forward pass. The layout adapts to the primitive's type, dimensionality, and length, supporting compact boxes and long, variable-length trajectories. Crucially, spatial accuracy need not require reproducing an autoregressive model's exact token sequence. VFlash therefore trains on annotated primitives and directly accepts structurally valid predictions without target-token agreement. Across the evaluated 2D and 3D robot trajectory tasks, VFlash consistently reduces prediction errors relative to autoregressive decoding. Preliminary single-rollout measurements indicate approximately 25x faster generation for 3D point motion forecasting. A world action model conditioned on VFlash plans achieves 70.2% average success across four MolmoSpaces task suites, compared with 62.6% for the baseline.