Where-OPD: Spatially Guided On-Policy Self-Distillation of MLLMs with Synthetic Scenes

AI summary

Overview

Research area: Computer Vision / Multimodal Large Language Models (post-training, distillation, synthetic data).

Technical level: Advanced. The paper assumes familiarity with on-policy distillation, KL/JS divergences, and MLLM post-training pipelines.

Scope: The paper introduces Where-OPD, a method that post-trains multimodal LLMs by giving a frozen teacher model textual hints about where question-relevant objects are located in procedurally generated synthetic scenes, and shows that the resulting improvements transfer to real-world benchmarks.

What This Paper Is About

Multimodal LLMs often fail on tasks that require finding several relevant pieces of visual evidence spread across an image, such as counting objects, reading scattered text, or linking chart labels to values. Prior on-policy self-distillation methods for MLLMs try to fix this by giving the teacher model better views of the image (crops, zooms, higher resolution), which helps fine-grained zooming tasks but hurts others. This paper asks whether privileged information can instead tell the teacher where the relevant evidence is, expressed as text, and whether that signal can be generated automatically from synthetic scenes rather than human annotations or external teacher models.

Key Contributions

1. Spatially grounded privileged guidance for on-policy self-distillation (OPSD). The teacher receives textual hints naming question-relevant visual elements and their image coordinates, while the student sees only the image and question.
2. Annotation-free generation of spatially grounded hints. Training data is built entirely procedurally from scene metadata — no pre-existing datasets, human annotations, or external higher-capacity teacher models are used.
3. Broad improvements and synthetic-to-real transfer. Across three MLLMs, the method improves counting, reading, and chart-understanding benchmarks, and also improves general visual perception on real-world images despite training only on synthetic scenes.
4. Ablation evidence that the form of privilege matters. Comparisons against SFT, GRPO, answer-only hints, cropped-image hints, and OPSD with no privileged information isolate spatial grounding as the effective ingredient.

Main Findings

• Broad average gains across three MLLMs. Where-OPD achieves the highest average across the 15 evaluated benchmarks for Qwen3.5-4B, Qwen3.5-9B, and Qwen3-VL-4B. Average scores move from 73.86 to 78.31 (+4.45) for Qwen3.5-4B, from 75.81 to 77.57 (+1.76) for Qwen3.5-9B, and from 71.81 to 72.82 (+1.01) for Qwen3-VL-4B. For Qwen3.5-4B and Qwen3.5-9B, scores are the mean of three training runs with different seeds, with a standard deviation of Avg. of ±0.20 for both.
• Specific task-level gains on Qwen3.5-4B. ChartQA improves by 7.20 points (79.32 to 86.52), EvoChart by 10.11 (68.32 to 78.43), CountQA by 2.53 (32.00 to 34.53), and OCRBench by 1.93 (87.30 to 89.23). DocVQA improves by 1.00 (96.00 to 97.00).
• Contrast with visual-zoom approaches. Vision-OPD improves Qwen3.5-4B by 8.90 points on V* and 14.56 points on ZoomBench, but decreases CountQA by 10.73 points. Vision-OPD and OPD-V also decrease CountQA and BLINK on both Qwen3.5-4B and Qwen3.5-9B.
• Synthetic-to-real transfer. The abstract reports a 3.23-point gain in average performance across CVBench, V*, ZoomBench, BLINK, HR-Bench, and MME-RealWorld. The introduction reports average gains of 3.23, 1.07, and 1.29 points for the three MLLMs across CVBench, V*, HR-Bench 4K, HR-Bench 8K, ZoomBench, MME-RealWorld, and BLINK. MME-RealWorld, HallusionBench, V*, HR-Bench 8K, GQA, and AMBER improve across all three models.
• Synthetic data alone is not enough. With the same 3,000 synthetic scenes and same number of steps, SFT reaches an average of 69.70 and GRPO 69.56, compared with the base model's 69.63, while the OPSD variant reaches 72.94 (Table 2, seven-benchmark average).
• No privileged information means no gain. OPSD without privileged information stays at the base-model average of 69.63, indicating the teacher-student information asymmetry is what drives the effect.
• Spatial coordinates are better than the answer alone. The answer-only hint yields an average of 71.27; spatial coordinates without the final count yield 72.35; coordinates plus the count yield the best average of 72.94. A cropped-image hint reduces the average to 68.76 and substantially degrades CountQA (20.22).
• A frozen teacher beats an EMA teacher. Keeping the teacher frozen (η = 0) gives the best average; increasing η generally degrades it. The authors hypothesize that updating the teacher on synthetic images introduces a slight domain shift.
• Thinking-mode evaluation. When the better of thinking-enabled and thinking-disabled results is taken per benchmark on Qwen3.5-4B, Where-OPD improves the 15-benchmark average by 3.28 points (77.06 to 80.34), while Vision-OPD gains only 0.45 points (77.06 to 77.50) and decreases accuracy on 12 benchmarks.
• Attention follows question-relevant regions. Qualitative attention maps show Where-OPD attending to all marked question-relevant regions on EvoChart and HallusionBench, and more clearly following the boundaries of stacked boxes on CountQA, where both the base model and Vision-OPD fail.

Methodology in Plain English

The authors train a student model against a frozen copy of itself that has been given extra help. Both see the same image and question. The teacher additionally receives a short text hint that names the object type being asked about, lists the coordinates of each matching object, and states the final count — for example, for "How many green crosses are there?", the hint notes one instance near (377, 400) and one near (213, 805), then concludes with a count of 2.

The key trick is that this hint is free. All training images are procedurally generated: 1024 × 1024 frames with solid backgrounds containing 12 to 40 colored geometric shapes (circles, squares, triangles, diamonds, stars, crosses, hexagons) in twelve colors, with radii of 14 to 64 pixels, placed without overlap. Because the generator itself knows every object's identity and position, the hint and the answer can be written automatically. Questions ask for the count of a sampled color-shape category, and the multiple-choice distractors are near misses of the true count.

Training samples rollouts from the student on-policy, then minimizes a generalized Jensen-Shannon divergence between the student's and privileged teacher's next-token distributions, with β = 0.5, computed over the student's top K = 100 tokens plus a tail-probability term. Gradients flow only through the student; the teacher distribution is a fixed target. At inference the hint is removed, so the model runs on the image and question alone.

The setup uses 3,000 generated image-question pairs, one epoch, eight on-policy rollouts per prompt, a global batch size of 96 (48 for the 9B model), a learning rate of 2×10⁻⁶, and 10 warm-up steps. Qwen3.5-4B and Qwen3-VL-4B receive multiple-choice questions; Qwen3.5-9B receives open-ended questions because multiple choice frequently elicited direct answers without intermediate reasoning tokens, limiting the supervision signal. Training took about 1.4 hours for Qwen3.5-4B, 1.2 hours for Qwen3-VL-4B, and 5.8 hours for Qwen3.5-9B on 4 H100 GPUs. Evaluation uses greedy decoding with thinking disabled by default, scored by a rule-based verifier, multiple-choice letter matching, and an LLM judge for unresolved responses, with unparseable or failed responses counted as incorrect.

Why This Matters

This work suggests that the most useful privileged information for distilling visual perception may be spatial guidance in language rather than better pixels. That distinction matters because textual hints can be generated automatically from synthetic data generators, removing the need for human grounding annotations or external teacher models — the two dependencies that limited earlier multimodal OPSD approaches. It also shows that training on simple synthetic geometric scenes can improve performance on real-world chart, document, hallucination, and high-resolution benchmarks, which is a much wider transfer than the training distribution would suggest.

Real-world applications:

• Document and form processing, where answers require collecting text from multiple spread-out regions of a scan.
• Chart and dashboard analysis, where labels, legends, and values must be associated across an image.
• Visual counting and inventory tasks, such as tallying objects in retail, industrial, or remote-sensing imagery.
• Reducing hallucination in assistants that answer questions about images, since HallusionBench and AMBER improve across all three models.

Industry relevance: The approach requires no annotation pipeline and no larger teacher model, and trains in roughly 1.2 to 1.4 hours on 4 H100 GPUs for the 4B models, making it a comparatively cheap post-training step that could be added on top of existing MLLM checkpoints.

Future Directions

• Extending beyond counting. The privileged hints are currently generated only for counting questions on geometric scenes; whether the same framework transfers to other spatial reasoning tasks (relations, spatial chains, layout) is left open.
• Closing the high-resolution gap. Where-OPD does not match the zooming-focused methods on V* and ZoomBench (it improves V* by 5.24 and ZoomBench by 0.87 on Qwen3.5-4B, versus 8.90 and 14.56 for Vision-OPD). Combining spatial guidance with enhanced visual observations is an obvious open question.
• Understanding why the frozen teacher wins. The finding that η = 0 is best, hypothesized to stem from domain shift on synthetic images, is not fully resolved and would benefit from further study.
• Scaling the data and generation recipe. The paper references data-size and scene-generation ablations (Tables 5 and 6) showing that 3,000 scenes and the 1024 × 1024, 12–40 object default are best among tested settings, but these specific tables are not reproduced in the available content, leaving the precise scaling behavior unreported here.

Target Audience

Researchers and engineers working on multimodal LLM post-training, on-policy distillation, or synthetic data generation for vision-language models. It is also relevant to practitioners who need to improve visual grounding and counting behavior in production MLLMs without building an annotation pipeline. Readers without a background in distillation objectives or transformer training will find the method section demanding, though the intuition — tell the teacher where to look, then let the student learn to find it itself — is accessible on its own.

Authors’ abstract

On-policy self-distillation has recently emerged as an effective approach for improving language-model reasoning by supervising students with a frozen or EMA version of themselves that receives privileged information. Its application to multimodal large language models (MLLMs), however, remains largely unexplored. Recent approaches use privileged visual information, such as image crops corresponding to a question, to improve fine-grained perception, but their gains are confined to tasks that benefit from such visual zooming and require either human-annotated grounding data or external teacher models. We introduce a different form of on-policy self-distillation for MLLMs that provides the teacher with textual, spatially grounded guidance identifying the visual elements relevant to a query. We use procedurally generated scenes with automatically available object identities and spatial coordinates, enabling scalable and annotation-free post-training. The teacher uses this spatial guidance to locate and integrate evidence from multiple relevant image regions, while the student learns to reproduce the resulting behavior from the image and question alone. Our approach consistently improves performance on counting, document and chart understanding benchmarks across multiple models. Importantly, although post-training uses only synthetic scenes, the resulting improvements transfer to real-world perception benchmarks, yielding a 3.23-point gain in average performance across CVBench, V*, ZoomBench, BLINK, HR-Bench, and MME-RealWorld. These results show that spatially grounded privileged information can induce broader perceptual capabilities through on-policy self-distillation, enabling substantial synthetic-to-real transfer beyond the task and data distribution used for post-training. Project page: https://github.com/sirkosophia/Where-OPD