Skill-Decoupled On-Policy Expert Distillation for Unified Vision-Language Models

AI summary

Overview

Research area: Multimodal post-training of vision-language models (VLMs), specifically reinforcement learning (RL) and knowledge distillation for combining multiple visual capabilities into one model.

Technical level: Advanced. The paper assumes familiarity with RL fine-tuning, on-policy distillation, KL divergence objectives, and gradient-conflict analysis.

Scope: The paper proposes SCOPED, a two-stage framework that trains separate capability-specific VLM experts with RL, extracts explicit "skills" from their reward-evaluated trajectories, and distills the skill-conditioned experts into one unified student model evaluated on eight multimodal benchmarks.

What This Paper Is About

Training a single vision-language model to be good at several different visual skills at once is hard: optimizing one shared model on mixed data creates conflicting training signals, so gains in one capability can hurt another. The common fix — train separate experts and merge or distill them — can also copy over bad habits the experts picked up while chasing reward. SCOPED addresses both problems by training experts independently, summarizing what each expert learned into an explicit written "skill," and using those skills only on the teacher side during distillation so the final student model needs no skill text at inference time.

Key Contributions

1. A decoupled training pipeline. Capability acquisition is separated from capability integration: four capability-specific experts (perception, OCR & chart, symbolic reasoning, multi-image reasoning) are optimized independently with capability-specific rewards, avoiding the gradient interference the authors measure in mixed-capability RL.
2. Reward-guided skill extraction. Rather than distilling raw expert predictions, the method splits each expert's sampled trajectories into successful and failed cohorts using a reward threshold, has the expert reflect on both, and synthesizes an explicit capability-specific skill that summarizes reusable solution patterns.
3. Asymmetric, skill-augmented on-policy distillation. The skill is given only to the teacher expert, not the student. The student is trained with a token-level reverse KL objective against these skill-regularized experts along student-generated responses, so it internalizes the procedure without needing skills at inference.
4. Empirical and mechanistic analysis. The paper provides gradient-conflict evidence for mixed-capability training (training reward, expert-direction alignment, pairwise gradient cosine similarity) and multi-level evidence for why skill conditioning helps (capability retention, trajectory consistency, attention to task-relevant visual regions).

Main Findings

• Best average among evaluated models with an 8B backbone. SCOPED-8B scores 73.13 average across the eight benchmarks, above Qwen3-VL-32B (72.01), Claude-Sonnet-4.6 (72.79), Claude-Opus-4.6 (72.86), GPT-5.2 (69.11) and GPT-5.4 (69.85).
• Consistent backbone gains. SCOPED-8B beats its Qwen3-VL-8B backbone on all eight benchmarks, with an average improvement of 5.83 points (67.30 to 73.13). SCOPED-4B improves Qwen3-VL-4B by 3.90 points on average (66.06 to 69.96).
• Distillation beats mixed RL and parameter merging. SCOPED is 3.82 points above Mix-RL (69.31) on average; SCOPED (Parameter Merging) reaches 70.89, and SCOPED without skill augmentation reaches 72.39, so skill augmentation adds further gains on top of plain on-policy distillation.
• Supervised fine-tuning hurts. Joint SFT falls below the zero-shot backbone (56.41 vs 67.30 average), which the authors read as evidence of cross-capability interference during joint fine-tuning.
• Mixed RL underperforms specialized experts on two capability groups. Mix-RL lags the corresponding specialized experts on symbolic visual reasoning and multi-image reasoning.
• Skills are capability-specific. Adding a skill to its matching expert consistently improves that expert's target capability, while effects on non-target capabilities are limited and occasionally negative — motivating explicit skill–expert matching.
• Measured gradient interference in shared-policy training. Independent capability RL holds a consistent average-reward advantage over Mix-RL for most of training; shared updates show weak and sometimes negative alignment with expert-guided directions; and recurrent negative pairwise gradient cosine similarities appear across capability pairs at different training stages.
• Skill conditioning changes expert behavior. Skill-conditioned experts show higher within-capability trajectory embedding similarity (measured with Qwen3-VL-Embedding-2B), higher attention hit rates on task-relevant visual regions in most upper decoder layers (largest gain in the final layer), and better preservation of expert-induced behavior in the distilled student than Mix-RL, and better than SCOPED without skill augmentation.

Methodology in Plain English

The pipeline has two stages.

Stage 1 — train experts separately. The training corpus is split into four capability subsets (perception & grounding; OCR & chart; symbolic reasoning; multi-image reasoning). Each expert starts from the same base model and is optimized only on its own subset with its own reward function, so no capability's objective competes with another's. Each trained expert then generates multiple rollouts per training example, producing a repository of both successful and failed behavior traces.

Stage 2 — turn experience into a skill, then distill. For each capability, trajectories are labeled successful or failed by the reward score against a threshold. The expert is prompted to reflect on the successful versus failed cohorts and then to synthesize that reflection into one explicit, written skill. During distillation, the student generates its own answers to each prompt; the matching expert — now shown its skill in addition to the input — scores those answers token by token, and the student is pushed to match the expert's next-token distribution via reverse KL divergence, averaged over capabilities. The student never sees the skill text, so at deployment it behaves as a normal single model. Training used EasyR1 for the RL stage and veRL for distillation on NVIDIA A800 GPUs.

Evaluation covers eight benchmarks (CountBenchQA, PixMo-Count, CC-OCR V2, ChartQA, VisualPuzzles, We-Math, MUIRBench, MMSI-Bench) with accuracy for all but OCR recognition, which uses micro-F1; the overall score is the arithmetic mean of the eight. The RL training corpus totals 37,999 samples after filtering (9,499 for perception & grounding, 9,500 each for the other three capabilities).

Why This Matters

Impact on research. The paper gives a concrete recipe for one of the field's practical bottlenecks — combining separately trained specialists into one deployable model — and backs it with diagnostic evidence (reward gaps, update-alignment and pairwise gradient cosine similarity) that mixed-capability RL is structurally limited rather than merely undertuned. Treating a distilled "skill" as teacher-side privileged information is also a reusable idea for filtering reward-hacking behavior out of expert supervision.

Real-world applications:

• Document and slide understanding assistants that must read text, parse charts, and reason about layout in one model.
• Visual question-answering products where a single model must handle counting, chart reading, puzzle-style reasoning and multi-image comparison instead of routing between specialists.
• Multi-image workflows such as comparing product photos, medical scans, or before/after images.
• Cost-sensitive deployment where a 4B or 8B unified model replaces an ensemble of experts or a much larger model.

Industry relevance. The result that an 8B model surpasses Qwen3-VL-32B and the tested closed-source models on the reported average is directly relevant to teams trading accuracy against inference cost. The two-stage design also fits existing infrastructure: the RL stage uses EasyR1 and the distillation stage uses veRL, both standard tooling in this space.

Future Directions

• Scaling the capability set. The study uses four capability groups; whether the approach holds with many more, finer-grained experts and correspondingly longer skill descriptions is not established here.
• Cheaper or automatic skill extraction. Skill quality depends on the expert reflecting over a large trajectory repository with an LLM; the paper does not report the cost or token budget of this step, and cheaper or more automated alternatives are an open question.
• Generality beyond the tested backbones. Only Qwen3-VL-8B-Instruct and Qwen3-VL-4B-Instruct are used as backbones, and training is limited to the constructed 37,999-sample corpus, so transfer to other model families or data regimes is untested.
• Residual reward hacking and evaluation breadth. The paper argues skills suppress reward-hacking biases but does not quantify how much biased behavior survives; broader held-out evaluation and direct measurement of reward-hacking artifacts would test the claim. Computational cost and variance across runs are also not reported.

Target Audience

Machine learning researchers and engineers working on multimodal post-training, RL for VLMs, or knowledge distillation and model merging. It is most useful to readers already comfortable with policy-gradient fine-tuning and distillation objectives, and to practitioners deciding whether to ship one unified model or an ensemble of specialists.

Authors’ abstract

Reinforcement learning (RL) can elicit specialized visual capabilities in vision-language models (VLMs), yet jointly optimizing heterogeneous objectives often introduces cross-capability gradient conflicts. Training experts independently mitigates this interference, but direct reward optimization may entangle specialized knowledge with reward-hacking biases. This paper proposes Skill-Decoupled On-Policy Expert Distillation (SCOPED), which trains capability-specific experts and extracts explicit skills through reward-guided reflection. During on-policy distillation, these skills guide expert supervision while remaining unavailable to the unified student, encouraging it to internalize procedural knowledge without explicit skills at inference time. Across eight multimodal benchmarks, SCOPED consistently outperforms the backbone and mixed-capability RL, demonstrating the effectiveness of skill-decoupled on-policy expert distillation for integrating independently acquired capabilities. Further analyses reveal interference during joint capability optimization and suggest that skill augmentation promotes more consistent expert trajectories, greater attention to task-relevant visual evidence, and better preservation of specialized expertise in the unified student. All code and data are available at https://github.com/NEUIR/SCOPED.