ViSkill: Reinforcing VLM Agents with Evolving Visual-Native Skills

AI summary

Overview

Research area: Reinforcement learning for Vision-Language Model (VLM) agents, specifically skill-augmented agents that reuse past experience in visually grounded decision-making tasks.

Technical level: Advanced. The paper assumes familiarity with reinforcement learning (PPO, advantage estimation, reward shaping), VLM agent architectures, and skill-library / memory-augmented agent designs.

Scope: ViSkill is a framework that stores reusable experience as images ("visual skill cards") rather than text, retrieves them by geometric similarity, and couples skill accumulation with policy optimization in a closed loop, evaluated on Sokoban, FrozenLake, and PrimitiveSkill.

What This Paper Is About

Agents that reuse skills from past successful episodes usually write those skills down as text, which flattens spatial layouts and action-state relationships into language and loses important geometric information. ViSkill instead stores each skill as a composite image built from annotated trajectory frames plus a distilled strategy description, and lets the skill library and the policy improve each other during training rather than being constructed separately. The goal is to make VLM agents solve visually grounded planning and manipulation tasks faster and more reliably by preserving visual structure in the reusable experience itself.

Key Contributions

1. A visual-native skill representation. ViSkill represents reusable procedural experience as composite "visual skill cards," each combining annotated trajectory frames with a policy-distilled strategy description in a single image that is fed directly into the VLM's visual context.

2. Geometry-aware retrieval plus a skill-guided reinforcement objective. Skills are retrieved by geometric similarity to the current task configuration (not language matching), and a skill-guided reward gives bounded partial credit to failed trajectories that still make skill-relevant progress, forming a closed feedback loop with the library.

3. Online distillation with quality and novelty gates. Successful trajectories that pass a trajectory-length quality gate and a library-coverage novelty gate are distilled back into the library as new visual skills, with least-useful skills evicted when the library reaches capacity.

4. Empirical validation across 2D and 3D environments. ViSkill achieves an overall success rate of 0.89, rising to 0.91 with cold-start initialization, outperforming all evaluated proprietary and open-source baselines and converging faster than standard PPO.

Main Findings

• Overall performance: ViSkill reaches an overall success rate of 0.89, exceeding the best evaluated baselines. VAGEN scores 0.80 overall and Atlas-VA scores 0.87 overall. ViSkill plus cold-start reaches 0.91, four percentage points above Atlas-VA.

• 2D environments: ViSkill obtains 0.82 on Sokoban and 0.84 on FrozenLake, compared with VAGEN at 0.79 and 0.74, and AtlasVA at 0.79 and 0.83. Cold-start raises these to 0.88 on Sokoban and 0.85 on FrozenLake.

• 3D manipulation: ViSkill achieves a perfect average of 1.00 on PrimitiveSkill (Place, Stack, Drawer, Align, Swap), matching AtlasVA and outperforming VAGEN by 12 percentage points. PrimitiveSkill remains saturated at 1.00 with cold-start.

• Faster convergence: Training curves show ViSkill consistently converges faster and reaches higher success rates than PPO across all three environments, with cold-start providing the largest early-stage gains on Sokoban and PrimitiveSkill.

• Visual representation matters: Replacing visual skill cards with Text-Skill drops Sokoban from 0.82 to 0.63 (-0.19) and FrozenLake from 0.84 to 0.59 (-0.25). OCR-Skill falls to 0.68 on Sokoban (-0.14) and 0.62 on FrozenLake (-0.22), even though it renders text back into image form.

• Component ablations: Removing Skill-Guided Reward costs -0.05 on Sokoban and -0.04 on FrozenLake. Removing Strategy (the high-level strategic text) costs -0.10 on Sokoban and -0.03 on FrozenLake. Removing Optimization entirely, keeping retrieval and distillation but disabling training, causes the largest degradation at -0.53 on Sokoban and -0.51 on FrozenLake — evidence that policy optimization and skill learning need to co-evolve.

• Reward weight sensitivity: Performance peaks at λ = 0.8 (0.82 Sokoban, 0.84 FrozenLake). λ = 0.0 yields 0.77 and 0.80; λ = 0.6 yields 0.71 and 0.79; λ = 1.0 yields 0.77 and 0.81.

• Retrieval threshold: Success peaks at δ_r = 0.80 on both environments (0.82 Sokoban, 0.84 FrozenLake). A lower threshold (0.75) admits poorly matched skills and degrades performance; a higher threshold (0.85) is overly restrictive and reduces retrieval frequency.

• Diversity threshold: δ_d = 0.90 produces the best success rate on both environments. δ_d = 0.85 is too permissive, growing the library rapidly while average utility declines; δ_d = 0.95 keeps utility high but severely limits library growth and coverage.

• Cold-start budget: Increasing the cold-start ratio γ improves performance on both environments. Sokoban saturates at γ = 0.6 with a 1.00 success rate; FrozenLake continues improving to 0.89 at γ = 1.0.

• Token efficiency: Success rate increases monotonically with skill-card resolution, but even the most compact variant, ViSkill (0.2), surpasses both Text-Skill and OCR-Skill on Sokoban while consuming the fewest tokens, and on FrozenLake ViSkill (0.2) and ViSkill (0.4) also beat both baselines with substantially lower token consumption.

• Library transfer: Applying an evolved library to the unoptimized base model gives consistent gains over a no-skill baseline. Libraries distilled with joint policy optimization reach 0.41 on Sokoban and 0.39 on FrozenLake, outperforming libraries distilled without optimization.

Methodology in Plain English

The approach is a three-stage loop repeated for each task.

Retrieval. At the start of an episode, a task-specific feature extractor converts the initial scene into a geometric descriptor — capturing things like relative positions, distances, and obstacles. Each stored skill is scored by a combination of geometric similarity to that descriptor and the skill's historical utility, using a weight α = 0.9 on similarity. The best-matching skill is retrieved only if its composite score clears a threshold δ_r = 0.8; otherwise the agent proceeds unaided.

Interaction. The retrieved skill card — a single image containing annotated trajectory frames, action labels, and a rendered strategy description — is appended to the VLM's visual context. The policy then acts conditioned on the task, the interaction history, and this card. After the episode, the environment returns a binary success indicator that updates the retrieved skill's utility via an exponential moving average (β = 0.1) and increments its usage count.

Distillation. Successful trajectories enter the library only if they pass two gates: the trajectory length must not exceed a solver-derived budget, and the best existing match score for the current configuration must fall below the novelty threshold δ_d = 0.9. The policy then summarizes its own successful interaction into a reusable strategy description, which is rendered together with annotated frames into a new skill card. When the library hits capacity, the skill with the lowest utility-frequency score is evicted.

Reward shaping. For failed trajectories where a skill was retrieved, an auxiliary reward measures how far along the reference trajectory the agent got, relative to the executed trajectory length. This term is added with weight λ = 0.8 and is bounded so it never exceeds the task-completion bonus, preserving global reward ordering.

Optimization. The resulting rewards feed into PPO, using a masked clipped objective that excludes observation tokens from the policy-gradient term. Policy improvement and library growth happen concurrently. An optional cold-start mechanism pre-populates the library with solver-derived seed skills selected via farthest-point sampling, sized as a fraction γ = 0.2 of the converged library size from a preliminary run; after training starts, all skills come from the policy itself.

Setup. The policy backbone is Qwen2.5-VL-3B-Instruct, following VAGEN's PPO configuration with a rollout batch size of 128, a PPO mini-batch size of 32, and actor and critic learning rates of 1×10⁻⁶ and 1×10⁻⁵. Sokoban and FrozenLake are trained for 300 steps on 4× H100 GPUs; PrimitiveSkill is trained jointly for 60 steps on 2× H100 GPUs with an additional H100 dedicated to the ManiSkill environment server. GPT-5.4-mini serves as the external model for cold-start strategy distillation.

Why This Matters

Impact on research. The paper separates two failure modes in skill-augmented agents — a representation gap (text flattens spatial structure) and a learning gap (skills are built separately from policy optimization) — and addresses both in one framework. The ablation showing a -0.53 drop when optimization is removed is a strong argument that skill libraries should be learned jointly with the policy rather than assembled offline. The result that visual skill cards still beat text and OCR baselines at 0.2 resolution also suggests the benefit comes from preserving raw visual trajectory structure, not merely from having an image.

Real-world applications (as suggested by the task domains studied):

• Robotic pick-and-place and stacking, where object coordinates and spatial arrangements matter more than verbal instructions (PrimitiveSkill's Place, Stack, and Align tasks).
• Drawer or cabinet manipulation and object rearrangement in cluttered environments, where a retrieval-by-geometry approach fits scenes that look similar but require different action sequences.
• Warehouse or grid-based navigation and logistics routing, analogous to Sokoban's push-objects-to-targets planning.
• Safe path planning in constrained spaces with hazards to avoid, analogous to FrozenLake's goal-reaching with holes as forbidden cells.

The paper notes it is conducted exclusively in simulated environments with no human participants, personal data, or physical systems, and that real-world deployment would require task-specific safety constraints and human oversight.

Industry relevance. Companies building embodied AI or GUI/computer-use agents could apply the pattern of an evolving, image-based skill library to avoid re-learning similar spatial procedures across episodes. The token-efficiency finding matters commercially: compact skill cards that still outperform text descriptions reduce the visual context cost per decision, which directly affects inference expense in deployed VLM agents.

Future Directions

• Generalization beyond VAGEN-supported environments. The authors state that whether visual skill cards provide consistent benefits in more complex or open-ended settings remains to be validated.
• Handling disjoint solution paths. The skill-guided reward assumes useful partial progress reaches states represented by the retrieved skill, which may give limited guidance when valid solutions traverse disjoint state paths.
• Reducing dependence on external models and solvers. Cold-start initialization currently uses offline task-specific solvers and an external model (GPT-5.4-mini) for seed strategy descriptions, and trajectory-length budgets rely on breadth-first search or scripted oracle plans. Whether this can be eliminated while keeping the same gains is an open question, though the γ = 0 setting provides a trade-off knob.
• Transfer to unoptimized or different policies. Library transfer to the base model yields 0.41 on Sokoban and 0.39 on FrozenLake — meaningful gains over no-skill but far below the 0.89 overall of the trained system, leaving room to study how skills transfer across model scales and architectures.

Target Audience

Researchers and graduate students working on VLM agents, reinforcement learning for embodied and multimodal decision-making, and memory- or skill-augmented agent architectures. It is also relevant to practitioners building interactive agents who need to reduce sample cost or inference token cost through reusable experience, and who are comfortable reading PPO-based training details and ablation tables. Readers without an RL background will find the framework description accessible, but the reward design and optimization sections assume prior familiarity.

Authors’ abstract

Skill-augmented agents improve sample efficiency by distilling successful trajectories into reusable strategies. Yet most existing approaches remain text-centric, linearizing spatial layouts and action-state correspondences into language that loses critical geometric structure. Recent efforts have begun incorporating visual evidence, but construct and update skills separately from policy optimization, leaving their mutual improvement underexplored. We propose ViSkill, a visual-native skill learning framework that encodes successful interactions as composite visual skill cards directly accessible to VLM agents. Retrieved skills guide both inference and reward shaping, while successful trajectories are distilled back into the library, forming a closed feedback loop in which skill accumulation and policy improvement reinforce each other. An optional cold-start mechanism further accelerates early-stage learning. Evaluated on Sokoban, FrozenLake, and PrimitiveSkill, ViSkill achieves an overall success rate of 0.89, rising to 0.91 with cold-start initialization, outperforming all evaluated proprietary and open-source baselines while converging faster than standard PPO. Our code is available at https://github.com/ZJU-REAL/ViSkill.