Video2Skill: From Streaming Experience to Reusable Embodied Skills

AI summary

Overview

• Research area: Embodied AI and vision-language modeling, specifically video-based skill discovery for robotic and human manipulation (submitted under arXiv category cs.CL, Natural Language Processing).
• Technical level: Intermediate. The paper is readable without a robotics background, but familiarity with Vision-Language Models, benchmark metrics, and supervised fine-tuning helps.
• Scope: The paper formulates Streaming Embodied Skill Discovery (SESD), introduces the Video2Skill benchmark built from HD-EPIC and RoboInter, evaluates 19 open-source VLMs under two inference paradigms, and tests three supervised fine-tuning strategies on two backbones.

What This Paper Is About

Vision-Language Models can describe individual manipulation events in language, but an embodied agent needs something more durable: a persistent library of reusable skills that shapes how it interprets later experience. The paper asks whether a model watching a stream of video can locate manipulation events in time, group events that share the same underlying physical transformation, and decide when to reuse an existing skill versus create a new one. The authors call this problem Streaming Embodied Skill Discovery (SESD) and build the Video2Skill benchmark to measure it.

Key Contributions

1. Problem formulation. The paper defines Streaming Embodied Skill Discovery (SESD), in which a model processes video in chunks, maintains a persistent skill library across videos, emits temporally grounded events expressed as symbolic skill calls, and carries unfinished events across chunks. Skills are semantic abstractions of observed transformations, not executable controllers.
2. A new benchmark. Video2Skill is assembled from two public sources spanning robot and human activity: RoboInter (robot tabletop videos drawn from DROID and RH20T) and HD-EPIC (long egocentric recordings of unscripted kitchen activity). Evaluation jointly measures event coverage, naming-invariant skill grouping, and creation–reuse decisions.
3. A 19-model evaluation. Nineteen open-source VLMs are tested under two paradigms: unified (the model jointly grounds events and updates the library in one pass) and factorized (the model first proposes events without library access, then performs text-only library updates). The paper identifies contrasting failure modes between the two.
4. Three supervision strategies compared. Qwen3.5-4B and LLaVA-OneVision-2-8B are fine-tuned under oracle-history SFT, Counterfactual Library-state Rebalancing (CLaRe), and on-policy correction using student-generated histories, with further analyses of video order, training frequency, and unseen transformations.

Main Findings

• Event description does not yield consistent skill abstraction. Across 19 open-source VLMs, many models group events at near-chance level, and scale does not consistently help. ARI remains at or below 27.0 across all configurations; ARI can be negative in some settings.
• Unified and factorized models fail in opposite directions. Unified models tend to overmerge, reusing existing schemas even when a new transformation appears, which produces high reuse recall but low creation recall and low pair precision. Factorized models tend to fragment, creating separate schemas for transformations they have already encountered. On HD-EPIC, switching Qwen3.5-2B from unified to factorized raises pair precision from 9.2 to 48.0 but reduces pair recall from 53.8 to 2.4; creation recall rises from 9.1 to 92.1 while reuse recall falls from 99.3 to 18.2.
• Scaling produces mixed effects. For unified Qwen3.5 on HD-EPIC, scaling from 4B to 27B increases pair precision from 12.5 to 26.9 but reduces pair recall from 32.7 to 22.9; ARI improves from 8.7 to 19.9. On RoboInter, the same comparison reduces ARI from 19.4 to 4.6. InternVL3.5 likewise shows no monotonic ARI improvement with size.
• Coverage and grouping quality must be interpreted together. Factorized LLaVA-OneVision-2-8B achieves the highest coverage on RoboInter (71.7%) but its pair recall is only 0.8 and ARI is 0.1. Its unified counterpart reaches pair recall of 100.0 but pair precision of 22.5 and ARI of 0.0. Unified MiniCPM-V-4.5 achieves the highest ARI on HD-EPIC (25.6) while covering only 7.5% of reference events, and unified Qwen3.5-0.8B attains creation recall of 50.0 with only 0.2% coverage.
• Supervision improves grouping but not the creation–reuse boundary. All three strategies improve ARI over zero-shot counterparts across both backbones, domains, and paradigms. Oracle-history SFT raises Qwen3.5-4B's pair precision on RoboInter to 67.6 and on HD-EPIC to 31.1, though gains should be read alongside coverage changes (coverage falls to 49.6 in the HD-EPIC setting). Trained unified models retain reuse recall of 97.0–98.7 while creation recall reaches only 19.1–40.0.
• Factorized training fixes fragmentation but leaves grouping weak. On RoboInter, on-policy correction raises LLaVA-OneVision-2-8B's pair recall from 0.8 to 25.1 and reuse recall from 29.1 to 94.6, while creation recall falls from 80.0 to 20.0. Improved reuse recall coexists with pair recall of only 5.5–31.4 in this paradigm.
• Libraries stall below half the reference size. Replaying the same 40 HD-EPIC clips as closed-loop episodes, all three video orders end with 21–22 schemas versus 46 reference classes. Under an innovation-heavy order, all 46 reference classes appear by the twentieth video, yet the model introduces only 20 schemas. Oracle-history SFT achieves a mean pairwise Jaccard similarity of 0.78 between final schema-name sets across orders, compared with 0.59 for the zero-shot model; CLaRe also reaches 0.78.
• Training gains are uneven across skills. High-frequency classes benefit consistently, rising from 19.0 to 34.9–40.0 on HD-EPIC and from 25.5 to 40.9–51.7 on RoboInter. Medium-frequency classes remain below 10 on both datasets, and low-frequency classes improve under oracle-history SFT and on-policy correction but not consistently under CLaRe. Performance does not vary monotonically with frequency.
• Unseen transformations are located but rarely created. Eleven HD-EPIC test classes comprising 59 segments are absent from task-specific training. Their temporal coverage after supervision is 57.6–66.1, exceeding seen-class coverage under every strategy, yet trained models assign a previously unused schema name at the first matched occurrence of only zero or one of the nine to ten covered unseen classes (0/11, 1/10, 1/9, 0/10 across the four conditions). Seen-class creation recall improves from 14.7 to 34.4–48.4. Oracle-history SFT raises unseen-class macro pair F1 from 8.6 to 25.5 yet creates a new name at only one of ten covered classes.

Methodology in Plain English

The researchers treat a video stream as an ordered sequence of chunks. At each step, the model receives the current chunk plus a state consisting of a persistent skill library and any unfinished event carried over from earlier. It then outputs new schemas, completed events, and an updated unfinished event. Each event carries temporal boundaries, a schema, and argument bindings drawn from seven roles: object, source, destination, instrument, direction, substance, and state. A schema has a free-form name, a definition, and typed argument slots. The library persists across videos and conditions later predictions, so earlier creation and reuse decisions shape how later events are interpreted.

To build the benchmark, the authors re-annotated source video rather than trusting original labels. For RoboInter they checked each candidate segment against the video, corrected annotation errors, and discarded labels not visually supported. They corrected 241 of the 12,429 verified calls across training and test splits, and retained RoboInter segments covering 55% of training video time and 82% of verified test video time. Verified calls were organized into 48 canonical transformation classes with definitions and aliases, forming the reference partition for naming-invariant evaluation; models never receive this class list. HD-EPIC provides 42 fully annotated training videos with 5,726 calls and 40 test clips from 29 held-out recordings with 1,137 reference segments; RoboInter provides 2,992 training videos with 5,032 verified calls and 132 test videos with 534 verified segments. The verified test splits contain 46 canonical classes in HD-EPIC and 15 in RoboInter, and median reference segments last 1.9 s and 4.5 s respectively. Train and test splits are disjoint at the source-video level.

Evaluation has three parts. Coverage matches each reference to its maximum-tIoU prediction, accepting matches at IoU ≥ 0.3, and reports the fraction of references recovered. Skill grouping is measured by pair precision and pair recall over pairs of matched events, plus Adjusted Rand Index, which corrects for chance; grouping is invariant to consistent renaming but synonymous strings remain distinct. Creation and reuse decisions are evaluated by traversing matched events in episode order and marking first occurrences of reference classes and predicted names, reporting Create recall, Reuse recall, and decision accuracy.

For the training experiments, all three strategies share a supervised objective over observation, prior state, and target. Oracle-history SFT uses states produced by replaying annotated episodes in order. CLaRe augments those histories with edited states while holding the observation fixed, removing required schemas, adding semantically distinct lexical distractors, removing unrelated entries, and, in the factorized variant, inserting equivalent schemas and intervening on unfinished-event state. On-policy correction rolls out the student over ordered training episodes and obtains teacher-labeled targets for states the student actually reaches, so supervision covers accumulated student errors. Both backbones are fine-tuned under both paradigms, with the factorized variant keeping a frozen event proposer and training only the text-only meta-policy.

Why This Matters

The paper reframes an assumption behind skill-based planning: that the set of skills is already known. It provides a benchmark and a measurement protocol for the inverse problem of recovering skills from observation, and shows that current VLMs describe events far better than they organize them into stable abstractions. The finding that supervision improves grouping while leaving novelty recognition unresolved identifies a specific bottleneck for future work rather than a general capability gap. The paper also argues that coverage, partition agreement, and creation–reuse decisions must be interpreted jointly, since high scores on one axis can reflect a small matched subset.

Real-world applications motivated by this work include:

• Robot learning from demonstration video, where skill-annotated experience could supervise future embodied agents instead of requiring each behavior to be learned from scratch.
• Household and kitchen assistance, since the benchmark spans both robot tabletop manipulation and egocentric kitchen activity, where wiping a table, a plate, or a window instantiate the same wiping skill.
• Training-data generation, producing symbolic skill labels and temporal boundaries from passive video at scale.
• Human activity understanding, where naming-invariant grouping of transformations could support procedural analysis of unscripted recordings.

Industry relevance: teams building general-purpose manipulation policies, robot foundation models, or data engines for embodied AI would need exactly the kind of online skill library maintenance this paper evaluates, and the reported failure modes, library stalling and missed novelty, are practical blockers for deploying such libraries rather than academic curiosities. The authors explicitly note that schemas describe observed physical transformations and that their utility for downstream planning and executable control still requires evaluation.

Future Directions

• Recognizing when the existing repertoire is insufficient. The paper identifies this as the central challenge: trained models consolidate familiar skills but rarely expand the library, and transformations unseen in training are located in time but almost never given a new skill.
• Extending beyond the current domains. The stated limitations note that Video2Skill covers robot manipulation and egocentric kitchen activity, while broader settings such as outdoor activities remain unexplored.
• Connecting discovered schemas to downstream planning and control. The paper leaves open whether these symbolic schemas are useful for planning or executable control.
• Reducing uneven gains across the skill repertoire. Medium-frequency classes remain below 10 on both datasets and low-frequency results vary by strategy, so how to learn balanced abstractions across frequency bands remains unresolved.

Target Audience

Researchers and practitioners working on embodied AI, robot learning, and video-language models will benefit most, particularly those interested in skill discovery, long-horizon planning, and evaluation of streaming video understanding. It is also relevant to teams building training-data pipelines for manipulation agents, and to readers studying novel versus known concept recognition in multimodal models. The paper is accessible at an intermediate level: the task formulation and metrics are clearly specified, but some background in VLMs and clustering-style evaluation metrics is assumed.

Authors’ abstract

Manipulation behaviors vary widely across objects and scenes, but they share a small set of reusable skills, and planning with these skills helps embodied agents generalize to new tasks. Yet an agent can only plan with skills it knows. Recovering skills from observed experience, the inverse of planning, builds this knowledge over time and yields skill data for training future agents. Vision-Language Models (VLMs) describe individual manipulation events well, but can they organize a stream of events into reusable skills? We formulate this problem as Streaming Embodied Skill Discovery (SESD): a model watches videos in sequence and maintains a persistent skill library that shapes its later decisions. To systematically measure this ability, we introduce Video2Skill, a benchmark that covers robot tabletop manipulation and human kitchen activity and tests three core capabilities: (i) locating manipulation events in time, (ii) grouping events of the same transformation, and (iii) deciding when to reuse an existing skill or create a new one. Across 19 open-source VLMs, many models group events at near-chance level, and scale does not consistently help. Their errors depend on how perception and library updates are coupled: joint models merge distinct transformations into one skill, while models that update the library from text descriptions duplicate recurring ones. Supervised fine-tuning, including our counterfactual library-state rebalancing (CLaRe), improves grouping but exposes a deeper bottleneck: trained models consolidate familiar skills yet rarely expand the library. Their libraries stall below half the reference size, and transformations unseen in training are located in time but almost never given a new skill. Recognizing when existing skills are insufficient thus emerges as the central challenge.