In-Context Robot Learning with VLM Agents

AI summary

Overview

Research area: Embodied AI and robot learning, at the intersection of vision-language models (VLMs), in-context learning, and closed-loop robot control.

Technical level: Intermediate. The paper is readable without deep robotics background, but familiarity with manipulation, inverse kinematics, and VLM/agent terminology helps.

Scope: The paper introduces GPT-Policy, a framework that connects a fixed, off-the-shelf VLM to real robot tools and asks how far in-context information (demonstrations, goal images, interaction history, human feedback) can drive robot behavior without any training or parameter updates.

What This Paper Is About

Robot policies are trained on finite datasets, so they inevitably encounter situations their training never covered. Humans handle this by watching others, reading examples, and learning from consequences — a capability known as in-context learning. This paper asks whether general-purpose commercial VLMs, which already learn from context in language and vision tasks, can do the same for robots: take a demonstration, a goal image, or a live interaction and turn it into executable, verifiable physical actions on a new task and a new initial state.

Key Contributions

1. A general-agent framework, GPT-Policy, for in-context robot learning. It connects a fixed VLM to robot tools through a shared context-to-action interface, with no gradient updates and no persistent task-specific parameter changes. The VLM policy stays frozen throughout execution.

2. A context compiler that structures heterogeneous test-time inputs. Goal images, human demonstration videos, teleoperated robot videos, time-aligned recorded action sequences with measured robot states, and online interaction history are all normalized into a common interleaved image-and-text input format.

3. A constrained execution layer with verification and feedback. VLM tool requests are resolved into Cartesian pose targets, interpolated (linear position plus SLERP orientation), solved through inverse kinematics with residual tolerances, timed under velocity/acceleration/jerk limits via Ruckig, and executed with measured settling feedback returned to the agent for replanning.

4. A systematic empirical study across five context families on real robots, covering cross-embodiment imitation, contact-sensitive manipulation, goal-image following, active exploration, and human-robot interaction — with matched model comparisons and context ablations.

Main Findings

• Human video demonstrations help without robot action labels. On "Pick Red Towel," success went from 0/3 to 2/3, decisions dropped from 96.3 to 76.7, and time from 24.6 to 18.9 minutes. "Pick Up Notebook" showed the same pattern (0/3 to 2/3; 94.0 to 66.7 decisions; 24.6 to 16.1 minutes). The agent generated robot-specific motion targets purely from observing a person.

• Aligned action references give the largest gains on contact-sensitive tasks. For "Unscrew Bottle Cap," success rose 0/3 (no context) to 2/3 (video) to 3/3 (video plus actions). For "Remove and Reinsert Plug," video alone did not help (0/3), but video plus actions reached 2/3. The authors attribute this to denser temporal information: 205 action samples for 13 keyframes in bottle opening, versus sparse keyframes that leave intervening motion ambiguous.

• A single goal image reliably specifies spatial layouts. The Target Image condition achieved 3/3 success on both "Arrange T Shape" and "Arrange Fruit," conveying object identity, relative position, and spacing more precisely than text descriptions.

• Self-interaction history supports intermediate subgoaling. With retained history, the agent scored 3/3 on "Lemon to Pink Plate" and "Movable Exploration." It autonomously removed a towel to uncover a hidden plate, and avoided obstacles while searching during mobile exploration — behavior consistent with reasoning about information-gathering as a subgoal.

• Online human interaction enables coordination and strategy. The agent achieved 3/3 on "Tic-Tac-Toe" (wins and draws counted as success) and "Pointed Fruit Pickup," tracking turn order and selecting optimal moves.

• Good planning does not guarantee good execution. The paper repeatedly stresses that better task understanding and action selection do not ensure precise contact, reliable outcome verification, or physical safety. The authors observed collisions between the two arms during manipulation.

• Model comparison is suggestive, not conclusive. On red towel pickup, GPT-6 Astra's task progress rose from 55% to 100% with human video, with roughly 35.6% shorter runtime and 58.9% lower estimated token usage. Fable 5.1 and Kimi K3 reached only 30% and 20% progress with fewer resources. The authors explicitly note this does not establish a model ranking, and that task progress is distinct from success rate.

• Decisions take time. VLM-driven control is slow and expensive per decision, suggesting specialized VLA/world-action models may retain an advantage in fast low-level control while VLM agents handle reasoning and replanning.

Methodology in Plain English

The researchers built a loop with three parts. First, a context compiler takes whatever task-relevant information is available — a goal image, a demonstration video, a video paired with recorded robot trajectories, or a running log of what the agent has already done and what the human has said — and packs it into an interleaved sequence of labeled images and text.

Second, a frozen VLM reads that context along with the current camera views and robot state, and outputs a structured tool request: either move the arm to a target pose (move_to), follow a short sequence of poses (move_eef_chunk), or change the gripper opening (set_gripper). Targets are expressed in a calibrated base frame with positions in meters and orientations as quaternions.

Third, a constrained controller makes the request physically real. It interpolates a smooth path between targets, solves inverse kinematics seeded from the current joint configuration, checks that position and orientation residuals stay within tolerance (0.002 m and roughly one degree), applies velocity, acceleration, and jerk limits, and dispatches timed joint references. The robot then reports measured state, endpoint errors, and whether it settled. That feedback, or a rejection reason, goes back into the context for the next decision.

Evaluation compares a "None" condition (instruction and current observations only) against each context condition. Every episode starts from a reset scene and is counted successful only if the final scene meets task-specific geometric and semantic criteria. Each experiment runs three times. Metrics are success rate (S/T), decision count, and execution time — with decisions and time averaged over all trials including failures. Unverified completion requests from the model are explicitly not treated as physical success.

Why This Matters

Impact on research. The paper reframes robotic in-context learning as a question about where adaptation succeeds and where it breaks down, rather than a binary capability claim. It argues that off-the-shelf general models already contribute real task understanding, and that the open problem is dependable execution — precise contact, outcome verification, and safety — not perception or planning. It also provides a shared harness so different models and context types can be compared under identical execution rules.

Real-world applications:

• Warehouse and logistics picking, where new SKUs and arrangements appear constantly and collecting demonstrations for each is impractical.
• Deformable and contact-rich assembly, such as inserting plugs back into sockets or opening containers without tipping them — tasks where action references measurably help.
• Assistive and service robots in homes, where a resident can point at an object or show a procedure once instead of programming the robot.
• Collaborative tasks and games, where turn-taking, human intent, and evolving board or scene state must be tracked live.

Industry relevance. The work speaks directly to the division of labor between general VLM agents and specialized vision-language-action policies. VLM agents are strong at reasoning, adaptation, and replanning but slow and token-expensive per step; VLA and world-action models are fast but narrow. The paper's System 1 / System 2 discussion, plus its explicit cost accounting (runtime and estimated token usage), offers a concrete framing for companies deciding where to place general models in a robotics stack.

Future Directions

• A dedicated physical safety layer. The authors observed repeated inter-arm collisions and call for trajectory checking across both arms jointly, separation and contact monitoring during execution, and the ability to interrupt unsafe commands independently of the VLM — with collisions and near misses reported alongside task success.
• Contact-aware execution harnesses. Building on the existing rejection feedback, future systems should expose grasp stability and post-contact object motion, using slip detection, force-aware limits, and local recovery instead of requiring the agent to reason through every correction.
• Hierarchical System 1 / System 2 control. Pairing a deliberative VLM agent with a fast VLA controller for pose refinement and bimanual coordination, with an explicit answer to when local feedback should trigger replanning.
• Compositional context for long-horizon tasks. Testing whether an agent can compose subskills from multiple demonstrations into a new sequence, retaining completed subgoals and discarding obsolete context rather than replaying an entire trajectory.
• In-context adaptation to physical dynamics. Using recent interactions to update predictions of friction, compliance, and object response before contact failures accumulate — adapting to changing physics, not just to a new task description.
• Mobile manipulation through active perception. Letting the agent choose where to look and stand, not just how to grasp, with persistent spatial memory and coordinated base-arm control.

The paper is also explicit about its own limits: small task series, selected platform and context conditions, incomplete ablations, and unobserved pretraining that prevents causal or novel-skill claims.

Target Audience

Robotics and embodied AI researchers evaluating whether general-purpose foundation models can substitute for task-specific policy training. It is also valuable for VLM and agent developers looking for a concrete, physically grounded application domain with measurable failure modes, and for industry practitioners deciding how to allocate work between general reasoning agents and specialized low-level controllers. Graduate students entering manipulation or robot learning will find the setup and metric definitions useful as a template.

Authors’ abstract

Enabling robots to adapt to unfamiliar environments as readily as humans remains a moonshot goal of embodied AI. No finite collection of demonstrations can cover every task and situation a robot will encounter, making the ability to learn from context at deployment essential for generalization. Such in-context learning (ICL), however, remains largely beyond the reach of existing robotic policies. The broad agentic capabilities of commercial vision-language models (VLMs), such as GPT-6 Astra, raise a compelling question: can these models learn from demonstrations, examples, and interaction feedback, then translate that information into executable and verifiable robot behavior from a new initial state without gradient updates or persistent changes to task-specific parameters? We introduce GPT-Policy, a general-agent framework for in-context robot learning. GPT-Policy integrates a context compiler that preserves task-relevant visual transitions, a VLM that proposes robot-tool actions, and a constrained controller that verifies and executes each action and reports its outcome. We evaluate its reliability and limitations through task success and efficiency metrics, matched comparisons across models, and controlled context ablations. In real-robot trials, human video demonstrations improve task completion even without robot action labels, while aligned action references yield further gains on contact-sensitive tasks. These findings position GPT-Policy as a step toward robot adaptation through in-context learning, providing an empirical foundation for translating the general-purpose capabilities of VLMs into physical behavior and clarifying the challenges that must be overcome for reliable deployment.