SHRIMP: Iterative Refinement of Robot Task Plans

AI summary

Overview

Research area: Human-robot interaction and end-user robot programming, specifically natural language interfaces for robot task planning and physics-based plan simulation.

Technical level: Intermediate. The interface concepts and study design are accessible to a general reader, but the paper assumes some familiarity with robot primitives, inverse kinematics, large language models, and mixed-effects analysis of user-study data.

Scope: The paper introduces SHRIMP (Simulation-driven Human-in-the-loop Refinement Interface for Manipulation Planning), a system that lets non-expert users generate a hierarchical robot primitive plan from a natural language prompt and iteratively refine it inside a physics-based simulation before running it on a physical bimanual robot, evaluated through a 35-participant within-participants ablation study.

What This Paper Is About

Collaborative robots are entering manufacturing, agriculture, healthcare, and other domains where the people who operate them usually lack robotics expertise, and current programming tools such as teach pendants require low-level joint-position specification. Natural language lowers that barrier, but it is inherently ambiguous (a phrase like "move the object left" does not specify distance, reference frame, or even which object) and LLM- or VLA-based planners behave as black boxes, so users cannot validate what the robot will actually do before it does it. SHRIMP addresses both problems by decomposing an LLM-generated plan into inspectable high-level and low-level primitives and grounding every revision in a physics simulation of the real robot setup, so users can iterate safely before execution.

Key Contributions

1. System contribution. SHRIMP enables physics-grounded robot plan iteration, letting users refine generated plans through multi-level language and primitive editing (re-prompting, re-ordering, adding primitives, or editing primitive parameters such as end-effector poses), which the authors argue increases transparency and control during refinement.

2. Primitive library and plan verification layer. The authors developed a two-level primitive vocabulary — six low-level primitives (home, move_to_pose, grasp, release, tilt_in_hand, wait) and three high-level primitives (pick, pour, pick_and_place) — plus three automatic validation checks that verify grasp targets against tracked object poses, insert retract motions to prevent arm-to-arm collision, and raise trajectory height to prevent robot-object collision.

3. Empirical contribution. An ablation within-participants study (N = 35) comparing the full system (Plan+Edit) against a plan-viewing-only condition (Plan-Only) and a black-box baseline (No-Plan), measuring perceived autonomy, transparency, usability, task load, completion time, and task success.

4. Design and research implications. The authors report implications for how to improve end-user transparency and control in robot planning interfaces based on their evaluation.

Main Findings

• Perceived control improved over the black-box baseline. Participants reported significantly higher perceived autonomy for Plan+Edit (M = 3.50, SD = 0.84) than for No-Plan (M = 3.15, SD = 0.99), p = 0.007. Qualitative data showed two explanations: parameter editing gave non-ambiguous, precise control, and manual adjustment was sometimes easier than describing a target location in words.

• Transparency improved over the black-box baseline. TOROS transparency scores were significantly higher for Plan+Edit (M = 4.77, SD = 1.27) than No-Plan (M = 4.34, SD = 1.34), p = 0.026. Nineteen participants explicitly said viewing the primitives increased their understanding of the plan.

• Viewing the plan alone was not sufficient. There were no significant differences between Plan-Only and No-Plan, nor between Plan-Only and Plan+Edit, for perceived autonomy or transparency. The authors conclude that users need both the viewing and the interaction features present in Plan+Edit to achieve higher autonomy and transparency.

• Higher control came at a task-load cost. NASA-TLX task load was significantly higher for Plan+Edit (M = 37.86, SD = 16.30) than Plan-Only (M = 29.92, SD = 20.24), p = 0.002, which participants attributed to the added plan information and editing features.

• The task-load cost depended on task difficulty. The Clean the Table task had lower task load than Prepare a Snack (p < 0.001) and Set the Table (p = 0.043), and higher task success than Set the Table (p = 0.024), across all conditions. The task-load cost between conditions was only significant for Clean the Table, suggesting that for easier tasks the extra Plan+Edit features introduced unnecessary complexity.

• Usability and task success did not differ between conditions. There were no significant differences between conditions for SUS usability ratings or task success. The hypothesis that Plan+Edit would be rated more usable was not supported by the reported results.

• Users behaved more iteratively with the full system. Participants revised their plans more often in Plan+Edit (M = 6.88, SD = 4.30) than Plan-Only (M = 3.80, SD = 2.53) or No-Plan (M = 3.68, SD = 1.61), and simulated their plans more often in Plan+Edit (M = 7.61, SD = 4.07) than No-Plan (M = 4.65, SD = 2.37) or Plan-Only (M = 4.31, SD = 2.94).

• Three prompt iteration strategies emerged. Twenty-one participants used a Stepwise strategy (testing subtasks one at a time, then combining), 10 used a Cumulative strategy (adding a subtask each iteration), and 28 used a Oneshot strategy (specifying the full plan first, then editing specific details). Eight participants used Oneshot to update spatial parameter information instead of using the parameter editing available in Plan+Edit.

• LLM non-determinism caused frustration. Six participants reported that similar prompts produced different plans, which the authors attribute to LLM non-determinism and to primitive reordering that produced awkward joint configurations. Usability was moderately positively correlated with task success (r = 0.429, p < 0.001).

• Learning effects appeared across trials. Task success was moderately correlated with trial progression (r = 0.337, p < 0.001), and 22 participants said they felt more proficient and comfortable as trials progressed.

• Control and usability were linked. Perceived autonomy scores were positively correlated with usability scores (r = 0.635, p < 0.001).

Methodology in Plain English

The researchers built an end-to-end system around a physical robot setup: two 7-degree-of-freedom Franka Emika Panda arms, each fitted with a 12-DOF Tesollo 3 Finger gripper, with a single RGB-Depth d435 Realsense camera using YOLO11 to localize objects. A digital twin of this setup runs in the Isaac Sim engine, sharing the same motion control layer as the physical robots so that simulated behavior corresponds to real behavior. Inverse kinematics uses RangedIK, and a PD torque controller with Coriolis compensation drives both simulated and physical Franka arms (with additional gravity compensation added in simulation, and torque approximated via duty cycle for the physical Tesollo grippers). Simulation and its control loop run at roughly 200 Hz versus 1000 Hz for the Franka and 500 Hz for the Tesollo, compensated with higher K_p gains.

Users first track the scene to replicate physical objects in simulation, type a task prompt, and receive an automatically generated hierarchical plan from GPT 5.1, which is given the robot configuration, the primitive list, all object positions, and the prompt. Each generated plan passes three validation checks before being shown. Users then execute the plan in simulation, step through it at high or low level, and revise it by re-prompting, re-ordering or adding primitives, or editing parameters (a red arrow in simulation marks any pose being edited). Revised plans are stored under the parent plan in a Task History so users can revert.

The evaluation was an ablation within-participants experiment with N = 35 participants (ages 18–65+; 14 male, 19 female, 2 preferred not to answer), lasting approximately 50 minutes with $15 USD compensation, approved by the University of Wisconsin–Madison IRB. Each participant completed one of three tabletop tasks (Set the Table, Prepare a Snack, Clean the Table) across three counterbalanced interface conditions: Plan+Edit (full system), Plan-Only (only high-level primitives visible, no parameter editing, no expanding or stepping), and No-Plan (no plan visible, only natural language prompting). Participants had up to 10 minutes to plan per trial. Measures were the Transparency Of Robots Scale (α = 0.89), the Perceived Autonomy Scale (α = 0.87), the System Usability Scale (α = 0.90), NASA-TLX (α = 0.78), trial completion time, and binary subtask success summed into a 0–3 score. Analysis used linear mixed effects models via statsmodels with participants as a random effect, Pearson and Spearman correlations via scipy, and Cronbach's Alpha via pingouin. Qualitative data from prompts and open-ended questions were analyzed with Thematic Analysis, iterating a codebook until inter-rater reliability reached at least 90% agreement.

Why This Matters

Impact on research. SHRIMP pushes back on the common pattern in LLM- and VLA-based robot planning systems of treating the planner as a black box with no interface for inspection or editing, and on systems that verify plans only through text or code. It shows that hierarchical primitives combined with physics-grounded simulation give users a place to catch problems — such as unintended object motion or incorrect placement — that trajectory-only visualizations cannot reveal. It also provides evidence that exposing a plan is not the same as enabling interaction with it: the Plan-Only condition produced no significant gains over No-Plan.

Real-world applications.

• Manufacturing and automotive settings where line operators need to reprogram cobots on the fly without robotics expertise.
• Agriculture, where field conditions and task variations change frequently.
• Healthcare and assistive settings, where tasks such as fetching, pouring, or arranging objects require precise, safe behavior around people.
• Tabletop service and kitchen tasks, the specific domain tested (setting a table, preparing a snack, cleaning a table), where placement precision matters.

Industry relevance. The system is built entirely from commercially available components — two Franka Emika Panda arms, Tesollo grippers, a RealSense d435 camera, Isaac Sim, ROS2 Jazzy, FastAPI, and WebRTC — and runs across two desktop computers, one with a real-time kernel patch for control and one with an NVIDIA GPU for simulation and the interface. The paper's evidence that richer editing features raise task load on simple tasks, but not on complex ones, is directly relevant to teams deciding how much plan detail to expose in a deployed product.

Future Directions

• Reducing the task-load cost of full plan editing. Task load was significantly higher for Plan+Edit than Plan-Only, and that cost was concentrated in the easiest task (Clean the Table). Determining how to make the full feature set valuable on simple tasks, rather than unnecessary complexity, remains open.

• Communicating LLM non-determinism to users. Six participants were frustrated when similar prompts produced different plans, and they could not see the underlying cause (primitive reordering leading to awkward joint configurations). Designing interfaces that surface this cause rather than leaving users confused is an open problem.

• Closing the usability gap. The paper reports no significant usability difference between conditions and no significant task-success difference, while usability correlated with task success (r = 0.429). Understanding what would make the plan-editing experience measurably more usable is unresolved.

• Generalizing beyond tabletop kitchen tasks. The evaluation covered three tabletop tasks with a fixed primitive vocabulary and a bimanual setup in one room. Whether the primitive-plus-simulation approach holds for broader task domains, mobile robots, or more varied object sets is not reported.

Target Audience

This paper is most useful to HCI and human-robot interaction researchers working on end-user robot programming and natural language interfaces; robotics engineers building LLM- or VLA-based task planners who need an interface layer for inspection and correction; and product or research teams deciding how much of an automatically generated plan to expose to non-expert operators. It is also relevant to researchers studying transparency, perceived control, and task load in human-in-the-loop AI systems more broadly.

Authors’ abstract

As collaborative robots have entered domains such as manufacturing, agriculture, and healthcare, programming or adapting robot behavior typically requires robotic expertise that most end users lack. Natural language lowers this barrier. Recent advancements in large language models (LLMs) have made it feasible to translate natural language into robot task plans. However, language-based task specification suffers from semantic ambiguity, and generative models lack transparency for how language instructions become robot actions, making it difficult for users to validate the plan before execution. To address these issues, we introduce SHRIMP, a system that allows users to automatically generate a hierarchical robot primitive plan using natural language and iteratively revise their plan through re-prompting and explicit correction. At each revision, SHRIMP allows users to validate their plan in simulation, and once satisfied, execute it on the physical robot. Through a user study involving participants planning tabletop kitchen tasks (n=35), we validate that SHRIMP improves perceived control and enhances robot transparency. System videos and source code are available at https://wisc-hci.github.io/SHRIMP.