Representation Handoffs for OpenArm-Based Laboratory Mobile Manipulation

AI summary

Overview

Research area: Robotics and embodied AI, specifically language-guided mobile manipulation for laboratory automation.

Technical level: Advanced. The paper assumes familiarity with ROS2 middleware, MoveIt motion execution, coordinate-frame calibration, 6D object pose estimation, and LLM-based task planning.

Scope: A field report describing an OpenArm-based mobile manipulation prototype whose language-to-action pipeline is organized as a chain of explicit "representation handoffs," evaluated through dry-run traces and startup checks rather than real-robot task success rates.

What This Paper Is About

Open-source robot arms, ROS-based middleware, and foundation-model perception have made it cheap to assemble embodied AI prototypes, but converting a natural-language laboratory instruction into a safe, executable robot action is still an unsolved integration problem. The authors build a prototype on the OpenArm platform and reframe the problem: instead of asking only how to represent the scene, they ask which representation is actually actionable by a real robot. Their answer is a pipeline in which instructions, sensor data, object priors, and validated skills each pass through an explicit intermediate form that can be inspected and debugged.

Key Contributions

1. A system view of the representation handoffs among lab instructions, maps, object poses, object priors, registered skill calls, runtime bindings, and motion goals.
2. An OpenArm-based integration that connects ROS2/MoveIt execution, mobile base navigation, vertical motion, RGB-D grounding, a skill bank, and deterministic skill templates.
3. Dry-run evidence and field lessons showing where missing calibration, object models, perception outputs, and capability contracts block real-robot deployment.

Main Findings

• Six representations form the pipeline: natural language requests constrained into registered SkillCall records; sensor observations grounded into maps and object poses; object priors supplying role and skill constraints; and runtime bindings compiling validated skills into executable MotionGoal records. These are the "handoffs" the paper is named for.

• A dry run can execute every handoff end to end. For the instruction "put source object on destination object," the trace covers scene setup (navigation target in the map frame before grounding), world state (WorldObject list), planning (SkillPlan plus capability gaps), runtime precheck (required objects, held-object state, safety gate), runtime execution (operation parameters and MotionGoal records such as pick-above, pick-down, gripper close, lift, place, release, retreat), and feedback (RuntimeFeedback with per-goal robot state, world snapshot, resolved args, and executor result).

• 6D pose is necessary but insufficient. FoundationPose outputs are useful, but the downstream planner needs pose plus frame, confidence, object identity, geometry, skills, roles, and target-specific operation parameters. The planner should see an actionable object, not a raw perception result.

• Profiles are representations, not merely configurations. The usage, calibration, and development profiles encode three distinct forms of state: object priors, geometric deployment facts, and capability contracts. Treating them as first-class representations made startup checks and dry-run traces more informative than ad hoc configuration files.

• Constrained intermediate forms make LLM planning debuggable. The LLM only proposes registered skill calls under a skill bank contract; it cannot invent arbitrary ROS commands, bypass role requirements, use undeclared arguments, or execute unregistered recovery. The trade-off is reduced flexibility, but unsupported capabilities surface as explicit unmapped_requests rather than hidden model assumptions.

• Deployment blockers are representation blockers. Remaining real-robot work is not just hardware bringup: it requires measured camera, lidar, and slide transforms; table frame localization; object meshes; real RGB-D masks or detections; and calibrated workspace bounds. The report's value is that these blockers appear at representation interfaces where they can be tested.

• No quantitative task results are reported. The paper states that at this stage the system validates the representation pipeline and software integration, and that it currently does not provide real-world success rates for visual grasping, pouring, or insertion. Errors can be localized to one of nine stages: scene origin preparation, world grounding, planner validation, skill coverage, operation-parameter resolution, runtime precheck, safety validation, executor routing, or hardware execution.

Methodology in Plain English

The authors built a prototype by integrating dual OpenArm manipulators, grippers, a vertical slide, a mobile base, RGB-D sensing, and lidar mapping, with ROS2/MoveIt handling execution. The software stack also uses the FoundationPose model and AprilTag detections, which serve as table anchors for estimating or verifying the tabletop frame.

Architecturally, the top-level agent is a deterministic orchestrator rather than an LLM. An LLM appears only inside a unified planner that must emit registered SkillCall records against a development profile skill bank. If the planner fails validation or cannot cover a requested capability, the system reports unmapped_requests. Output is validated against the skill bank, object role requirements, argument types, and step-id format, with one validation-repair attempt allowed before the request is declared unmapped.

At runtime, the system pre-checks the plan, refreshes robot and world states before each skill call, resolves object- and target-specific operation parameters, validates safety limits, and executes a sequence of MotionGoal records (move_to, open_gripper, set_do, slide_move, nav_to_pose). Every motion goal passes a safety gate before routing to an executor. The policy is deliberately stop-on-failure: if an object is missing, below confidence, violates a held-object precondition, or fails during execution, the runtime records RuntimeFeedback and stops, so a language planner cannot paper over missing perception or unsupported skills.

Perception is also made deterministic downstream. A WorldObject carries a 6D pose, frame id, confidence, roles, skills, and metadata. The FoundationPose service boundary is a concrete HTTP/RGB-D protocol with health, model, and object endpoints, plus a mock mode for software integration before the GPU perception stack and masks are ready. An operation-parameter planner then generates approach, grasp, lift, place, insert, pour, dispense, tap, swipe, and twist parameters, stored as perception_args, skill_args, and target_skill_args, so a trace can explain why a particular pick pose, insertion depth, or pour clearance was chosen.

The evidence base is dry-run traces and startup checks, not physical trials.

Why This Matters

Research impact: The paper argues that the interesting artifact is not a new perception model or controller but an auditable path from laboratory instructions to motion goals and runtime feedback. It positions itself between classic robot middleware and recent LLM/VLA robotics work, showing how constrained intermediate forms make language-based planning inspectable rather than opaque.

Real-world applications:

• Moving containers and picking/placing items in laboratory settings.
• Handling tubes or racks, where an object can carry roles such as movable item, liquid container, reaction vessel, insertion target, sample container, fixture, surface, or place target.
• Preparing actions such as pouring or sample transfer.
• Future task-level evaluation targets named in the paper: pick, place, insert, pour, clean, and screen-interaction tasks.

Industry relevance: Because the hardware is low-cost and open-source (OpenArm) and the stack is built on widely used ROS2/MoveIt middleware plus FoundationPose and AprilTag, the integration lessons are transferable to any team assembling laboratory automation or embodied AI prototypes. The paper's contribution to practice is a debugging interface: when something fails, the failure lands on a named representation boundary instead of somewhere in an end-to-end black box.

The work is supported by the Shenzhen Fundamental Research Program (JCYJ20200109141235597), the National Natural Science Foundation of China (72401122, 61761136008), the Guangdong Basic and Applied Basic Research Foundation (2024A1515012241, 2021A1515110024), the Shenzhen Peacock Plan (KQTD2016112514355531), the Program for Guangdong Introducing Innovative and Entrepreneurial Teams (2017ZT07X386), and the Australian Research Council Discovery Grants DP210101093 and DP220100803.

Future Directions

• Replace placeholder calibration and model assets with measured field data, including first-time calibration records, camera/lidar/slide transforms, and table height measurements.
• Align the FoundationPose service with the current usage-profile object registry so perception outputs match declared object priors.
• Enable strict real-scene visual grounding by populating object mesh and point-cloud assets and supplying real RGB-D input with masks or a detector.
• Evaluate task-level failures across pick, place, insert, pour, clean, and screen-interaction tasks. An open question the paper leaves implicit is what real-world success rates these handoffs would produce, since none are reported here.

Target Audience

Robotics engineers and embodied AI researchers who are integrating open-source arms, mobile bases, and ROS2/MoveIt with foundation-model perception, and who need a concrete way to structure, validate, and debug the interfaces between language, perception, planning, and robot safety. It is also useful for laboratory automation developers evaluating how far an LLM-driven pipeline can be trusted before hardware execution, and for students studying system-integration approaches as distinct from novel model architectures.

Authors’ abstract

Open-source robotics and foundation models have lowered the barrier to embodied AI, yet language-guided laboratory automation still requires reliable alignment from instructions and observations to safe actions. This field report presents an OpenArm-based mobile manipulation prototype for laboratory-style tasks, built by integrating dual OpenArm manipulators with a mobile base, vertical slide, RGB-D sensing, lidar-based mapping, ROS2/MoveIt execution, and profile-defined skill interfaces. The system is organized around representation handoffs: natural language requests are constrained into registered skill calls, sensor observations are grounded into maps and object poses, object priors provide role and skill constraints, and runtime bindings compile validated skills into executable motion goals. We use dry-run traces and startup checks to evaluate this integration path, showing how the prototype exposes missing calibration, incomplete object assets, and unfinished real-scene visual grounding as explicit deployment blockers. These intermediate representations serve as practical debugging interfaces for integrating language, perception, planning, and robot safety in embodied systems.