RACAS: Controlling Diverse Robots With a Single Agentic System

AI summary

Overview

• Research area: Robotics / robot control, LLM- and VLM-based agentic AI, cross-embodiment generalization.
• Technical level: Intermediate. The architecture is conceptually simple (prompt-configured LLM modules in a closed loop), but the paper assumes familiarity with robot APIs, perception pipelines, and agentic LLM systems.
• Scope: The paper introduces RACAS, a single cooperative LLM/VLM agentic system that controls three radically different real robot platforms without retraining, code changes, or reward-function modification.

What This Paper Is About

Robots expose low-level APIs for commanding actuators and reading sensors, but turning those APIs into high-level autonomous behaviour normally requires a pipeline of perception, planning, and control modules built by experts from different fields. Existing solutions either require retraining for every new robot body or have only been validated on structurally similar platforms. RACAS aims to bridge this gap by decomposing closed-loop control into three LLM/VLM modules that talk to each other only in natural language, so that adapting to a new robot requires nothing more than new prompt files.

Key Contributions

1. RACAS architecture: A cooperative multi-module agentic architecture for closed-loop robot control in which a Controller, a set of Monitors, and a Memory Curator communicate exclusively in natural language.
2. Declarative adaptation: All embodiment-specific and task-specific knowledge is confined to three declarative prompt configuration files (robot description, action definitions, task description), requiring no modification to source code, model weights, or reward functions.
3. Zero-training cross-embodiment generalization: The authors report the first zero-training generalization of a single control framework across three radically heterogeneous robot platforms, including a recently published robotic limb that an LLM would have no prior knowledge of.
4. Open release: Source code for all experiments, including prompts, is available at https://github.com/janprz11/robot-agnostic-control.

Main Findings

• All tasks solved on all platforms: The unmodified RACAS completed every assigned task across the Alhakami et al. Limb, the Clearpath Dingo (simulation and real world), and the BlueRobotics BlueROV2 (small tank and large pool).
• Dingo simulation: RACAS averaged 16.40 ± 0.96 steps versus a random baseline of 25.00 ± 0.00 steps, against an oracle minimum of 10 steps.
• Dingo real world: RACAS averaged 25.80 ± 4.94 steps against an oracle minimum of 7 steps; no random baseline was run because it was unsafe.
• Alhakami et al. Limb: RACAS averaged 9.56 ± 1.50 steps versus 22.07 ± 0.98 for the random baseline, against an oracle minimum of 3 steps.
• BlueROV2 small tank: RACAS averaged 14.00 ± 1.75 steps against an oracle minimum of 5 steps; no random baseline was run.
• BlueROV2 large pool: RACAS averaged 19.20 ± 6.51 steps against an approximate oracle minimum of approximately 5 steps; no random baseline was run, and wind added stochasticity.
• Statistically significant gains where baselines were safe: Where a random baseline was possible, RACAS performed significantly better with family-wise confidence of p < 0.01 using Student's t-test and Holm-Bonferroni corrections. Random control policies were artificially limited to 25 steps.
• Time is dominated by search, not approach: Most of the completion time came from the system trying to locate the object; once spotted, RACAS directed the robot in an almost straight line toward the objective. The authors conclude performance is limited more by sensor fidelity and available tools than by the underlying models or architecture.
• Memory Curator works as a memory mechanism: In a separate proof-of-concept on a modified OpenAI Gym Blackjack environment where the target was changed to 42 (with the agent only told the target lies in the range [12, 100]), the Memory Curator allowed the system to infer the hidden target across episodes and maintain high performance, compared with a non-structured memory baseline, a complete memory-removal baseline, and a random-choice policy.
• Low cost and latency profile: The control loop executes at roughly one action per 5–10 seconds, dominated by API latency, and the total cost of API calls was less than $100.

Methodology in Plain English

The team designed a control loop made of three parts, each driven by a large AI model, that pass natural language messages to each other.

1. The Controller is re-initialized at every step with a system prompt assembled from six pieces: a natural language robot description, the action interface, the accumulated environment memory, proprioceptive state (joint displacements from an internal position tracker), the action history, and the task specification. It first asks a targeted visual question, then, after receiving the answers, produces a reasoning trace and selects one action.
2. The Monitors receive each camera image plus the Controller's query and use a vision-language model to return a natural language scene description. This differs from conventional perception pipelines: the query changes with the execution stage, so perception is task-adaptive, the output is in the same representational space as the Controller's input, and no domain-specific training is needed. For low-resolution cameras (for example, 100 × 100 pixels on the limb), a two-stage Swin2SR super-resolution pipeline upscales images to 768 × 768 pixels before VLM inference.
3. The Memory Curator sits outside the real-time loop and rewrites, rather than appends to, a bounded structured memory after every step, compressing redundant information, resolving contradictions, and discarding irrelevant details. It organizes knowledge into four categories: physical environment (scene description, spatial layout, dynamic object inventory), robot state (position, orientation, joint configuration), curated history of significant commands and outcomes, and task state. It also performs cross-modal position inference by intersecting which camera saw an object with which action brought it into view, compensating for the VLM's lack of metric depth estimation.

Adapting to a new robot means authoring three text files: a robot description, a structured JSON tool definition of the admissible actions, and the task description. All LLM and VLM inference ran through the official OpenAI API using GPT-4.1-mini or GPT-4.1; the smaller model was used for all experiments except Dingo (real-world) and BlueROV2 (large pool).

The evaluation covered three platforms. The Alhakami et al. Limb is a 4-DOF arm with four active cameras at 100 × 100 px and nine actions (eight directional primitives plus a reset command); the task was locating a fire extinguisher in a cluttered laboratory workspace over 30 runs. The Clearpath Dingo is a 3-DOF wheeled robot with four actions (forward, backward, rotate left, rotate right), three cameras at 640 × 480 px in NVIDIA Isaac Sim and one camera at 1280 × 800 px in the real world; the tasks were navigating to a fire extinguisher in a simulated 20 × 30 m warehouse and a real 9.5 × 5.5 m curtained area, 10 runs each, with success defined as coming within 1 m of the target. The BlueROV2 is a 6-DOF ROV with one 1920 × 1080 px camera and six actions (surge forward/backward, yaw left/right, heave up/down), used in a 2.00 × 1.10 × 1.50 m tank filled to 80 % of its volume with the image cropped to 512 × 512 px, and in a 15 × 15 m pool with depth up to 3.8 m; these settings comprised 10 runs and 5 runs respectively.

Why This Matters

Impact on research: The paper argues that agentic AI can eliminate the usual barriers to robot deployment, which stem from collecting demonstration data, engineering custom state representations, and training platform-specific policies. It positions natural language plus a dynamic structured memory as a substitute for learned parameters and engineered modules. The authors report that prior systems demonstrated zero-training generalization only across structurally similar platforms (for example, different manipulator arms), and note that a custom-engineered system would still be needed for production-grade deployment.

Real-world applications:

• Rapid prototyping of autonomy for field or inspection robots whose APIs already exist but whose high-level stack does not.
• Underwater inspection and navigation using ROVs such as the BlueROV2 in tanks, pools, or open water.
• Laboratory or industrial manipulators such as the Alhakami et al. Limb, where a natural language task specification replaces bespoke perception and planning code.
• Low-cost robotics education and research, given the reported API spend of less than $100 and the public release of code and prompts.

Industry relevance: Robot hardware vendors expose APIs but not autonomy stacks. RACAS suggests a vendor or integrator could ship only a robot description, an action schema, and a task prompt to make a platform usable, reducing the need for separate perception, planning, and control specialists. Operating limits remain: roughly one action per 5–10 seconds is far slower than conventional control, and the authors state the stochastic nature of the approach means production deployment would still need custom engineering.

Future Directions

• Extending RACAS to contact-rich tasks such as manipulation. The authors note that the high per-step inference cost of the current architecture makes long-horizon manipulation experiments prohibitively slow, and propose integrating existing manipulation primitives as callable tools to reduce the number of LLM-mediated decisions per task.
• Integrating LIDAR or a more sophisticated architecture into the Monitor subsystems. The lack of depth information was a consistent issue: depth had to be inferred from images, those inferences were sub-optimal, and they left the robots under- or over-confident about impact chances.
• Testing the architecture on a wider range of tasks and platforms, since the current evaluation is restricted to target localization and navigation, which represents only one stage of a typical robotic mission.
• Validating whether improvements in foundation model reasoning and perception make the approach viable for more tasks, as the authors expect.

Target Audience

Robotics researchers and practitioners who want to apply robots to new tasks without building embodiment-specific pipelines; engineers integrating LLM and VLM agents into physical systems; and students or researchers interested in agentic architectures, structured agent memory, and cross-embodiment generalization. Readers focused on high-frequency control, contact-rich manipulation, or production-grade reliability will find the paper most useful as a boundary statement about what prompt-configured agentic control can and cannot yet do.

Authors’ abstract

Many robotic platforms expose an API through which external software can command their actuators and read their sensors. However, transitioning from these low-level interfaces to high-level autonomous behaviour requires a complicated pipeline, whose components demand distinct areas of expertise. Existing approaches to bridging this gap either require retraining for every new embodiment or have only been validated across structurally similar platforms. We introduce RACAS (Robot-Agnostic Control via Agentic Systems), a cooperative agentic architecture in which three LLM/VLM-based modules (Monitors, a Controller, and a Memory Curator) communicate exclusively through natural language to provide closed-loop robot control. RACAS requires only a natural language description of the robot, a definition of available actions, and a task specification; no source code, model weights, or reward functions need to be modified to move between platforms. We evaluate RACAS on several tasks using a wheeled ground robot, a recently published novel multi-jointed robotic limb, and an underwater vehicle. RACAS consistently solved all assigned tasks across these radically different platforms, demonstrating the potential of agentic AI to substantially reduce the barrier to prototyping robotic solutions.