Towards Agentic Robots: Model, Data, Environment, and Harness

AI summary

Overview

Research area: Robotics and embodied AI — specifically the emerging paradigm of "agentic robots," positioned at the intersection of foundation models, vision-language-action (VLA) models, world models, and robot system design.

Technical level: Advanced. This is a large-scale survey/overview paper organized around a taxonomy of seventeen categories, written for readers already familiar with robot learning, VLA models, and foundation-model agents.

Scope (one sentence): The paper provides a unified taxonomy of agentic robots across four functional domains — Model, Data, Environment, and Harness — tracing how agents shape robot learning, data curation, world and task construction, and runtime orchestration, and connects this to a research agenda for practical deployment.

Note on completeness: The supplied content is truncated, ending mid-sentence in Section 4.2 (Perception). Sections 4.3 onward, the Data, Environment, and Harness detail sections, and the open-challenges discussion are described only at the level of the abstract, introduction, taxonomy table, and section previews. Details not present in the supplied text are flagged below as not reported.

What This Paper Is About

Foundation models are shifting robotic intelligence from low-level action prediction toward goal-directed orchestration of data collection, model training, task planning, and real-world execution. The authors call this paradigm agentic robots: systems where agents reason over human instructions, interact with robotic environments, autonomously rectify execution errors, and dynamically adjust operational strategies. The core problem the paper addresses is fragmentation — research on model learning, data acquisition, environment construction, and runtime system design is studied separately, and there is no consistent view of how these agentic functions combine into a complete robot system. The goal is a unified overview and taxonomy that clarifies where agents make decisions and how those decisions contribute to the overall system.

Key Contributions

1. A unified illustration for agentic robotics. The paper traces the evolution from robot policies to agentic systems and clarifies the role of agents in robot task execution and capability development. It introduces a taxonomy spanning Model, Data, Environment, and Harness, bringing these research directions into a common framework.

2. A systematic analysis of agentic methods. It synthesizes the mechanisms through which agents acquire, coordinate, and improve robotic capabilities, comparing representative approaches across the taxonomy and explaining how agent decisions depend on available tools, robot executors, and interaction feedback, along with the strengths and limitations of these approaches.

3. A research agenda for practical agentic robots. The paper outlines research directions for establishing agents' contributions and translating current advances into reliable, efficient robot operation, connecting fair evaluation with the challenges of physical deployment and generalization across robotic platforms.

4. A seventeen-category operational taxonomy. Existing research is organized into seventeen non-exclusive categories across the four domains, defined in Table 1, with an accompanying literature landscape (Figure 3) and a public repository at https://github.com/chonghanqin/Awesome-Agentic-Robots.

Main Findings

• Four paradigms of robot intelligence. The paper characterizes an evolution across task-specific policies for direct control, vision-language-action models grounded in language model knowledge, imagination-centric world models or world-action models, and agentic decision-making that reasons beyond learned policies (Figure 1).

• Four connected stages of robot learning. Conventional robot policies (learning executable behavior), VLA models (connecting knowledge to action), world models and world-action models (anticipating physical outcomes), and agentic robots (organizing execution and improvement). The paper states these stages describe overlapping research traditions and complementary capabilities that can coexist in one system.

• Foundation-model agents can already act as robot manipulation agents without task-specific fine-tuning. GPT-6 Astra uses visual observations to generate end-effector commands and adjust actions from execution feedback. On RoboDojo it achieves a 22.48% average success rate across 42 simulation tasks, outperforming DM0.5 (19.34%), GalaxeaVLA G0.5 (14.88%), Xiaomi-Robotics-1 (13.93%), OpenWAM-α (11.92%) and π0.5 (6.91%).

• Inference latency and resource costs matter. The paper argues that because inference latency and resource costs are significant, how foundation models are integrated into robotic systems is as important as model scaling — agentic robots depend on both the model and the tools, observations, and feedback available in the environment.

• The taxonomy separates functional contributions, not systems. A paper can appear in multiple categories. Zero2Skill links agent-directed data collection to subsequent policy training; HarnessVLA integrates memory, orchestration, monitoring, and recovery around fixed executors; ENPIRE shows a research agent coordinating experiments to improve robot policies.

• Seventeen categories across four domains. Model covers Training, Perception, Planning, Robot Control, and Adaptation; Data covers Collection, Annotation, and Refinement; Environment covers Reconstruction, Task Generation, and Benchmarking; Harness covers Orchestration, Memory, Monitoring and Recovery, Skills, Infrastructure, and Layered Systems.

• Training interventions differ mainly in the scope of the editable artifact. Reward-design methods keep the optimizer fixed and revise only the signal it consumes; program- and experiment-level methods extend revision to policy code, training pipelines, and the experimental workflow itself.

• Reward design trades generality for cost. Eureka and RDA evaluate candidate rewards through full policy training, making each evaluation computationally expensive. Later methods reduce this cost by validating rewards with cheaper evidence or narrowing the search: EvoNav screens reward programs with rules and short proxy training; CRWM conditions a one-pass reward-code generator on a causal prior from offline reward interventions; Demo2Reward uses a VLM meta-critic to tune a VLM reward prompt on labelled demonstration prefixes; ReCoVLA maps failure observations to descriptors that a deterministic compiler binds to predefined state-based reward components while the base VLA stays frozen.

• Training infrastructure exposes different bottlenecks. ENPIRE first constructs reset, verification, and safety interfaces with human feedback, then exposes them to coding agents that inspect real rollouts and modify policy or training code; it depends on reliable physical resets. HARBOR organizes simulation reinforcement learning into typed artifacts and gate-checked stages and depends on agreement among generated task, reward, and algorithm interfaces. Nautilus standardizes policy, robot, and benchmark contracts and isolates their execution environments.

• A design principle for agentic training workflows. Expose objectives, recipes, and experiments for revision while keeping the measurements used to judge them stable — motivated by the risk that reward, prompt, tool, and data edits can change both the intervention and its judge. RoboLineage stores rollouts, reviews, dataset decisions, training runs, policy metadata, and deployment recommendations as typed artifacts; FluxVLA Engine supplies configuration-resolved contracts connecting data, training, evaluation, and deployment.

• Perception is an active, evidence-seeking process. In the portion supplied, the paper describes systems that gather information through action: MG-VQA (formerly Probe) lets a VLM invoke perception, grasp-and-remove, and pushing tools to expose occluded information; SCOPE treats camera orientation and zoom as explicit actions; HINT updates intent and routes camera views at manipulation-pattern transitions and re-localizes the same committed target when tracking confidence is low.

• Distinctions from related concepts. Embodied AI broadly investigates learning and acting through environmental interaction, while the agentic perspective focuses on how agents select capabilities, coordinate execution, and revise decisions from feedback. Autonomous robotics already integrates planning, acting, monitoring, observation, and learning, so these are shared foundations rather than capabilities unique to agentic robotics. Robot foundation models concern capabilities learned by a model, whereas agentic robotics examines how goals, available actions, and execution feedback govern subsequent decisions. The paper states that "agentic" identifies how decisions are organized — it does not imply complete independence from human guidance or a higher level of autonomy.

• Where the paper positions itself relative to prior surveys. PAPAV compares computer use and robot use through Perceive, Anticipate, Plan, Act, and Verify; citation [288] examines model–robot integration through protocol interfaces, orchestration, and embedded policies; [130] surveys robot-learning approaches spanning action-predicting weights and executable skills; [251] studies executable software organizing agent execution; [332] examines where verification signals come from and whether they are available and trustworthy. The paper's stated focus is how these components work together during robot use and capability building.

• Not reported in the supplied content. The detailed findings of the Data, Environment, and Harness sections (Sections 5–7), the full Perception section beyond Section 4.2, the Planning, Robot Control, and Adaptation subsections, and the explicit open-challenges list are not included in the provided text.

Methodology in Plain English

This is a survey and taxonomy paper rather than an experimental study. The authors take three steps.

First, they trace the historical development of robot learning through four connected stages, from conventional image-based and 3D-aware policies through VLA models to world models, world-action models, and finally agentic systems, describing representative methods at each stage.

Second, they define agentic robots and propose a functional taxonomy. Rather than assigning each system a single label, they organize work by the role an agent plays in the robotic system, splitting it into four domains — Model, Data, Environment, Harness — and defining seventeen non-exclusive categories within them. Each domain is framed by a guiding question: what should the agent learn, understand, decide, and control; how should it collect, annotate, and refine robot data; what worlds, tasks, and tests should it build; and how should robot capabilities be orchestrated, recovered, and evolved.

Third, within each domain they group methods by function and compare representative approaches, focusing on how agent decisions depend on available tools, robot executors, and interaction feedback. The paper uses illustrations to make the distinctions concrete — for instance, a drawer-retrieval task where failure can require better evidence about the handle, a revised order of actions, a feasible grasp command, or a policy that improves after repeated failures, each corresponding to a different model-side intervention. It also includes comparison tables defining categories and summarizing training methods by target, approach, and feedback source.

Why This Matters

Impact on research. The paper argues that agentic robotics research is fragmented across model learning, data acquisition, environment construction, and runtime system design, and that related terms such as embodied agents, autonomous robots, and self-improving robotic systems blur the field's scope. A shared taxonomy lets researchers locate where agents make decisions and how those decisions contribute, and the accompanying research agenda targets fair evaluation and generalization across robotic platforms.

Real-world applications discussed in the paper:

• Long-horizon manipulation. The drawer-retrieval example illustrates tasks where robots must track task progress, adjust plans amid environmental changes, and recover from failures — capabilities the paper says VLA policies alone cannot consistently deliver.
• Robot data collection. AutoRT is cited as using foundation models to propose and filter tasks for robot data collection; Zero2Skill links agent-directed data collection and preparation to subsequent policy training.
• Navigation. AgenticNav provides heading-indexed views, selective recall, a compact trajectory map, and geometric tools; EvoNav applies evolutionary search and iterative refinement to navigation rewards.
• Simulation evaluation. RoboDojo is used as an evaluation setting with 42 simulation tasks, and the Environment domain covers reconstruction of executable scenes, task generation, and benchmarking procedures.

Industry relevance. The paper emphasizes that inference latency and resource costs highlight the importance of how foundation models are integrated into robotic systems, that agentic robots depend on both the model and the surrounding tools and feedback, and that deployment requires reliable, efficient operation. Its agenda explicitly connects fair evaluation with physical deployment and generalization across robotic platforms, which matters to anyone building or deploying robot systems rather than only training policies.

Future Directions

The supplied content states that the paper discusses open challenges and outlines future research directions, but the details of that discussion are not included in the provided text. The directions it does raise explicitly are:

• Establishing agents' contributions. The research agenda calls for work that establishes what agents actually contribute to robotic capability and operation, rather than assuming benefit.
• Translating advances into reliable, efficient robot operation. Connecting current research advances to trustworthy and efficient deployment.
• Fair evaluation and physical deployment. The agenda connects fair evaluation with the challenges of physical deployment.
• Generalization across robotic platforms. Building capability that transfers across different robot embodiments and platforms.

The paper also surfaces concrete bottlenecks that point to further work: the cost of full policy training as a reward-evaluation signal versus cheaper but more constrained reliability sources (proxy rollouts, offline causal structure, labelled demonstrations, predefined reward components); ENPIRE's dependence on reliable physical resets; HARBOR's dependence on agreement among generated task, reward, and algorithm interfaces; and the risk that editing rewards, prompts, tools, or data changes both an intervention and its judge.

Target Audience

Robotics and embodied-AI researchers, VLA and world-model developers, and graduate students who need a structured map of agentic robot research. It is most useful to readers who already know the basics of imitation learning, reinforcement learning, and vision-language-action models and want a framework for connecting model learning, data curation, environment construction, and runtime orchestration. Practitioners designing robot system architectures — especially those deciding how foundation models are integrated with tools, observations, and execution feedback — will also find the taxonomy and research agenda relevant. Readers looking for new experimental results should note that this is an overview paper, and that the supplied content is truncated before the Data, Environment, and Harness sections and the open-challenges discussion.

Authors’ abstract

Foundation models are driving a paradigm shift in robotic intelligence, moving beyond low-level action prediction to goal-directed orchestration of data collection, model training, task planning, and real-world execution. We characterize this emerging paradigm as an agentic robots, in which agents reason over human instructions, interact with robotic environments, rectify execution errors autonomously, and dynamically adjust operational strategies. Although this field develops fast, current studies are not well connected. Researchers have no consistent view of how to combine different agentic functions into a complete robot system. This paper provides a unified overview of existing research on agentic robots. We first trace the development from robot policies to agentic systems, covering vision-language-action models, world models, and world-action models. We then clarify the definition and scope of agentic robots and organize existing research into seventeen categories across four functional domains: Model, Data, Environment, and Harness. Across these domains, we examine how agents shape model learning and robot decisions, collect and curate robot data, construct worlds, tasks, and tests for learning and evaluation, and organize capabilities for robot use. Finally, we discuss open challenges and outline directions for future research. Looking forward, we hope this work contributes to a clearer understanding of agentic robots and supports new approaches to making them more capable, reliable, and adaptive.