GuideFetch: A Task Coordination Framework for Concurrent Navigation and Object Retrieval in Assistive Robot Dogs

AI summary

Overview

Research area: Assistive robotics / heterogeneous multi-robot task coordination, combining large language model (LLM) planning, legged locomotion, and legged manipulation.

Technical level: Advanced. The paper assumes familiarity with task graphs, dependency scheduling, reinforcement-learning locomotion policies, inverse kinematics, and LLM-based plan generation.

Scope (one sentence): The paper presents GuideFetch, a framework in which an LLM instantiates a schedule-conditioned four-action schema that is deterministically validated and then executed by two heterogeneous robot dogs (a guider and a fetcher) to complete a concurrent navigation-and-object-retrieval mission, with mission success verified from measured robot and object states rather than from plan validity.

What This Paper Is About

GuideFetch addresses a scenario where one robot guide dog leads a blind user to a seat while a second robot dog retrieves and delivers an object to that same seat. The core problem is that once guidance and retrieval are split across different robot embodiments, simply running two skill sequences is not enough: each action must go to a robot that can actually perform it, temporal dependencies must hold, conflicting commands to the same robot must be blocked, and mission completion must be verified from physical state rather than assumed from a well-formed plan. The goal is therefore to coordinate a heterogeneous guider-and-fetcher team from one natural-language request while keeping plan validity, execution progress, and verified mission completion as distinct states.

Key Contributions

1. A schedule-conditioned four-action schema with deterministic validation. An LLM instantiates action records over the vocabulary navigate, move, grasp, and carry; a validator resolves aliases and rejects any assignment that violates registered targets, robot capabilities, or the selected schedule before simulation begins.

2. A dependency compiler and shared-clock executor for two schedules. Validated plans are compiled into nominal dependency levels (three levels with two, one, and one actions for parallel; four single-action levels for sequential) and executed by branch-local state machines under one simulation clock, with failure propagation that prevents one branch from concealing an unmet objective on the other.

3. State-based completion verification decoupled from planning. Mission success is defined by a conjunction of measured booleans (visit, pickup, grasp, lift, retain, delivery), so a valid plan or a completed trajectory is not treated as evidence of mission success.

4. A controlled paired evaluation plus a mutation test. The paper reports 360 simulator executions over 90 matched scene–seed cases across scripted and online plan provenance, and a simulator-free mutation test that accepts two valid controls and rejects 32 rule-violating variants (10 action-inventory, 8 schedule-structure, 8 registered-target, 4 capability-assignment, 2 carried-object-identity edits).

Main Findings

• Normalized-plan agreement is perfect on the tested set. All 180 online responses from the locally served four-billion-parameter Qwen3.5 model (identifier frob/qwen3.5-instruct:4b) parse as one JSON object, pass all checks on the first request, and normalize to their scripted references. Because of this, the paper states the plan-provenance comparison tests normalized-plan agreement rather than a distinct execution factor.

• Sequential and parallel success rates are close. Sequential achieves 72/90 (80.0%, 95% CI [70.6, 87.0]) and parallel achieves 71/90 (78.9%, 95% CI [69.4, 86.0]) operational successes.

• Parallel execution is faster across the 56 common successes. Mean makespan drops from 164.78 s to 96.79 s, a 1.70× ratio and a 41.3% reduction. The paired reduction is 67.99 s, with bootstrap 95% interval [54.05, 82.04] s and sign-flip p < 0.001.

• The timing benefit is scene-dependent and system-level. In Home, makespan falls from 114.13 to 69.04 s (1.65×) and distance rises from 22.05 to 24.95 m; in Supermarket, from 84.02 to 67.94 s (1.24×) and distance rises from 14.86 to 16.76 m; in Hospital, from 280.74 to 144.20 s (1.95×) and distance falls from 68.25 to 40.60 m. Time falls by 39.5% in Home, 19.1% in Supermarket, and 48.6% in Hospital. The authors attribute the aggregate gain to role assignment, action overlap, and scene geometry rather than faster controllers.

• Schedule outcomes are discordant on 31 of 90 matched seeds. Of the 90 pairs, 56 succeed under both schedules, 16 only sequentially, 15 only in parallel, and 3 under neither.

• All operational failures occur during pre-transport manipulation. Counting 37 distinct scene–schedule–seed failures once: 19 exceed the 0.25 m grasp-reference consistency bound during retention, 15 retain final bilateral contact but exceed the 17° tilt limit, and three lose final bilateral contact (two of which also exceed tilt). These comprise 18 sequential and 19 parallel cases. No simulator episode fails validation or pickup arrival, and every parallel guider reaches its goal.

• Validator rejection is clean on the controlled mutation set. Two valid controls are accepted and all 32 single-edit mutations are rejected, which the paper describes as evidence on a controlled set rather than open-ended understanding or repair.

• Successful episodes show consistent object rise. The 143 successful scripted episodes comprise 41 Home, 51 Supermarket, and 51 Hospital runs. Mean peak and retained object rise are 69.89 and 62.05 mm in Home, 66.48 and 61.89 mm in Supermarket, and 69.43 and 63.05 mm in Hospital, all measured before post-lift object synchronization.

Methodology in Plain English

The researchers built a two-robot team in the InternUtopia simulator (built on GRUtopia): a Unitree Go2 guider and a Unitree B2 fetcher equipped with a Z1 arm. They fixed the scene geometry and routes in advance using the official Home, Supermarket, and Hospital GRScenes environments, registering robot starts, pickup location, shared goal, route waypoints, and object pose.

For each run, the execution schedule is chosen as an experimental condition before prompting; the LLM does not choose it. The prompt lists the available robots, the four permitted skills, the registered targets, the selected schedule, and the required JSON format. The model returns four action records, each with a step, robot, skill, target, and parameters. A parser extracts one JSON object and normalizes robot, skill, and target aliases against the scene registry. A validator then checks that all four expected actions are present, that each assignee actually provides the requested skill, and that the carry action retains the registered object. Sequential plans must contain four ordered fetcher actions; parallel plans must contain guider navigation and fetcher motion at step 1, followed by fetcher grasp and carry. Rejected responses are stored, and both scripted and online plans pass through exactly the same parser and validator.

Validated plans are compiled into nominal dependency levels and executed under a single 200 Hz simulation clock. Guider navigation uses a recurrent policy trained with PPO from 180 planar lidar measurements and an 11-element state vector, with a waypoint progress shield advancing the root along the registered route; the visit is satisfied when the guider reaches within 0.20 m of the final waypoint. The fetcher combines an ALORE-derived locomotion policy with joint-space arm staging and world-frame inverse kinematics. Grasp requires bilateral contact, lift requires a peak object rise of at least 30 mm, and retention is evaluated over a 1.0 s hold at 200 Hz requiring at least 20 mm retained rise, bilateral contact in at least 80% of hold frames and in the final frame, at least 5 mm separation between target and support collision bounds, at most 17° object tilt, and grasp-reference distance within 0.25 m. Object synchronization with the end effector begins only after these checks pass.

The experiment crosses two schedules (sequential, parallel) with two plan-provenance conditions (scripted, online) over 90 matched scene–seed cases, giving 360 executions. Each of the three scenes uses 30 seeds, and every seed produces all four conditions. Seeded perturbations vary robot starts by no more than 10 mm, object translation by 1.5 mm, object yaw by 0.5°, and object mass between 5.00 and 5.25 g, with friction varied in a narrow calibrated range; the perturbation record is generated once per scene and seed and copied into all four launch configurations. Four semantically equivalent instruction templates per schedule are rotated across seeds. The evaluated payload is a runtime-spawned 60 × 74 × 85 mm handled calibration object weighing 5.00–5.25 g, which the paper notes is not a filled beverage container. Success rates are reported with Wilson 95% confidence intervals, and paired mean differences use a deterministic 20,000-sample percentile bootstrap interval and a 50,000-draw sign-flip test. Failed episodes remain in the success-rate denominator, and makespan is compared only among the 56 runs completed by both schedules.

Why This Matters

Impact on research. The paper isolates a question that multi-robot LLM planning work often blurs: whether a valid-looking plan, a completed controller command, and a verified mission outcome should be treated as the same event. GuideFetch demonstrates an architecture where the LLM supplies structured action records while deterministic checks, capability constraints, and embodied state govern execution and completion. It also provides a paired, scene-and-seed-matched evaluation protocol for comparing schedules, and it shows that concurrency gains must be measured only over missions both schedules actually complete.

Real-world applications (as motivated or implied by the paper):

• Robot guide dogs assisting blind and low-vision users with point-to-point wayfinding.
• Assistive object retrieval and delivery, such as bringing a cup to a user who has reached a seat.
• Capability-aware dispatch in heterogeneous robot fleets, where tasks must be routed to robots that possess the required skills.
• Structurally checked interfaces between language models and physical robot execution in safety-adjacent settings.

Industry relevance. The paper's architecture — LLM instantiation, deterministic normalization and validation, dependency compilation, shared-clock execution, and state-verified completion — is directly relevant to teams building robot fleets, logistics and service robots, and any product where a language model proposes actions that a constrained physical system must accept, reject, or verify. The finding that all operational failures arise during manipulation identifies object handling, not planning or navigation, as the practical bottleneck.

Future Directions

• Language generalization. Evidence for the language interface is limited to four paraphrase templates per schedule and a fixed action inventory. The authors call for held-out instructions, larger action schemas, infeasible or conflicting requests, and state feedback that requires replanning, since the mutation test only covers deterministic rejection of controlled plan violations.
• Broader manipulation scope. The manipulation evidence covers one lightweight handled object under bounded pose, mass, and friction perturbations, and physical checks end before assisted transport begins. Future work should test varied objects, require unaided retention throughout loaded motion, and verify delivery until object release at the destination.
• Deployment realism. The study is limited to three registered scenes, controlled routes, disabled background physics, and kinematic root transport, and destination arrival is only a proxy for guidance because human following and escort safety are not modeled. The authors propose dynamic base control, changing obstacles, and unseen layouts.
• Human factors. User studies with blind and low-vision participants examining task time, uncertainty, workload, and perceived risk are named as future work but are not conducted in this paper.
• Causal attribution of discordant outcomes. The 31 discordant seed pairs share perturbations but follow different fetcher routes and pre-acquisition histories, and the study does not isolate the causal state difference.

Target Audience

Robotics and embodied-AI researchers working on multi-robot coordination, LLM-based task planning, or assistive robotics will gain the most from this paper, particularly those interested in separating plan validity from verified task completion. It is also relevant to engineers building heterogeneous robot fleets who need deterministic validation layers between language-model outputs and physical execution, and to accessibility researchers evaluating assistive robot dogs, though the paper does not report human-subject results. Readers should come with background in task graphs, legged locomotion policies, and manipulation control, since the paper assumes that context rather than introducing it.

Authors’ abstract

Consider a robot guide dog escorting a blind user to an available seat while a second assistive robot dog concurrently retrieves a cup of coffee and delivers it to the same seat. This setting motivates concurrent execution because navigation and object retrieval can overlap. A syntactically valid Large Language Model (LLM) plan may still violate embodiment constraints, and successful-looking controller motion does not by itself establish task completion. We introduce \textsc{GuideFetch}, a coordination framework for concurrent navigation and object retrieval by a heterogeneous guider and fetcher team. An LLM instantiates a schedule-conditioned four-action schema from a natural-language instruction. Before execution, robot, skill, and target aliases are normalized, and proposed actions are validated against registered targets, robot capabilities, and the selected schedule. Robot and object states then govern sequential and parallel execution. In a matched $2\times2$ study across 90 combinations of scene and seed (360 executions), all 180 online LLM responses validate without fallback or replay and match the corresponding scripted plans. For each planner source, sequential and parallel execution achieve $72/90$ and $71/90$ operational successes, respectively. Among the 56 cases completed by both schedules, parallel execution reduces mean makespan by 41.3\%. Within this controlled setting, role specialization and action overlap shorten completed missions, while state checks distinguish plan validity from verified mission completion. Source code will be available.