Cook and Clean Together: Teaching Embodied Agents for Parallel Task Execution

AI summary

Overview

Research area: Embodied AI and 3D multi-modal large language models (MLLMs), specifically task planning, operations research (OR) based scheduling, and 3D visual grounding.

Technical level: Advanced. The paper assumes familiarity with embodied agents, 3D point-cloud encoders, MLLM fine-tuning, knapsack-style combinatorial optimization, and 3D grounding metrics.

Scope: The paper defines a new task (ORS3D), releases a 60K-scale dataset (ORS3D-60K), and proposes a model (GRANT) that combines an MLLM, an external optimization solver, and a 3D grounding head to produce time-efficient, spatially grounded task schedules.

Authors and affiliations: Dingkang Liang, Cheng Zhang, Xiaopeng Xu, Jianzhong Ju, Zhenbo Luo, Xiang Bai — Huazhong University of Science and Technology and MiLM Plus, Xiaomi Inc. Posted as arXiv:2511.19430v1 [cs.CV], 24 Nov 2025. Code is listed at https://github.com/H-EmbodVis/GRANT.

What This Paper Is About

Embodied agents that receive a multi-step household instruction (for example, a composite task mixing cooking and cleaning) should not simply produce a plausible list of actions. They should exploit parallelizable subtasks — operations such as running a microwave or filling a sink that only need to be started and re-checked — so that other work happens during the waiting time, and they should also say where in the 3D scene each action must happen. The paper argues that existing task-planning datasets ignore both operations research knowledge and explicit 3D grounding, and it builds a task, a dataset, and a model to fix that.

Key Contributions

1. A new task, ORS3D (Operations Research knowledge-based 3D Grounded Task Scheduling), which requires an agent to jointly understand lengthy natural-language composite tasks, optimize total completion time using OR knowledge, and localize the target object of every step in a 3D scene.
2. A large-scale dataset, ORS3D-60K, containing 60,825 composite tasks across 4,376 real-world indoor scenes, with an average data-item length of 311 words. The paper states it is the first dataset to incorporate operations research knowledge for task scheduling in 3D scenarios, and the largest among the compared related datasets.
3. GRANT, a grounded task scheduling agent — an embodied MLLM with a scheduling token mechanism (STM) that links the language model to an external optimization solver, plus a 3D grounding head that produces point masks for target objects.
4. Extensive benchmarking on ORS3D-60K covering language quality, scheduling efficiency, and 3D grounding, including comparisons against commercial LLM/MLLMs, object-level methods, and scene-level methods, along with ablations and runtime analysis.

Main Findings

• Scheduling gains over the scene-level baseline: Against Grounded 3D LLM (Vicuna-1B), GRANT improves task completion time efficiency by 30.53%, 3D grounding accuracy by 1.38%, and overall performance by 10.46%. GRANT's overall score is 53.49 versus 43.03 for Grounded 3D LLM.
• Full test-set results for GRANT: METEOR 42.82, ROUGE 62.78, Time Efficiency (TE) 72.99, grounding accuracy 35.38. The baseline Grounded 3D LLM scores METEOR 41.96, ROUGE 53.71, TE 42.46, grounding accuracy 34.00.
• Commercial models cannot ground in 3D: Gemini-2.0-flash reports METEOR 41.67, ROUGE 58.48, TE 24.75, overall 31.22, with 3D grounding unsupported; GPT-4o reports METEOR 49.16, ROUGE 62.19, TE 45.27, overall 39.15, with 3D grounding unsupported. DeepSeek-R1 (DeepSeek-V3) reaches TE 72.63 with METEOR 32.40, ROUGE 41.50, and overall 36.63 — strong scheduling attributed to reinforcement learning on mathematical problems, but it cannot process point clouds.
• Object-level methods ground well but do not understand language: 3D-VisTA (with Mask3D) scores 54.90 grounding accuracy and 13.73 overall; PQ3D (with Mask3D) scores 56.12 grounding accuracy and 14.03 overall, with language output unsupported. LEO (adapted with Vicuna-1B) scores METEOR 46.61, ROUGE 60.32, TE 45.63, overall 38.14, with grounding unsupported.
• Dense grounding metrics (Table 3a): GRANT reaches AP@0.25 of 35.38, AP@0.50 of 24.79, and mIoU 26.71, ahead of Grounded 3D LLM (34.00, 23.93, 25.56) but below Mask3D-based 3D-VisTA (54.90, 41.88, 43.29) and PQ3D (56.12, 44.01, 46.37), which use an external detector.
• Subtask type recognition drives scheduling: GRANT attains 84.65% accuracy on subtask type identification, with parallelizable-subtask precision 73.82, recall 54.70, F1 62.84, and non-parallelizable-subtask precision 95.94, recall 90.67, F1 93.23 — and TE 72.99. LEO reaches 79.73 accuracy and TE 45.63; Grounded 3D LLM reaches 77.14 accuracy and TE 42.46.
• The scheduling token mechanism is the decisive component: Removing scheduling content drops scores to METEOR 35.60, ROUGE 48.89, TE 21.03, grounding 15.95. Adding scheduling content as text raises them to 41.29, 55.28, 47.04, 34.74. Adding STM yields 42.82, 62.78, 72.99, 35.38 — a further 25.95% gain in time efficiency. With ground-truth scheduling content the model reaches 53.34, 75.06, 90.29, 38.52.
• Difficulty scales with subtask count: Overall score falls monotonically from four to seven subtasks for most methods: GRANT 60.23 → 52.98 → 52.03 → 48.70 (overall 53.49); Grounded 3D LLM 54.35 → 45.13 → 36.59 → 36.04 (43.03); LEO 42.14 → 40.12 → 36.42 → 33.91 (38.14); PQ3D 14.82 → 14.15 → 13.40 → 13.73 (14.03).
• Larger LLM helps modestly: Moving from a 1B to a 7B LLM raises METEOR from 42.82 to 45.19, ROUGE from 62.78 to 63.55, TE from 72.99 to 73.21, and grounding accuracy from 35.38 to 36.25. The 1B setting is the default for cost reasons.
• Solver overhead is negligible: Runtime is 1.14 ms for 4 subtasks, 1.28 ms for 5, 1.31 ms for 6, 1.42 ms for 7, 1.49 ms for 10, 2.01 ms for 20, and 3.94 ms for 50 — under 4 ms even at 50 subtasks.
• Qualitative example: In the illustrated case, the model identifies a microwave operation as parallelizable and schedules other subtasks during its 30-minute waiting period, saving 29 minutes (a 39% efficiency gain) versus sequential execution.

Methodology in Plain English

The work has three parts.

1. Defining the task. A composite task is a set of n subtasks, each an operation on a target object with an expected duration, expressed in natural language. The agent must output a schedule of s steps, and each step must pair a textual action description with a 3D location for the target object. Subtasks are split into two types: non-parallelizable, which demand continuous attention (wiping a table, dusting a shelf), and parallelizable, which only need to be started and re-checked (running a microwave, filling a sink). Efficiency comes from filling parallelizable waiting intervals with other work.

2. Building the dataset. Point clouds from five real-world datasets — ScanNet, HM3D, ARKitScenes, 3RScan, and MultiScan — are converted into textual 3D scene graphs, and GPT-4o is used to generate subtask meta-information. Outputs are refined for correctness and completeness, and expected subtask times are perturbed by ±10% to create diverse optimal schedules. A second stage computes the optimal schedule with an optimization solver, then uses GPT-4o to turn it into step-wise natural-language instructions with phrase-level object grounding; OR-based scheduling explanations are generated from templates.

3. Building the model (GRANT). Scene point clouds (N points, each with x, y, z, r, g, b) go through a sparse convolutional network and then a 3D scene encoder with K learnable scene queries that cross-attend to point features, producing scene tokens. These are prepended to text tokens and fed to an LLM (Tiny-Vicuna-1B, fine-tuned with LoRA; encoder initialized from pretrained CLASP with all weights frozen except the projection layer).

The LLM first predicts each subtask's type and expected time as constraints. A special <SCH> token passes those constraints to an external solver, which formulates the problem as a 0–1 knapsack (the waiting interval of the parallelizable subtask is the capacity; non-parallelizable durations are the item weights and values) and solves it with dynamic programming to return an optimal schedule of subtask IDs. That schedule is converted back to text via templates and re-injected into the model to guide step-wise action generation.

For localization, <GRU> tokens in the output are projected through an MLP and matched to the most similar scene query by cosine similarity; the matched query is combined with point features and passed through a sigmoid to produce a point mask. Training uses next-token cross-entropy for language and sigmoid focal loss over a binary correspondence matrix for grounding. Training details: AdamW, cosine schedule with initial learning rate 8×10⁻⁴, weight decay 0.1, 10 epochs, batch size 1, on 8× RTX 4090 GPUs.

Evaluation metrics. METEOR and ROUGE for language, AP@25% for grounding, and a new Time Efficiency (TE) metric defined as TE = (T_worst − T_pred) / (T_worst − T_opt) × 100%, where T_worst is fully sequential execution and T_opt is the solver's optimum, so 0% is purely sequential and 100% matches optimal.

Why This Matters

The paper reframes embodied planning as an efficiency problem rather than a plausibility problem, and it shows that bolting a small, exact combinatorial solver onto an MLLM through a single token outperforms asking the language model to reason about timing itself. It also demonstrates that grounding quality and scheduling quality can be improved together in one model, and it releases a dataset an order of magnitude larger and far longer in text than the compared prior datasets (311 average words versus 9–98).

Real-world applications:

• Domestic service robots that must interleave cooking, cleaning, and appliance-waiting time instead of executing instructions serially.
• Warehouse and logistics automation, where machine or oven waiting intervals can absorb pick-and-place work to shorten order fulfillment time.
• Industrial maintenance and inspection, where a robot starts a long-running process and performs inspection rounds during the wait.
• Assistive robotics for elderly or disabled users, where a compound spoken request must be converted into an efficient, physically executable sequence.

Industry relevance: The work comes from a collaboration between Huazhong University of Science and Technology and Xiaomi's MiLM Plus group, and the technique it introduces — a learnable token that hands off from a language model to an exact classical optimizer — is cheap to run (under 4 ms even at 50 subtasks) and therefore practical for on-robot deployment. It also supplies a benchmark that hardware- and robotics-oriented teams can use to compare scheduling capabilities of different 3D MLLMs.

Future Directions

• Physical robot deployment. The paper states that future work will deploy the framework on physical robots to validate robustness in dynamic environments; the current results are all on the ORS3D-60K benchmark.
• End-to-end differentiable reasoning. The authors plan to explore integrating the external optimization solver directly inside the language model so that scheduling reasoning becomes end-to-end differentiable rather than a token-mediated handoff.
• Scaling and efficiency trade-offs. The 1B versus 7B comparison shows only modest gains, leaving open how far larger LLMs, or

Authors’ abstract

Task scheduling is critical for embodied AI, enabling agents to follow natural language instructions and execute actions efficiently in 3D physical worlds. However, existing datasets often simplify task planning by ignoring operations research (OR) knowledge and 3D spatial grounding. In this work, we propose Operations Research knowledge-based 3D Grounded Task Scheduling (ORS3D), a new task that requires the synergy of language understanding, 3D grounding, and efficiency optimization. Unlike prior settings, ORS3D demands that agents minimize total completion time by leveraging parallelizable subtasks, e.g., cleaning the sink while the microwave operates. To facilitate research on ORS3D, we construct ORS3D-60K, a large-scale dataset comprising 60K composite tasks across 4K real-world scenes. Furthermore, we propose GRANT, an embodied multi-modal large language model equipped with a simple yet effective scheduling token mechanism to generate efficient task schedules and grounded actions. Extensive experiments on ORS3D-60K validate the effectiveness of GRANT across language understanding, 3D grounding, and scheduling efficiency. The code is available at https://github.com/H-EmbodVis/GRANT