Human-Centric Open-Future Task Discovery: Formulation, Benchmark, and Scalable Tree-Based Search

AI summary

Overview

• Research area: Computer Vision / Embodied AI, at the intersection of Large Multimodal Models (LMMs), robot task discovery, and human-robot collaboration.
• Technical level: Advanced.
• Scope (one sentence): The paper formalizes the problem of finding robot tasks that stay useful across many possible human futures, releases a benchmark (HOTD-Bench) with a simulation-based evaluation protocol, and proposes a multi-agent search-tree method (CMAST) that outperforms existing LMMs on that benchmark.

What This Paper Is About

Robots powered by LMMs can currently propose actions based on what they see happening right now, but real people run several activities at once, change their intentions mid-stream, and rarely spell out their future plans. The paper defines a new problem, Human-centric Open-future Task Discovery (HOTD): inferring tasks that reduce human effort across a range of plausible, uncertain futures — for example, proactively wiping a table because that helps whether the person later cooks, cleans, or rests. It then builds a benchmark and a method to attack that problem.

Key Contributions

1. Formalizes HOTD. The paper gives a formal definition of the problem and of "human-centric tasks," grounded in a cost function: a task is helpful if inserting it lowers the cost of reaching the human's latent goal, and unhelpful otherwise. Human-centric tasks are placed at the middle ("action") level of a three-level taxonomy (action primitive, action, activity).
2. Builds HOTD-Bench. A benchmark of 2450 curated clips from two video datasets — Toyota Smarthome Untrimmed (TSU, contributing 2K videos) and Charades (CHA, contributing 0.4K videos) — totaling nearly 40 hours, drawn from two different distributions.
3. Provides two evaluation routes. A simulation-based protocol in which an LLM simulator deduces the future outcome of any candidate task, plus an open-vocabulary label-based evaluation whose "ground truth" is produced by a semi-automated annotation pipeline following three principles (Helpful, Non-disruptive, Executable).
4. Proposes CMAST. The Collaborative Multi-Agent Search Tree framework, which combines a scalable search tree module with a collaborative multi-agent system of 7 agents, is training-free, and can be plugged into existing LMMs.

Main Findings

• Existing LMMs are weak at HOTD. Under simulation-based evaluation, all baseline LMMs achieve relatively low Valid Task Ratio, ranging from 29.1% to 62.4%.
• CMAST leads on Valid Task Ratio. CMAST surpasses the second-best method by 15% to 22% in the TSU subset and reaches the same level as the second-best method in the CHA subset. Its best reported TSU numbers are 3.90 Valid Task Count and 71.7% Valid Task Ratio at 20 seconds of observation, and 3.89 / 72.0% at 60 seconds.
• CMAST is competitive on Valid Task Count. It outperforms the second-best method by an average of 7.6% in the TSU subset, which the authors attribute to the search tree module exploring many future scenarios and activity procedures.
• Bigger models are not automatically better. Across the baselines, larger models show no clear advantage over smaller ones, and there is generally a trade-off between Valid Task Count and Valid Task Ratio — for example, InternVL2-8B has the highest Valid Task Ratio among baselines but a low Valid Task Count, while LLaVA-Next-Video-7B has the highest Valid Task Count but a low ratio.
• The search tree is the key component. Removing the entire search tree module and replacing it with a single LMM agent that directly predicts future actions reduces the Valid Task Ratio by 37%; the authors argue a single agent produces confined prediction paths.
• More search buys more tasks. Greedy search (beam = 1) discovers only 1.4 helpful tasks per video. As more time-intensive search strategies are used, the number of discovered tasks increases significantly while the accuracy ratio stays roughly unchanged.
• The framework transfers across LMMs. Swapping component agents (CMAST-LLaVA as the main version, CMAST-InternVL2, CMAST-Qwen2), the LMMs inside the framework improve the Valid Task Ratio by at least 39% compared with the same LMMs used in isolation.
• The simulator agrees with people. In a human evaluation, 25 helpful and 25 unhelpful tasks marked by the simulator were presented in mixed order to 5 annotators, and the ratings showed strong agreement between the simulator's judgments and human preferences.
• Human-level comparison. On 10 randomly selected examples, the CMAST framework achieves performance comparable to human participants; the authors state this is meant to show potential, not to claim a surpass of human capability.

Methodology in Plain English

The authors first separate "the task itself" from "whether it helped." Labeling every helpful task for every possible future is impractical, so instead of trying to enumerate good answers, they verify candidate answers: given a proposed task, a simulator rolls the future forward with and without the robot's intervention, models how the human would adapt, and compares the time cost of the resulting action sequences. Because the simulator is an LLM with chain-of-thought prompting, it can reason about futures that never actually occurred in the video. The same team also builds a cheaper, more stable label-based evaluation by taking the dense action annotations in the source datasets, selecting actions the person ends up performing, expanding them into full task sentences with Qwen-VL, and filtering them through three principles: the task must be helpful, non-disruptive, and executable.

For the method, CMAST treats future reasoning as a tree. The trunk is the observed history action sequence, one node per action. Past that point the tree branches into possible next actions, each expanded by a Next Action Prediction Agent and scored by a Likelihood Estimation Agent, with a Redundancy Removing Agent pruning branches; a "finish activity" node marks the end of an activity. The team uses a pruned exhaustive search with a 0.5 probability threshold. Seven agents split the work: scene description, history action recognition, next action prediction, likelihood estimation, redundancy removing, dependency recognition (to drop actions with unmet prerequisites), and task converting (to phrase each remaining action as a robot task). The three LMM agents use LLaVA-Next-Video and the remaining agents use Qwen-LM. Nothing is fine-tuned — the whole pipeline is training-free, which is what lets it be swapped onto different base models. Results are reported with two metrics: Valid Task Count (how many helpful tasks are found) and Valid Task Ratio (what fraction of the predictions are helpful), measured at different observation lengths (for example, vc@40 means 40 seconds of observed video).

Why This Matters

Impact on research. Task discovery has largely assumed fixed goals or closed environments. This paper reframes the target as anticipatory usefulness across open futures, and it contributes both a problem statement and an evaluation infrastructure. The simulation-based evaluator is notable because it can score hypothetical futures that never appear in the dataset, which observational benchmarks cannot do.

Real-world applications.

• Domestic robots that pre-emptively handle chores (the paper's own "wipe the table" example) regardless of what the human does next.
• Assistive and elderly-care robots that reduce the effort a person spends on daily activities.
• Service and hospitality robots that need to act usefully when human intentions shift mid-task.
• Any embodied system that gathers its own training experience by proposing physically interactive tasks worth attempting.

Industry relevance. Because CMAST is training-free and improves whatever LMM it wraps, it is a drop-in style enhancement rather than a retraining project. The finding that larger models do not automatically win also matters for deployment economics: capability here comes from structuring the reasoning, not simply from scaling the model. The benchmark itself gives industry a way to compare candidate models on human-centric assistance rather than generic dialogue quality.

Future Directions

• Broaden the benchmark. HOTD-Bench draws on TSU and CHA only; extending to other activity domains and distributions would test whether CMAST's advantages generalize.
• Enrich the cost function. The simulator defines cost in terms of time consumption and evaluates relative rather than absolute time. Other notions of effort — physical labor, cognitive load, risk — are left unmodeled.
• Reduce reliance on the LLM simulator. The current evaluation leans on an LLM's world knowledge and is validated as "simple yet reliable," but a hand-verified or hybrid evaluator could remove the dependence on a model that may itself be biased.
• Move from benchmark to robot. The work is a benchmark and reasoning study; whether the discovered tasks execute reliably on physical robots, under real safety and prerequisite constraints, is not demonstrated here.
• Explore search strategies further. The paper compares greedy and beam search, but the trade-off frontier between test-time thinking budget and task quality is only partly mapped.

Target Audience

Researchers and graduate students working on embodied AI, robot task planning, LMM reasoning, and human-robot interaction; benchmark designers interested in open-ended evaluation protocols; and industry engineers building assistive or domestic robots who want a training-free reasoning layer that improves an existing multimodal model without retraining it.

Authors’ abstract

Recent progress in robotics and embodied AI is largely driven by Large Multimodal Models (LMMs). However, a key challenge remains underexplored: how can we advance LMMs to discover tasks that assist humans in open-future scenarios, where human intentions are highly concurrent and dynamic. In this work, we formalize the problem of Human-centric Open-future Task Discovery (HOTD), focusing particularly on identifying tasks that reduce human effort across plausible futures. To facilitate this study, we propose HOTD-Bench, which features over 2K real-world videos, a semi-automated annotation pipeline, and a simulation-based protocol tailored for open-set future evaluation. Additionally, we propose the Collaborative Multi-Agent Search Tree (CMAST) framework, which decomposes complex reasoning through a multi-agent system and structures the reasoning process through a scalable search tree module. In our experiments, CMAST achieves the best performance on the HOTD-Bench, significantly surpassing existing LMMs. It also integrates well with existing LMMs, consistently improving performance.