COOPERA: Continual Open-Ended Human-Robot Assistance

AI summary

Overview

Research area: Robotics and embodied AI — specifically human-robot collaboration (HRC), long-horizon task inference, and LLM/VLM-driven assistive agents.

Technical level: Advanced. The paper assumes familiarity with LLM/VLM prompting, vision-language models, supervised classification, 3D household simulation (Habitat), and reinforcement-free planning benchmarks.

Scope in one sentence: COOPERA is a simulation framework, benchmark, and method that studies continual, open-ended human-robot collaboration over multiple days using simulated humans whose whole-day behavior is driven by psychological traits, habits, and long-term intentions.

What This Paper Is About

Most robot assistants are evaluated in short, episodic settings with tasks specified in advance, and they have no model of the person they are helping. COOPERA addresses this by placing an LLM-driven simulated human — with Big-5 personality traits, habits, and hour-by-hour intentions — into 3D household scenes, and asking a robot to infer and assist with the human's goals without being told what those goals are. The goal is to see whether a robot can become a progressively better, more personalized collaborator across days by learning from human behavior and end-of-day feedback.

Key Contributions

1. COOPERA framework. A novel HRC framework for continual, open-ended collaboration with humans who exhibit individual traits across long horizons, including a human model with preferences, a feedback mechanism, benchmarks, and metrics.
2. Trait-driven human simulation. A method for simulating humans with long-term behavior models driven by individual traits and habits, producing dynamic intention-driven, environment-conditioned, and traits-driven behavior.
3. A benchmark and personalization method. Within the framework, a benchmark plus an approach that decouples robot task inference into intention inference and task inference, maintaining a human profile that is updated with daily feedback so collaboration becomes increasingly adaptive and personalized over multiple days.
4. Validation of the simulated humans and the framework. Experiments measuring whether simulated humans reflect realistic human behavior, whether evaluation metrics correlate with real human verification, and whether the robot generalizes to new scenes and new humans.

Main Findings

• Simulated humans are machine-distinguishable. Two fine-tuned BERT-large-uncased classifiers identified which of 10 simulated humans produced a behavior: intention classification accuracy 0.995 and task classification accuracy 0.830. Task classification was harder than intention classification because tasks such as drinking water can map to multiple intentions.
• Simulated humans are diverse. Average per-trait standard deviation of Big-5 traits (1–5 scale) is reported as 0.939 for aligned and 0.342 for mismatched in Table 1; the high value exceeds the typical 0.7–0.9 range in real-world distributions cited by the paper.
• Inferred traits track psychometric ground truth. Pearson correlation between robot-inferred Big-5 scores and ground truth drops to -0.497 for one-step mismatched pairs, which the authors say confirms alignment between inferred traits and psychometric data.
• Human behavior shows temporal dependence. A next-intention prediction task reached accuracy 0.789 and F1 0.790, indicating that a person's current-hour intention depends on earlier hours.
• User studies. With 25 participants each: 0.764 accuracy on multiple-choice questions identifying the same simulated human across days and scenes, and 0.712 accuracy on matching trait descriptions to full-day behaviors. Identifying the same human was easier than distinguishing between different humans.
• Simulated intentions align with real human intentions. Against generic prompting (SBERT 0.554, OpenAI embeddings 0.537) and mismatched pairs (SBERT 0.523, OpenAI embeddings 0.543), the paper's main method reached 0.810 with SBERT (all-mpnet-base-v2) and 0.772 with OpenAI embeddings (text-embedding-3-small). Six participants recorded daily intentions over five days.
• The proposed assistive agent improves the most within a day. In predicate-based evaluation, the method achieved the highest within-day improvement, while prompting, random, and oracle showed little to no improvement or even declined.
• The proposed assistive agent also improves the most across days, second to oracle. Prompting and finetuning showed minimal gains, which the authors attribute to their tendency to build a 1-to-1 mapping between time and human intentions/tasks.
• Generalization to a new scene is easier than to a new human. Scene generalization rose from 0.269 to 0.465 with finetuning; human generalization rose from 0.258 to 0.343.
• Evaluation metrics agree with real humans. L1 distance between predicate-based labels and real-human verification ranged from 0.085 to 0.120 across the four settings, and between LLM-based labels and real-human verification from 0.075 to 0.080.
• Offline real humans and human-in-the-loop collaboration work. Predicate-based success rates on the final day were 0.498, 0.471, 0.426, 0.322 for offline real humans and 0.488, 0.467, 0.431, 0.349 for human-in-the-loop across Settings 1–4 — comparable to simulated humans and sometimes higher.
• Ablations confirm the design choices. Removing profile extension lowered intention classification to 0.950 and task classification to 0.800; single-shot (all-day) intention proposal lowered temporal dependence to 0.751 accuracy and 0.740 F1. Removing trait inference hurt most in multi-human settings (0.239 and 0.206 in Settings 3 and 4), removing temporal context hurt all settings, and swapping the robot backbone to LLaVA-1.6-Mistral-7B still produced reasonable success (0.487, 0.424, 0.362, 0.310), versus 0.505, 0.465, 0.439, 0.344 for the main method.

Methodology in Plain English

Simulating the human. The researchers sample conversations from the Synthetic-Persona-Chat (SPC) dataset and prompt an LLM to write an extended profile — job, preferences, common activities — and to produce Big-5 (openness, conscientiousness, extroversion, agreeableness, neuroticism) scores, taking a majority vote across five inference trials with 0.5 bins on a 1–5 scale. Ten human profiles are generated. Each day is modeled as 12 one-hour intervals from 9 am to 9 pm (the rest is sleep). At the start of each hour an LLM proposes a high-level intention (for example, leisure) and decomposes it into a sequence of tasks, using the environment's object list, the human's traits, and the history of prior intentions and tasks. To keep long contexts manageable, the pipeline uses a semantic Search over objects and motions and a Memory Retrieval step combining recency (decay factor λ = 0.95) with relevance, retrieving the top 3 intentions and top 5 tasks; two rounds of Reflexion correct errors about traits, temporal dependencies, and object use. Behavior diversity across days is encouraged by resetting history daily and using a high temperature. Tasks are executed as chained 3D whole-body motions from Motion-X and AMASS, rendered in SMPL-X format.

The environment. Simulation runs on Habitat 3.0 with HSSD scenes containing 18,656 static objects, plus 20 dynamic objects sampled from the YCB Dataset placed in contextually appropriate locations. Five scenes were selected, varying in rooms (4–11), static objects (51–140), and dynamic objects (33–94).

The robot's approach. The robot sees frames from a video of the human's first task. A robot-VLM (Llama-3.2-11B) proposes a superset of possible intentions, a binary intention classifier filters them, the VLM then proposes a superset of tasks for the surviving intentions, and a task classifier selects the tasks to assist with. Classifiers are fine-tuned on Mistral-7B-Instruct-v0.2 with LoRA in instructional yes/no format. At the end of each day the human gives feedback; the robot summarizes an inferred profile (traits, habits, psychometrics), which is fed back into the VLM prompts and classifiers for later hours and later days. The human LLM used for simulation is Llama-3.1-8B at temperature 0.7, and search/retrieval uses MiniLM-L6-v2.

Evaluation. Two collaboration types are defined: type 1 is an open-ended variant of the Watch-and-Help challenge, where an intention decomposes into 3 pick-and-place tasks and the robot gets the first task's video plus a text description; type 2 is harder, with 5 tasks per intention involving free-form motion around static objects and only the first task's video, with no text. Four settings increase difficulty: same human and scene (5 days, 1 scene); same human, different scenes (5 days, 5 scenes); different humans, same scene (9 days, 1 scene, rotating Humans 1–3); and different humans, different scenes (9 days, 3 scenes). Baselines include Direct Prompting, Direct Finetuning, Oracle, Random, Intention Agnostic, and Human & Context Agnostic. Success is measured by F1-based success rate under three evaluation methods: predicate-based functions (following Watch-and-Help), LLM judgment, and human verification.

Why This Matters

Impact on research. COOPERA reframes HRC evaluation from short, predefined episodes toward multi-day, open-ended collaboration in which the robot must reason about who the human is, what time it is, and what that person has already done. It provides a reusable human model, benchmark, and metric suite, and it shows that LLM/VLM-based evaluation correlates with real human judgment in these settings.

Real-world applications (as motivated by the paper's household assistance scenario):

• Home robots that adapt to personal preferences over time — the paper's own example is a robot that understands someone may prefer coffee cooler in the morning but stronger in the

Authors’ abstract

To understand and collaborate with humans, robots must account for individual human traits, habits, and activities over time. However, most robotic assistants lack these abilities, as they primarily focus on predefined tasks in structured environments and lack a human model to learn from. This work introduces COOPERA, a novel framework for COntinual, OPen-Ended human-Robot Assistance, where simulated humans, driven by psychological traits and long-term intentions, interact with robots in complex environments. By integrating continuous human feedback, our framework, for the first time, enables the study of long-term, open-ended human-robot collaboration (HRC) in different collaborative tasks across various time-scales. Within COOPERA, we introduce a benchmark and an approach to personalize the robot's collaborative actions by learning human traits and context-dependent intents. Experiments validate the extent to which our simulated humans reflect realistic human behaviors and demonstrate the value of inferring and personalizing to human intents for open-ended and long-term HRC. Project Page: https://dannymcy.github.io/coopera/