CausGround: Causal Interaction Grounding for Adaptive Embodied Planning

AI summary

Overview

Research area: Embodied AI and robot planning — specifically, using multimodal large language models (MLLMs) as high-level planners that must adapt to the physics and preconditions of a specific environment (cs.RO).

Technical level: Advanced. The paper assumes familiarity with MLLM-based planners, world-state representations, action preconditions/effects, and embodied benchmarks such as VirtualHome and RoboCasa.

Scope (one sentence): The paper proposes CausGround, a non-parametric framework that distills interaction experience into reusable, action-centered causal knowledge (preconditions and effects) and grounds it in the current observation to guide long-horizon embodied planning without any model fine-tuning.

What This Paper Is About

MLLM-based embodied planners can often infer what should be done, but they frequently fail on when an action is executable and how it changes the world — for example, attempting to place an object into a fridge before the fridge is open. Existing fixes either require costly environment-specific parameter updates or rely on unstructured interaction memory (raw trajectories, task-level recollections) that is hard to ground in a new scene and hard to reuse across tasks.

CausGround addresses this by turning interaction experience into explicit, reusable causal knowledge: for each action, which state factors must hold before it (preconditions) and which state factors change after it (effects). At planning time this knowledge is matched against a factorized representation of the current visual observation, producing a task-relevant causal graph that a frozen planner can use for backward reasoning (what must happen first) and forward reasoning (what will change next).

Key Contributions

1. An action-centered formulation of embodied interaction knowledge. The authors define interaction knowledge in terms of action preconditions and state effects, and introduce a factorized world-state representation S_t = (O_t, R_t) in which each object is a semantic category plus attribute–value pairs and each relation is a typed edge (e.g., ON(cup, table), INSIDE(cup, fridge)). This schema is shared between observations and stored causal knowledge so that preconditions and effects can be matched directly.

2. CausGround, a non-parametric distillation framework. Interaction trajectories are collected per action family and mined for preconditions (state factors that consistently distinguish successful from failed executions) and effects (state changes that consistently follow successful execution). Extracted regularities are split into generic knowledge K_gen(a), shared across grounded instances such as OPEN(fridge) and OPEN(microwave), and instance-specific knowledge K_spec(a(x)).

3. Task-grounded planning with a frozen planner. Given a task goal and initial state, CausGround builds task knowledge as K_task = K_causal ∪ K_sem by backward reasoning from an interpreted goal state plus semantic retrieval from the knowledge library, then selects actions as a_t = π_θ(I_t, S_t, g, K_task). Action effects that satisfy other actions' preconditions are linked into cross-action causal dependencies for multi-step composition.

4. Broad empirical validation. Evaluation spans adapted VirtualHome and RoboCasa simulations, the real-world egocentric SWITCH benchmark, and real-robot sequential manipulation, across multiple MLLM backbones from three model families — with no parameter updates at any point.

Main Findings

• Large gains in vision-based high-level planning on VirtualHome. Averaged over the Qwen3.5-9B, Qwen3.5-27B and Qwen3.5-35B-A3B backbones, CausGround raises average task success from 24.9% to 53.3% and average final progress from 38.4% to 71.3% relative to the Vanilla planner. Gains hold for short, medium and long horizons — for example, on Qwen3.5-27B, long-horizon success rises from 12.0% (Vanilla) to 44.0% and long-horizon progress from 31.9% to 77.2%.

• Improved closed-loop replanning on RoboCasa. Averaged across backbones, one-shot planning success rises from 26.9% to 38.9% and closed-loop replanning success from 43.0% to 57.0%. On Qwen3.5-27B replanning, success goes from 57.8% to 75.0%.

• More reasoning is not automatically better. The two stronger baselines sometimes underperform the Vanilla planner. SelfReVision can introduce unsupported prerequisites and corrupt an otherwise valid plan (e.g., wrongly inferring an object is inside a cabinet and inserting unnecessary OPEN actions); LookPlanGraph's discrete scene graphs can propagate grounding errors (e.g., reading a closed microwave as open) and cause the planner to skip necessary actions.

• Better interaction reasoning on SWITCH. Across the four probed categories — Action, Recovery, Verification Action, Verification State — average accuracy rises from 25.8% to 31.9% over Vanilla across backbones. Per-backbone averages: Qwen3.5-9B 27.7% to 32.9%, Qwen3.5-27B 27.9% to 35.2%, Qwen3.5-35B-A3B 21.7% to 27.7%.

• Both components contribute, and they contribute differently. In the component ablation, Vanilla reaches 33.3% SR / 46.9% progress; adding only the factorized state gives 36.0% / 55.1%; adding only causal knowledge gives 53.3% / 70.8%; the full CausGround reaches 62.7% / 79.8%. The authors attribute this to the shared schema letting the planner match observed state directly against retrieved preconditions and effects.

• The benefit generalizes across model families and scales. Evaluating six models from Qwen3.5, InternVL3.5 and Gemma4, ranging from E4B to 38B parameters, CausGround improves average success from 23.3% to 50.7% and final progress from 38.4% to 67.3% over Vanilla.

• Real-world feasibility. In a physical robot setting, CausGround interprets the goal state and reasons backward over causal dependencies to derive an executable sequence — Open(Microwave), Pick(Bread), Place(Bread, Plate) — and completes the multi-step interaction. A qualitative VirtualHome case shows Vanilla repeatedly executing SIT without satisfying the NEAR precondition, while CausGround recovers the missing prerequisite.

Methodology in Plain English

The approach has two phases.

Phase one — building the knowledge base. Rather than memorizing whole trajectories, CausGround collects experience at the level of individual actions (GRAB, OPEN, etc.) under deliberately varied starting states. For each grounded action, it looks across many trials and asks two questions: which state factors were true in the successful trials but not the failed ones (those become preconditions), and which state factors changed consistently after successful executions (those become effects). Incidental factors are discarded. Comparing instances of the same action family lets the system separate regularities that hold everywhere (generic knowledge) from those that hold only for a particular object (instance-specific knowledge). All of this is written into the same object-attribute-plus-relation schema used to describe observations, so the two can be compared factor by factor.

Phase two — planning. Before each task, the system converts the natural-language goal into a goal state in that same schema and reasons backward from it through effects and preconditions, gathering the causal dependencies that lead to the goal; in parallel it semantically retrieves related generic and instance-specific knowledge. This merged set, grounded against the current observation, forms a local causal graph handed to a frozen MLLM planner together with the RGB image, the extracted world state and the task description. The planner uses preconditions to judge which actions are currently executable and effects to anticipate what happens next.

Evaluation setup. VirtualHome was converted into a vision-based closed-loop benchmark: 75 household tasks (25 each in short, medium and long splits, with 3–6, 7–10 and 11–22 reference actions respectively; 643 reference actions total), spanning six scenes, 33 action types and 78 object classes, with interaction budgets of 20, 40 and 60 attempts. Because VirtualHome's native renderer sometimes produces images inconsistent with the underlying simulator state, the authors used Qwen-Image-Edit-2511 to repair those frames under a constrained prompt (no future states, no hallucinated occluded objects, unrelated content preserved), validating each repair against the simulator state. RoboCasa tasks were adapted by treating annotated skill stages as atomic actions. Success rate and final progress are reported, with progress measuring partial completion of a private semantic task specification; plans need not match the reference action sequence. All methods share the same planner backbone, observations, task descriptions and action space, and all planner parameters stay frozen.

Why This Matters

The paper argues that the bottleneck for deploying MLLM planners in new environments is not raw reasoning ability but missing, environment-specific interaction structure — and that this structure can be externalized and reused rather than baked into weights. That reframes adaptation as a knowledge-representation problem, which is cheaper to iterate on and auditable, since preconditions and effects are explicit rather than implicit in a fine-tuned policy.

Real-world applications:

• Household service robots that must respect ordering constraints (open before place, grasp before move) across kitchens and appliances they were not trained on.
• Warehouse and logistics manipulation, where pick-and-place sequences require verified preconditions such as a container being open or a gripper being empty.
• Assistive robotics for older adults or users with motor impairments, where a failed action is not merely inefficient but potentially unsafe, and where recovery from precondition violations matters.
• Rapid deployment in new facilities, where a robot must absorb site-specific interaction rules without a fine-tuning pipeline or a large site-specific dataset.

Industry relevance: the method is non-parametric, so a deployed system can accumulate and swap interaction knowledge per site while keeping the underlying MLLM frozen — attractive when models are large, proprietary, or served behind an API, and when re-certifying a fine-tuned policy would be costly. The reported gains across three model families and sizes from E4B to 38B suggest the wrapper works with off-the-shelf backbones rather than requiring a specific one.

Future Directions

• How is the knowledge library built and maintained at scale? The paper's library is constructed through controlled variations of pre-action factors; how well knowledge mined in one environment transfers to a genuinely different one — and how conflicts between generic and instance-specific knowledge are resolved — remains open.
• What happens when the factorized state extraction is wrong? The paper notes that baselines suffer from grounding errors propagating into decisions; the degree to which CausGround's own MLLM-extracted state is robust to noisy perception, and how errors compound over long horizons, is a natural next question.
• Beyond high-level discrete actions. The evaluated actions are atomic high-level templates; extending precondition/effect reasoning to continuous control or low-level manipulation skills is not addressed here.
• Scaling the evaluation. Results cover 75 VirtualHome tasks, adapted RoboCasa tasks, four SWITCH categories and a small number of real-robot demonstrations; broader real-world task diversity and larger physical evaluations would test whether the reported gains hold outside the curated settings.

Target Audience

Researchers and engineers working on embodied AI, robot task planning, and MLLM-based agents — particularly those interested in non-parametric adaptation, neuro-symbolic state representations, and precondition/effect modeling. It is also useful for practitioners who need to deploy a frozen foundation model in a new physical environment and want a reusable external knowledge layer rather than a fine-tuning pipeline. Readers without background in embodied planning will find the core idea accessible, but the benchmark protocols and baseline adaptations assume familiarity with the literature.

Authors’ abstract

Multimodal large language models (MLLMs) show promising reasoning capabilities, but reliable physical interaction requires planning that accounts for when actions are executable and how they change the world. These preconditions and effects are environment-dependent and often not captured by MLLMs, making adaptation challenging. Yet existing approaches either require costly environment-specific training or rely on unstructured interaction experience that is difficult to reuse. Therefore, we introduce CausGround, a non-parametric causal interaction grounding framework that distills interaction experience into reusable action-centered causal knowledge. CausGround identifies action preconditions and effects and organizes them in a factorized world-state representation capturing object attributes and inter-object relations. At planning time, task-relevant causal dependencies are grounded in current observations to support backward reasoning over preconditions and forward reasoning over effects for action selection and multi-step planning. We evaluate CausGround on simulated high-level planning, interaction reasoning, and real-world robot manipulation. On adapted VirtualHome and RoboCasa benchmarks, CausGround improves vision-based high-level planning across MLLM backbones, increasing average task success from 24.9% to 53.3% on VirtualHome and closed-loop planning success from 43.0% to 57.0% on RoboCasa. On SWITCH, average interaction reasoning accuracy increases from 25.8% to 31.9%, indicating improved action selection, recovery, and verification. Moreover, real-world experiments show that CausGround reasons backward from MLLM-interpreted goals to derive executable high-level action sequences for sequential manipulation. These results demonstrate that reusable action preconditions and effects support long-horizon embodied planning across diverse environments without parameter updates.