IMBench: A Benchmark for Intuitive Robotic Manipulation

AI summary

Overview

• Research area: Robotics — benchmarks and datasets for robot learning, robot manipulation, physical reasoning, and vision-language-action (VLA) models.
• Technical level: Advanced (assumes familiarity with robot learning benchmarks, VLMs, VLA policies, and simulation environments such as robosuite).
• Scope: The paper introduces IMBench, a 35-task robosuite benchmark with roughly 14K curated trajectories that measures whether models can convert physical understanding into executable manipulation actions, and reports that both frontier VLMs and state-of-the-art policies fall well short of that integration.

Note: The paper is from Manav Robotics (Bengaluru, India), is posted as arXiv:2607.15641v1 [cs.RO], and is licensed CC BY 4.0. The published content provided to this summary is truncated partway through the per-task appendix specifications.

What This Paper Is About

Humans solve manipulation problems by combining physical intuition with motor control — for example, sliding a thin plate to the table edge before grasping it because a direct top-down grasp is geometrically impossible. The paper calls this combined capability intuitive manipulation: the conversion of physical understanding into goal-directed action. The authors argue existing benchmarks test either physical reasoning or policy execution in isolation, and they build IMBench to measure the whole path from perception to reasoning to closed-loop execution under explicit physical constraints.

Key Contributions

1. IMBench, a 35-task robosuite benchmark designed around "inference bottlenecks" (hidden facts such as infeasibility, geometry, dynamics, causality, hidden state, or stability) that must be inferred from observations for the task to be solved reliably; tasks span single-arm and bimanual setups with parallel-jaw and suction grippers.
2. A dataset and tooling release: approximately 14K filtered trajectories (6,000 teleoperation + 8,000 scripted = 14,000) plus scripts to generate more data, all passed through a three-stage curation pipeline (human scoring on a 1-5 scale keeping only scores ≥ 4, VLM-based anomaly and consistency checks, then a final human review).
3. A formalization of intuitive manipulation as an Understand–Infer–Act decomposition, evaluated with a three-stage VLM protocol (constraint understanding → plan proposal → execution) plus closed-loop policy rollouts.
4. An empirical characterization of failure modes showing where current foundation models and generalist policies break down, including an out-of-distribution (OOD) generalization study along each task's target physical axis.

Main Findings

• VLMs understand constraints better than they plan actions. GPT-5.5 reaches approximately 74% on constraint understanding (Stage 1) but drops to approximately 70% on high-level planning (Stage 2) in the abstract's summary of results; in Table 1, Claude Sonnet 4.6 scores 74.5% ± 31.2 and GPT-5.5 scores 74.1% ± 33.0 on Stage 1, while GPT-5.5 leads Stage 2 at 69.5% ± 35.7 (Sonnet 4.6: 64.1% ± 34.6).
• Execution collapses even when plans are correct. In Stage 3, closed-loop execution success falls to 11% per the abstract; the table reports IMBAgent at 11.3% mean and IMBAgent-obj (privileged object poses) at 18.8% mean across the evaluated tasks. GPT-5.5 as an agent achieved non-zero success on only three of sixteen evaluated tasks.
• Failures concentrate on sequencing and timing, not constraint recognition. Tasks such as slide-catch had Stage-1 scores of 88–95% but saw most models fail completely at Stage 2 because plans missed timing and tracking steps; similar failures appeared on tool-retrieve, cup-inversion, stack-collapse-recovery, and cup-extract. In contrast, gap-funnel, sheltered-grasp, ramp-sort, and domino-single transferred from understanding to planning at 85–100%.
• Stability tasks are the hardest category for reasoning. P7 (stability and equilibrium) was the weakest overall category at 37–80% except for the miscellaneous category. Interestingly, balance-hard often outperformed balance-medium because its prompt explicitly encouraged center-of-mass reasoning.
• End-to-end policies are near-zero zero-shot and weak after finetuning. Zero-shot VLA performance was ≤ 0.02 mean. Category-averaged means in Table 2: π0.5 zero-shot 0.01 → finetuned 0.15; GR00T N1.5 zero-shot 0.00 → finetuned 0.02; Diffusion Policy trained from scratch 0.24. Rollouts were over 20 seeds and 5 rollouts each.
• Different policies succeed on different task types. Diffusion Policy performed best on mirror reasoning and contact-tolerant tasks (mirror-pick-place, domino-select, occluder-push, slip-recovery), while finetuned π0.5 mainly succeeded on balancing tasks (balance-medium, balance-hard).
• Some tasks remain unsolved by every policy, including cup-inversion, shape-stack, stack-collapse-recovery, packing, mass-sort, and both cracker-box grasp variants.
• Generalization is poor under targeted perturbation. balance-medium dropped from 0.71 in-distribution to 0.12 under a center-of-mass perturbation, and keyboard-typing fell from 0.32 to 0.04; contact-tolerant tasks such as domino-single remained relatively robust. A few tasks showed small improvements under perturbation, but overall performance degraded.
• Privileged state can sometimes hurt. Providing object poses on mirror-pick-place caused the agent to follow reported coordinates directly without applying the required 180° mirror transformation.
• Cost is a practical constraint on evaluation. The paper states that a complete Stage 3 evaluation across 16 tasks costs approximately $1800, which is why only GPT-5.5 was benchmarked as an agent in Stage 3.

Methodology in Plain English

The authors built 35 manipulation tasks in the robosuite simulator, each designed around a hidden physical fact the agent must work out for itself — for example, that a flat plate cannot be grasped directly (P1 geometry), that a thrown cube follows projectile motion (P2 dynamics), that a target can only be reached through an intermediate contact chain (P3 causal/indirect action), that a tool is needed for reach (P4), that the decision-relevant variable is hidden behind an occluder (P5), that the scene will change mid-execution (P6), or that success depends on static balance (P7). They added two miscellaneous tasks: mirror-pick-place, which rotates camera frames by 180° to decouple observation from action frames, and keyboard-typing, a bimanual task.

Tasks use a Gymnasium-style interface with seeded resets, sparse rewards (1 for success, 0 otherwise), termination on simulator-derived success, failure (e.g., object leaves the workspace or a forbidden contact), or a time limit. Observations include synchronized 2–4 RGB views, proprioception, gripper state, and wrist force/torque; actions are a unified continuous space of 6-DoF end-effector deltas plus gripper commands for both single-arm and bimanual setups.

Evaluation is split into two halves. For high-level reasoning, five models (GPT-5.5, GPT-5.4-mini, Gemma-4, Claude Haiku-4.5, Claude Sonnet 4.6) receive multi-view renders, the task description, and the controller interface, and produce chain-of-thought outputs scored on two stages: constraint understanding against a task-specific rubric, and plan correctness (a human verifies whether proposed sub-goals satisfy task constraints and can achieve success). Results are aggregated over five temperature settings. Stage 3 extends this into a closed-loop ReAct framework where the VLM selects predefined action primitives with iterative feedback after each step, using GPT-5.5 in both vision-only and privileged object-centric settings. For low-level execution, three visuomotor policies — Diffusion Policy trained from scratch per task, π0.5, and GR00T N1.5, the latter two evaluated zero-shot and finetuned — are trained or finetuned on 200 curated human demonstration trajectories per task, with scripted policies used where teleoperation is infeasible. Training configurations were kept as close as possible to the original implementations.

Why This Matters

The paper positions intuitive manipulation as a "missing axis" of capability that is orthogonal to what current benchmarks reward: physical-reasoning suites such as PhysBench and CLEVRER stop at inference and never synthesize executable actions, while manipulation benchmarks such as LIBERO, RoboCasa, ManiSkill, RLBench, Colosseum, VLABench, and OGBench measure policy execution without isolating whether success depends on reasoning about task-specific physical constraints. IMBench is presented as a measurement instrument to localize where intuitive physical intelligence lives in current systems.

Real-world applications:

• Warehouse and logistics manipulation, where grasping flat or flush objects, slot insertion, and contact-rich placement are routine and fail under naive top-down grasps.
• Household and service robots, which must handle tool use, occluded objects, and multi-stage tasks like inverting cups or packing items under tight geometric tolerances.
• Bimanual and handover-based assembly or kitchen tasks, where asymmetric grippers force explicit role splits, such as suction-to-jaw handovers.
• Safety-critical or disturbance-prone settings, where mid-execution events such as gripper slip or a knocked stack require online failure detection and replanning rather than replay of a memorized trajectory.

Industry relevance: The results suggest that near-saturating scores on existing manipulation benchmarks can coexist with near-zero success on physics-grounded tasks — a caution for anyone deploying VLA policies commercially. The paper also notes that the evaluated baselines do not use the released tactile and force-feedback signals, leaving a concrete direction for improving performance.

Future Directions

• Extend beyond the current embodiment and scope. IMBench is limited to simulation and evaluates only Franka with parallel-jaw and suction grippers, excluding dexterous hands, mobile platforms, and humanoids; tasks are low-to-mid horizon, task descriptions are fixed, and deformable object manipulation is not considered.
• Integrate tactile and force-feedback modalities. The authors release force-torque and tactile signals, but no evaluated baseline uses them, and principled integration "remains an open direction that may improve performance."
• Close the understanding-to-execution gap. The largest reported gap is between correct high-level plans and successful execution, and all tasks requiring precise alignment, timing, tool use, hidden-state reasoning, or balancing scored 0% in both agent settings.
• Improve robustness to out-of-distribution physical shifts. Policies appear to rely on memorized patterns rather than robust physical understanding, given the sharp drops measured under perturbations along each task's target physical axis; comparing learned approaches against explicitly reasoning planners such as TAMP is left as an open comparison, since IMBench focuses on generalist agents rather than task-specific engineered systems.

Target Audience

Researchers and engineers working on robot manipulation benchmarks, vision-language-action models, and embodied physical reasoning will benefit most. It is also relevant to practitioners evaluating whether generalist robot policies can be trusted on tasks where success hinges on physical feasibility rather than semantic generalization, and to benchmark designers interested in the Understand–Infer–Act decomposition and the three-stage VLM evaluation protocol.

Authors’ abstract

Humans combine reasoning and motor control to solve complex manipulation tasks under diverse constraints. They build an understanding of the physical world that helps them convert reasoning into actions and quickly adapt to new scenes, tasks, and rules. We refer to this capability as intuitive manipulation. Existing benchmarks fail to capture this integration: they evaluate physical reasoning in isolation from execution, or measure policy performance without requiring explicit reasoning. We introduce IMBENCH, a benchmark designed to evaluate intuitive manipulation as an integrated capability spanning perception, physical reasoning, action generation, and iterative execution. Our tasks require models to infer task-relevant physical structure and generate feasible action sequences under explicit constraints, including contact-rich manipulation, tool use, and multi-stage dependencies. We introduce a benchmark of 35 tasks, 14K filtered trajectories, and scalable tools for generating diverse scenarios. Experiments reveal a consistent gap: vision language models show partial physical reasoning ability but fail to produce executable plans, while state-of-the-art vision-language-action models struggle to satisfy task constraints and generalize across scenarios. These results identify intuitive manipulation as a missing axis in current foundation models and generalist robot policies, and position IMBENCH as a step toward evaluating and enabling more integrated, adaptive physical intelligence.