MALLVI: a multi agent framework for integrated generalized robotics manipulation

AI summary

Overview

Research area: Robotics — LLM/VLM-driven task planning and control for robotic manipulation (multi-agent, closed-loop planning).

Technical level: Advanced. The paper assumes familiarity with large language models, vision-language models, open-vocabulary detection (GroundingDINO, OwlV2), segmentation (SAM), inverse kinematics, and manipulation benchmarks (VIMABench, RLBench).

Scope: MALLVi is a multi-agent framework in which specialized LLM and VLM agents decompose a natural-language instruction into atomic robot actions, ground them visually, execute them, and verify each one through visual feedback until the task succeeds or is declared unrecoverable.

What This Paper Is About

Most LLM-based robot planners generate a plan once and execute it blindly, which makes them fragile when the environment is dynamic, when objects are unfamiliar, or when a step simply fails. MALLVi addresses this by splitting the manipulation pipeline across several specialized agents that share state and communicate in a closed loop, so a vision-language model can check whether each subtask actually succeeded and trigger a targeted recovery instead of a full replan. The goal is a generalized, zero-shot pipeline that handles diverse real-world and benchmark manipulation tasks without task-specific fine-tuning.

Key Contributions

1. A distributed multi-agent architecture for robotics. Rather than one monolithic model, MALLVi coordinates specialized agents (Decomposer, Descriptor, Localizer, Thinker, Actor, Reflector) that combine LLM-based planning with VLM-based monitoring in a self-correcting process.

2. A Reflector agent that performs targeted, bounded error recovery. The Reflector evaluates the completion or failure of each subtask using visual feedback, removes successful subtasks from the queue, produces natural-language failure explanations, reactivates only the failing agent, and escalates to a full scene re-analysis by the Descriptor only when needed — avoiding costly global replanning.

3. An optional Descriptor agent providing visual memory. The Descriptor builds a spatial graph of detected objects and their pairwise spatial relationships from the initial environment state, giving downstream agents a structured scene memory to query.

4. Validation in simulation and the real world. The framework is evaluated on 8 real-world tasks (20 repetitions each), 12 VIMABench tasks across 4 partitions, and RLBench tasks over 100 repetitions, including ablations for a single-agent baseline, removal of the Reflector, and substitution of open-source backbone models.

Main Findings

• Real-world tasks: On 8 tasks with 20 repetitions, MALLVi reached 100 (Place Food), 95 (Put Shape), 90 (Stack Blocks), 90 (Shopping List), 80 (Put in Mug), 80 (Math Ops), 85 (Stack Cups), and 75 (Rearrange Objects), the highest of any method reported. Baselines were ReKep (80, 85, 75, 90, 75, 60, 40, 60), MALMM (75, 65, 55, 70, 55, 25, 50, not reported for Rearrange Objects), and VoxPoser (70, 55, 40, 45, 40, 15, 35, 0).

• VIMABench: Over 100 repetitions, MALLVi scored 100 (Simple Manipulation), 95 (Novel Concepts), 90 (Visual Reasoning), and 73 (Visual Goal Reaching). Comparisons were Wonderful Team (100, 85, 90, not reported for Visual Goal Reaching), PERIA (93, 78, 76, 68), and CoTDiffusion (86, 70, 54, 44).

• RLBench: Over 100 repetitions, MALLVi scored 92 (Put in Safe), 94 (Put in Drawer), 83 (Stack Cups), 96 (Place Cups), and 90 (Stack Blocks). PerAct scored 44, 68, 0, 0, 36 on the same five tasks.

• The Reflector matters most on complex tasks. Removing it dropped real-world Rearrange Objects from 75 to 45, VIMABench Visual Reasoning from 90 to 30 and Visual Goal Reaching from 73 to 40, and RLBench Stack Cups from 83 to 63. The paper states the pipeline still functions without the Reflector but that verification and retry significantly improve reliability.

• Single-agent collapse performs poorly. Collapsing all functionality into one LLM agent produced 25, 10, 15, 10, 30, 5, 10, 0 on the real-world tasks, and 25, 10, 15, 10 on the VIMABench partitions, indicating the single agent struggles with compositional reasoning and grounding.

• Open-source backbones degrade on compositional and multimodal tasks. Replacing the default GPT-4.1-mini backbone over 100 repetitions: GPT-4.1-mini scored 100, 95, 95, 73 across the four VIMABench partitions; Qwen-3B with Qwen-VL scored 70, 54, 10, 46; Qwen-7B with Qwen-VL scored 85, 50, 30, 62; LLaMA-3.1-8B with LLaMA-Vision-3.2-11B scored 80, 50, 27, 59. The authors attribute the residual accuracy to the separation of work into smaller per-agent duties.

• Subtask-level VIMABench breakdown: Perfect scores of 100 were recorded for visual manipulation, scene understanding, rotate, and novel noun; 95 for novel adj, same texture, same shape, and manipulate old neighbor; 90 for novel adj and noun; 85 for follow order; 76 for rearrange then restore; and 70 for rearrange.

• Extended RLBench results across nine tasks (100 repetitions): MALLVi scored 89 (Basketball in Hoop), 81 (Close Jar), 71 (Empty Container), 66 (Insert in Peg), 94 (Meat off Grill), 93 (Open Bottle), 100 (Put Block), 91 (Rubbish in Bin), and 90 (Stack Blocks), versus MALMM (82, 76, 59, 67, 88, 91, 93, 81, 47), the w/o Reflector ablation (78, 67, 42, 57, 82, 86, 95, 83, 78), and the Single-Agent approach (45, 37, 34, 26, 41, 78, 89, 41, 22).

• Grasp point selection: Bounding-box centers are described as suboptimal for irregular objects, so objects are categorized into four geometry classes. For the first three (round perfect objects, rimmed objects, and similar categories) the grasp point is assumed manually; for irregular shapes it is computed as the minimal radial distance from the centroid to the mask boundary along a uniformly random angle, using SAM segmentation masks.

Methodology in Plain English

The framework takes two inputs: a human-language instruction and a current image of the scene. From there it runs a pipeline of specialized agents that share a common state object.

• The Decomposer converts the high-level instruction into a sequence of atomic subtasks that match the robot's primitive action vocabulary (for example move, reach, push), annotating each with memory tags such as object identities, positions, and contextual references. Failed steps can be retried without replanning the whole sequence.
• The Descriptor runs in parallel, using a vision-language model to detect objects, extract spatial relationships between them, and build a spatial graph of the scene as reusable visual memory.
• The Localizer group handles perception: a Perceptor identifies task-relevant versus non-target objects and refines grasping strategies across attempts; a Grounder fuses outputs from multiple detectors (GroundingDINO and OwlV2) using a confidence-based selection mechanism that weighs each detector's predictions against the Descriptor's spatial graph; a Projector uses SAM to find candidate grasp points, verifies they lie inside the segmentation mask, and projects the 2D points into 3D using the depth map and a pinhole camera model, then converts those coordinates into joint angles through inverse kinematics.
• The Thinker turns subtask information into concrete action parameters — pick-and-place positions and rotations — using stored memory tags when the task references earlier objects, or the localizer's output directly for memoryless tasks.
• The Actor executes through a predefined API, remaining agnostic to high-level reasoning and low-level motion planning.
• The Reflector, a vision-language model, checks each executed subtask against visual feedback. Success removes the subtask from the queue; failure triggers a reattempt, a natural-language explanation, an update to shared memory, reactivation of the specific failing agent, or full scene re-analysis by the Descriptor. Persistent failure after re-evaluation is treated as unrecoverable and the run terminates with a structured failure report.

Integration is implemented with LangGraph, with a custom logging wrapper and a Dash-based log visualizer used as the primary debugging tool. Evaluation covers 8 real-world tasks, a subset of 12 VIMABench tasks drawn from 4 of its 6 partitions (out of 17 tasks total), and RLBench tasks, all in a zero-shot setting. VIMABench itself is described as 17 tabletop scenarios in an OpenAI Gym environment with objects of 29 shapes, 17 colors, and 65 textures.

Why This Matters

Impact on research: The paper argues that monolithic LLM/VLM planners create bottlenecks when tasks are ambiguous or when perception and reasoning need specialization, and that unconstrained model outputs raise safety concerns. MALLVi's contribution is showing that modularity plus a lightweight verification loop can improve generalization without retraining any component, and that removing the Reflector substantially damages performance on compositional tasks.

Real-world applications:

• Household and kitchen manipulation: placing food in appropriate containers, putting small objects into mugs, and other commonsense-constrained placement.
• Retail and logistics picking: retrieving items from a shopping-list style instruction while ignoring distractors, with memory of what has already been collected.
• Educational and symbolic sorting: the Math Ops / Put Shape task, where blocks labeled with numbers or operators must be recognized and inserted into matching slots.
• General-purpose robot assistants that receive ad-hoc natural-language commands in unstructured, changing environments, where recovery from partial failure is more valuable than a single perfect plan.

Industry relevance: The paper's open-source substitution experiment is directly relevant to deployment economics — it quantifies how much accuracy is lost when a proprietary backbone (GPT-4.1-mini) is replaced by Qwen-VL at 3B and 7B or LLaMA-Vision 3.2 at 11B, showing that smaller models remain competitive on simple manipulation but fall off on compositional and multimodal tasks. For companies choosing between cost, latency, and reliability, this is a concrete trade-off. The modular design also means individual agents can be swapped or improved independently.

Future Directions

• Adaptive low-level execution. The authors note the framework relies on predefined atomic actions, limiting adaptability under unforeseen kinematic constraints, contact dynamics, or highly dynamic environments. They propose integrating reinforcement learning, imitation learning controllers, or differentiable motion planning modules so atomic actions can be adapted at deployment time.
• Stronger perception and grounding. More sophisticated perception and grounding modules are suggested to improve performance on novel objects, complex textures, and highly dynamic scenes.
• Extending the reflection loop. The Reflector already escalates from a targeted retry to a full scene re-analysis; an open question is how much further this escalation hierarchy can be structured before it becomes as costly as global replanning.
• Broader benchmark coverage. The paper evaluated 12 of VIMABench's 17 tasks across 4 of its 6 partitions, and reports additional RLBench tasks in the appendix, leaving the remaining partitions and tasks as untested territory.

Target Audience

Robotics and embodied-AI researchers working on LLM/VLM-based task planning, manipulation, and closed-loop control; engineers building multi-agent robot systems who need to choose between monolithic and modular pipelines; and practitioners evaluating backbone-model trade-offs between proprietary and open-source LLMs for deployment. Readers should have working knowledge of LLM prompting, vision-language models, and manipulation benchmarks to get full value from the results tables.

Authors’ abstract

Task planning for robotic manipulation with large language models (LLMs) is an emerging area. Prior approaches rely on specialized models, fine tuning, or prompt tuning, and often operate in an open loop manner without robust environmental feedback, making them fragile in dynamic settings.We present MALLVi, a Multi Agent Large Language and Vision framework that enables closed loop feedback driven robotic manipulation. Given a natural language instruction and an image of the environment, MALLVi generates executable atomic actions for a robot manipulator. After action execution, a Vision Language Model (VLM) evaluates environmental feedback and decides whether to repeat the process or proceed to the next step.Rather than using a single model, MALLVi coordinates specialized agents, Decomposer, Localizer, Thinker, and Reflector, to manage perception, localization, reasoning, and high level planning. An optional Descriptor agent provides visual memory of the initial state. The Reflector supports targeted error detection and recovery by reactivating only relevant agents, avoiding full replanning.Experiments in simulation and real world settings show that iterative closed loop multi agent coordination improves generalization and increases success rates in zero shot manipulation tasks.Code available at https://github.com/iman1234ahmadi/MALLVI.