PackLab: A Comprehensive Framework for Developing, Training, and Evaluating MLLMs in Robotic Bin Packing

AI summary

Overview

Research area: Robotics, specifically multimodal large language models (MLLMs) applied to robotic bin packing, combining physics-based simulation, supervised policy training, and standardized benchmark design.

Technical level: Advanced. The paper assumes familiarity with multimodal model architectures (visual encoders, vision-language mergers, autoregressive decoding), supervised fine-tuning objectives, reinforcement learning baselines, and bin-packing heuristics.

Scope: The paper introduces PackLab, an integrated framework consisting of a physics-grounded simulator and data generator (PackLab-Suite), a packing-specialized MLLM (PackLab-VLM), and a standardized multi-difficulty benchmark (PackLab-Bench) for closed-loop robotic bin packing.

What This Paper Is About

Robotic bin packing asks a robot to place objects into a container so that as much volume as possible is packed as compactly as possible. Each placement changes the free space available for every later object, so the task is a long-horizon sequential decision problem, not just a geometry puzzle. The paper asks whether a single multimodal large language model can directly choose objects and predict placements in a closed loop across many different object sets and container sizes, rather than relying on hand-crafted heuristics or reinforcement learning policies trained for narrow, predefined configurations.

Key Contributions

1. PackLab, an integrated framework for developing, training, and evaluating MLLM-based robotic bin packing policies, released as code, model, dataset, and benchmark at https://github.com/Correr-Zhou/PackLab.
2. PackLab-Suite, a physics-grounded simulation platform with gravity dynamics, rigid-body collision, and stability feedback, plus a layout-first data generation engine that produces the PackData-20K dataset of 20K training packing trajectories.
3. PackLab-VLM, a packing-specialized MLLM fine-tuned on PackData-20K that jointly selects an object and predicts its placement at each step, conditioned on container heightmaps, candidate-object attributes, and observation-action history.
4. PackLab-Bench, a standardized benchmark of 60 test cases evenly split across easy, medium, and hard difficulty, spanning 905 packing steps, with standardized metrics of Success Ratio, Compactness, and their product (Overall Score).

Main Findings

• Best average performance across all compared methods: PackLab-VLM reaches an average Overall Score of 0.660 on PackLab-Bench, beating the strongest heuristic baseline SDF-Pack by 0.059 and the strongest official RL checkpoint TAP-Net by 0.186.
• Improvement comes from both metrics: PackLab-VLM obtains the highest average Success Ratio (0.882) and highest average Compactness (0.718) among compared methods, meaning it both packs more object volume and arranges it more tightly, rather than winning on a single metric.
• Difficulty-dependent behavior: On easy cases it reaches 0.999 Success Ratio, 0.923 Compactness, and 0.922 Overall Score. On medium cases it scores 0.966 / 0.680 / 0.665, with a Success Ratio comparable to the best baselines but better Compactness. On hard cases it scores 0.680 / 0.552 / 0.394, where geometry-search heuristics such as SDF-Pack and DBLF retain advantages in some densely constrained cases.
• Task-specific training is essential: Comparing the untrained Qwen3.5-9B backbone with the trained model raises the average Overall Score from 0.209 to 0.660. The gain is consistent by difficulty: Easy 0.246 to 0.922, Medium 0.220 to 0.665, Hard 0.162 to 0.394.
• Visual geometry matters: Replacing the real heightmap with a black distractor heightmap at inference drops the Overall Score from 0.660 to 0.494.
• Sequential history matters: Using only the current observation scores 0.533 and removing assistant action history scores 0.589, both below the full-history setting of 0.660.
• More data helps: Overall Score rises from 0.578 with 25% of training data to 0.630 with 50% and 0.660 with 100%.
• Difficulty diversity in the data recipe helps: Easy-only training reaches 0.587, Easy+Medium reaches 0.604, and the mixed easy/medium/hard recipe reaches 0.660.
• Larger models pack more compactly: Overall Score increases from 0.617 for a trained 2B model to 0.645 for 4B and 0.660 for 9B, with all three achieving strong Success Ratio after training, indicating larger models mainly improve Compactness.
• RL policies struggle across heterogeneous configurations: When TAP-Net and IR-BPP are retrained under the same task distribution with an equivalent number of trial-and-error iterations, they achieve average Overall Scores of 0.405 and 0.345 respectively, both below their official-checkpoint numbers and far below PackLab-VLM.
• Physical feasibility demonstrated: On a real-world packing case with eight objects, PackLab-VLM packed all objects into a compact arrangement while the baseline packed only a subset of three objects and left much of the container unused. Figure 1 reports a real-world packing result with 1.00 Success Ratio and 0.72 Compactness.

Methodology in Plain English

The problem is framed as a loop: at each step the system knows the current container occupancy (encoded as a top-down heightmap) and a buffer of candidate objects (each described by identity and 3D dimensions). A policy outputs which object to pick, which of two horizontal orientations to use (0 or 90 degrees), and where to place it in the horizontal plane. The vertical coordinate is not predicted; it is determined by gravity, computed as the maximum heightmap value under the object's footprint at that pose.

To generate training data without exhaustively sampling every possible placement, the team uses a "layout-first" pipeline. They first sample a container size and object count, then partition the usable container space into stacked horizontal layers and non-overlapping regions that define a terminal packing layout. They then recover a forward packing trajectory by iteratively removing geometrically accessible objects and reversing the removal order. Each trajectory is replayed in simulation under gravity, collisions, and stability checks, and invalid trajectories are discarded. Valid ones become PackData-20K.

PackLab-VLM uses Qwen3.5-9B as its backbone. At each step, text describing the candidate objects and past actions is serialized into a chronological prompt with image placeholders, heightmaps (current and historical) are encoded by a visual encoder and merged into the language embedding space, and the model autoregressively decodes a structured text output that is parsed into an executable action. Training is supervised fine-tuning with a token-level negative log-likelihood loss on the ground-truth action strings from PackData-20K. Training used AdamW with a learning rate of 1e-5 and global batch size 8 over 5 epochs on an 8-GPU cluster, with FSDP, bf16 precision, and gradient checkpointing.

Evaluation uses documented metrics: Success Ratio is the volume fraction of objects whose post-placement geometric centers lie within the container; Compactness is the volume of successfully packed objects divided by the packing-envelope volume (container base area times the maximum occupied height in the final heightmap); Overall Score is their product. Test cases vary buffer size, number of layers, object count, and container dimensions, and PackLab-Bench is explicitly disjoint from PackData-20K.

Why This Matters

Impact on research. The paper argues that existing MLLM approaches to packing use the model only to infer object properties or semantic constraints and then hand the decision to a conventional planner. PackLab puts the MLLM itself in the closed-loop decision seat across heterogeneous object sets and container configurations, and supplies the missing infrastructure — physics-grounded simulation, scalable trajectory generation, packing-specific training, and a standardized benchmark — that prior work lacked as an integrated whole. It also provides direct evidence that reinforcement learning policies trained on predefined distributions generalize poorly to varied packing settings, which is a useful point of comparison for the field.

Real-world applications:

• Logistics and warehousing, where maximizing container space utilization directly reduces shipping volume and cost.
• Robotic manipulation in fulfillment centers, where a suction-equipped arm must decide object order and placement rather than follow canned scripts.
• E-commerce and parcel packing, where object sets and container sizes vary from order to order and a single policy must handle all of them.
• Palletizing and mixed-SKU container loading, where re-planning is needed when execution deviates from the plan.

Industry relevance. The paper's real-world platform — an AUBO robot arm with a suction cup, a RealSense RGB camera, a Photoneo depth camera, and a 30 cm x 30 cm x 10 cm container, with re-planning triggered when the observed container state deviates substantially from prediction — shows the pipeline is meant for deployment-style settings, not only simulation. The demonstrated comparison (all eight objects packed versus three for the baseline) is a concrete utilization argument for automated packing.

Future Directions

• Broader object geometries and physical settings. The conclusion states that future work will extend PackLab beyond the current object geometries and physical conditions.
• User-defined objectives and constraints. The authors want more flexible packing behaviors driven by user-specified goals rather than a single fixed objective.
• Scaling to larger MLLMs. The model-size ablation shows Overall Score rising from 0.617 (2B) to 0.645 (4B) to 0.660 (9B), but computational limits capped this study at 9B; larger MLLMs remain an explicit open direction.
• Closing the gap on hard cases. PackLab-VLM's weakest results are on hard cases (0.394 Overall Score, 0.680 Success Ratio), where heuristics such as SDF-Pack and DBLF retain advantages in densely constrained layouts — an unresolved weakness the paper openly reports.

Target Audience

Researchers and engineers working at the intersection of robotics, multimodal foundation models, and combinatorial packing are the primary audience, along with practitioners building automated warehousing and logistics systems. The paper is also relevant to readers interested in benchmarks and training-data-generation methodology for embodied MLLM policies, and to those evaluating whether general-purpose multimodal models can replace specialized planners in sequential decision tasks. Because it involves physics simulation, multimodal model internals, and supervised fine-tuning details, readers without machine learning background will find the methodology sections demanding.

Authors’ abstract

Robotic bin packing requires long-horizon sequential decision-making, as each object placement affects the available space for subsequent packing. Existing methods primarily rely on hand-crafted geometric heuristics that optimize predefined objectives or reinforcement learning policies learned through trial and error over predefined training configurations. Despite recent advances in multimodal large language models (MLLMs) for this task, their potential for closed-loop sequential decisions across heterogeneous packing configurations remains underexplored. To address this gap, we introduce PackLab, a comprehensive framework for developing, training, and evaluating MLLMs for closed-loop robotic bin packing. PackLab-Suite provides a physics-based simulation platform for scalable generation of diverse training packing trajectories and evaluation of their physical outcomes. PackLab-VLM is a packing-specialized MLLM that understands the evolving object and container states to jointly select objects and predict placements in a closed-loop manner. PackLab-Bench provides standardized packing scenarios at multiple difficulty levels for systematic evaluation. Extensive experiments demonstrate that, on average, PackLab-VLM outperforms conventional packing heuristics, traditional reinforcement learning methods, and general-purpose MLLMs across object sets and container configurations, highlighting the potential of MLLMs for long-horizon robotic packing. The code, model, dataset, and benchmark are available at https://github.com/Correr-Zhou/PackLab .