Real Garment Benchmark (RGBench): A Comprehensive Benchmark for Robotic Garment Manipulation featuring a High-Fidelity Scalable Simulator

AI summary

Overview

Research area: Robotics — specifically robotic manipulation of deformable objects (garments), physics-based cloth simulation, and sim-to-real transfer.

Technical level: Intermediate. Readers should be comfortable with basic concepts from robotics simulation, reinforcement learning environments, and 3D meshes, but the paper explains its simulation choices in accessible terms.

Scope: This paper introduces RGBench, a benchmark suite consisting of a 6,000+ model garment dataset with real measured physics, a new GPU-accelerated cloth simulator called GarmentDynamics, and a standardized protocol for measuring how far simulated garment behavior drifts from reality during grasping, flinging, and folding.

What This Paper Is About

Robots that fold laundry or handle clothing need to be trained somewhere, and training in simulation is far cheaper and safer than training in the physical world. The problem is that cloth is exceptionally hard to simulate: it is nearly infinite-dimensional, highly nonlinear, and constantly folding and sliding against itself. Existing robotics simulators either sacrifice physical accuracy for speed (producing stretched, exploding, or self-penetrating cloth) or are too slow for large-scale reinforcement learning.

This paper builds a benchmark to measure that accuracy problem honestly, and a new simulator designed to solve it. The authors collect real garments, measure their actual fabric properties with laboratory instruments, record real dual-arm robots manipulating them, and then score how closely each simulator's output matches the recorded reality.

Key Contributions

1. A large, physically grounded garment dataset. Over 6,000 garment mesh models — 4,000+ self-collected industry-grade assets with real seam lines, elastic edges, and measured material parameters, plus 2,000+ meshes from ClothesNet — spanning 10 fabric types from stiff leather to highly deformable silk.

2. GarmentDynamics, a new cloth simulator. A GPU-accelerated engine built on a continuum finite-element model with anisotropic stretching and bending, quadratic bending resistance for realistic wrinkling, hybrid potential-based collision handling with untangling, and implicit time integration accelerated by a multilevel preconditioner.

3. A benchmark protocol for quantifying the sim-to-real gap. Three manipulation primitives (Grasp, Fling, Fold) performed by real dual-arm robots and captured as ground-truth point clouds, evaluated with bidirectional Chamfer and Hausdorff distances.

4. Empirical demonstration that the new simulator is both more accurate and faster. Head-to-head comparisons against MuJoCo, PyBullet, and NVIDIA Isaac Sim across garment types, actions, and mesh resolutions.

Main Findings

• Large accuracy gains on an independent cloth benchmark. On the external BCM benchmark using simple chequered cloth, GarmentDynamics beat the previous state of the art (FLEX) by 46% in Chamfer distance and 45% in Hausdorff distance on quasi-static deformation. It also won on dynamic tasks despite being fed raw, noisy input trajectories without preprocessing.

• Substantial speed and scalability advantages. GarmentDynamics ran 3.0x to 7.0x faster than Isaac Sim (the strongest baseline), roughly 65x faster than MuJoCo at 20k mesh vertices, and 16x faster than PyBullet at 40k vertices. MuJoCo failed outright at 40k vertices. Initialization overhead dropped by over 90% compared to all baselines.

• Smaller sim-to-real gaps across the board. Averaged over tasks, the simulator cut simulation error by over 20%, and by as much as 77% for topologically complex garments. On grasp and fold it reduced real-to-sim Chamfer error by up to 35% and 58% respectively; on the high-speed fling task the gap widened for everyone, but GarmentDynamics still improved by over 20%.

• Baselines fail in qualitatively different ways. Isaac Sim became unstable when grippers contacted cloth, producing exaggerated twisting and inflation. PyBullet frequently failed to grasp at all, rendered cloth as implausibly thin, and suffered penetration where grippers passed through the fabric. GarmentDynamics remained stable throughout.

• Errors grow with fabric thickness and garment length. The authors attribute this to complex surface friction under spatial constraints, and identify it as a remaining source of sim-to-real error.

• Robot mode is harder than pseudo mode. When full arm dynamics and gripper joint angles are simulated (rather than directly scripting cloth vertex motion), inaccuracies compound into slippage, weak grip, and erratic deformation, magnifying differences between simulators.

Methodology in Plain English

The authors work in three parallel tracks: measure reality, build a better simulator, and score everyone against reality.

Measuring reality. Rather than guessing fabric parameters, they physically test them. Tensile testing machines stretch fabric samples to measure stretch stiffness; bending testers measure how stiffly each fabric resists folding. Area density and thickness come from precision scales and gauges. All of this follows ASTM textile standards. Geometrically, they import production-grade DXF pattern files (the actual files used to cut and sew real garments) to build 1:1 scale models, including seam lines and elastic or folding edges. The result is a digital twin that mirrors both the shape and the mechanical behavior of a real garment.

Recording manipulation ground truth. A dual-arm robot setup — AGILEX Piper or JAKA K1 arms with PGC-50-35 grippers — performs three canonical actions on real garments. An Intel RealSense L515 camera records the whole process as an RGB point cloud stream, and Grounded-SAM segments the garment out of the scene. That point cloud is the ground truth against which everything is compared.

Aligning simulation to reality. Getting a fair comparison requires careful bookkeeping. The garment's starting pose is standardized using a physical outline template. The camera-to-robot transform is recovered through hand-eye calibration or iterative closest point registration. Finally, a fixed optimal time delay is applied to the data streams to compensate for system latency, so that frame t in simulation corresponds to frame t in reality.

Scoring. Two metrics are used. Chamfer distance measures the average distance from each point in one cloud to the nearest point in the other. Hausdorff distance measures the worst-case distance, which catches catastrophic failures like a sleeve ballooning outward. Crucially, the authors compute these in both directions: simulated-to-real and real-to-simulated. The reverse direction matters because unstable simulators can "explode," inflating the simulated point cloud and artificially lowering sim-to-real error. Real-to-sim metrics are less vulnerable to that gaming, so the authors weight them more heavily.

The simulator. GarmentDynamics treats cloth as a continuous elastic surface discretized into triangles. Stretching and bending are modeled separately along the fabric's warp, weft, and diagonal directions, with bending resistance growing quadratically with curvature so wrinkles form realistically. Real measured fabric properties feed directly into the solver. Collisions combine potential-based contact forces with an untangling mechanism to prevent the cloth from passing through itself or the robot. Each time step is solved as an energy minimization using implicit Euler integration, run through a small number of inexact Newton iterations, each with a fixed number of preconditioned conjugate gradient steps. The preconditioner divides the mesh into many small non-overlapping blocks across multiple resolution levels, inverting each block once with Gauss-Jordan elimination and storing the results on the GPU. Collision detection uses a GPU bounding volume hierarchy leveraging RTX ray intersection hardware, handling millions of broad-phase tests per second.

Evaluation targets. The benchmark is engine-agnostic. Any simulator that supports rigid and deformable bodies, robust collisions, variable timesteps, and vertex state queries can be plugged in. MuJoCo, PyBullet, and Isaac Sim are included as baselines.

Why This Matters

Robotic garment manipulation has been held back by a feedback loop: simulators are too inaccurate to train good policies, so policies fail in reality, so there is little incentive to invest in better simulators. This paper breaks that loop by providing both a rigorous measuring stick and a simulator that measures well on it. The bidirectional metrics are a notable methodological point — they prevent the common practice of a simulator looking good simply because it produces a huge, diffuse point cloud.

Real-world applications:

• Household assistance robots that pick up, sort, and fold laundry, one of the most-requested but least-solved domestic tasks.
• Apparel and textile manufacturing, where digital twins of garments can validate automated handling, sewing, and quality inspection before any physical line is built.
• Warehouse and logistics automation for soft goods — sorting returned clothing, bagging, and packing items that cannot be handled by rigid-body grippers.
• Virtual garment design and try-on, since the same physically accurate fabric parameters and simulator that drive robot training also drive realistic cloth rendering and drape prediction.

Industry relevance. The dataset comes in part from Style3D Research and includes production-ready industrial assets, meaning the models can be sent directly to physical fabrication. That positions this work as much as a manufacturing and digital-fashion tool as a robotics benchmark.

Future Directions

• Closing the residual sim-to-real gap on dynamic tasks. Fling remains the hardest case for every simulator, because high-speed aerial motion involves aerodynamic drag and inertial effects that are modeled only approximately.

• Better initial state alignment. Perfectly matching the garment's starting wrinkle configuration is still not achievable. Improved sensor precision, more accurate delay compensation, and direct measurement of friction and damping coefficients are the named levers.

• Learning manipulation policies from the benchmark. The authors explicitly plan to use RGBench's diversity to train a generalized action policy, rather than only using it for evaluation.

• Extending the simulator's scope. Handling wet fabric is called out as a near-term target, along with generalizing GarmentDynamics to deformable objects beyond cloth, such as ropes, bags, and soft containers.

Target Audience

Robotics researchers working on deformable object manipulation and sim-to-real transfer will find the most direct value, particularly those who need a credible evaluation harness rather than an ad hoc comparison. Cloth simulation and computer graphics researchers will be interested in the GPU solver design, the multilevel additive Schwarz preconditioner, and the quadratic bending model. Reinforcement learning practitioners benefit from the simulator's speed and scalability, which makes large-scale parallel training on high-resolution meshes practical. Finally, engineers in apparel manufacturing, digital fashion, and warehouse automation will find the production-grade asset pipeline and measured fabric parameter tables immediately reusable.

Authors’ abstract

While there has been significant progress to use simulated data to learn robotic manipulation of rigid objects, applying its success to deformable objects has been hindered by the lack of both deformable object models and realistic non-rigid body simulators. In this paper, we present Real Garment Benchmark (RGBench), a comprehensive benchmark for robotic manipulation of garments. It features a diverse set of over 6000 garment mesh models, a new high-performance simulator, and a comprehensive protocol to evaluate garment simulation quality with carefully measured real garment dynamics. Our experiments demonstrate that our simulator outperforms currently available cloth simulators by a large margin, reducing simulation error by 20% while maintaining a speed of 3 times faster. We will publicly release RGBench to accelerate future research in robotic garment manipulation. Website: https://rgbench.github.io/