RoboCloth: A Large-Scale Real Cloth Material Dataset for Neural Reflectance Reconstruction

AI summary

Overview

Research area: Computer vision and physically based rendering — specifically inverse rendering and measured material (SVBRDF/BRDF) capture for neural reflectance reconstruction.

Technical level: Intermediate. The core ideas are explained without heavy math, but the work assumes familiarity with BRDFs, path tracing, and neural material representations.

Scope (one sentence): The paper introduces RoboCloth, a dataset of 500 real cloth materials captured with a two-robot-arm rig under flexible camera and light positioning, and shows that a neural BRDF decoder trained on it produces a stronger, more transferable cloth material prior than decoders trained on existing measured reflectance datasets.

What This Paper Is About

Faithfully rendering real cloth is hard because fabrics have spatially varying reflectance driven by weave structure, fiber microstructure, roughness, and anisotropic scattering, and these properties change nonlinearly with viewing and lighting direction. Recent neural approaches can learn a shared decoder that maps compact per-point latent codes to reflectance, but such a decoder needs training captures that are simultaneously real, diverse, and densely sampled over both the view and light hemispheres — a combination the authors argue no existing dataset provides. RoboCloth fills that gap with 500 real fabrics captured by a calibrated robotic system, and the paper demonstrates the dataset's value by training and evaluating a two-stage neural reconstruction pipeline on it.

Key Contributions

1. A real cloth reflectance dataset: 500 fabric materials, each with approximately 580 high-dynamic-range (HDR) images, captured under flexible view-light coverage, released with pre-processed data, train/test splits, and loaders for neural BRDF training under CC BY 4.0 on the Hugging Face Hub.

2. A calibrated robotic acquisition pipeline: an end-to-end capture and processing system producing aligned HDR observations, camera and light metadata, COLMAP-refined camera poses, and sparse surface reconstructions, with per-material structured observation tensors.

3. A neural decoder method for cloth material reconstruction: a two-stage pipeline in which Stage 1 trains a shared MLP decoder across the corpus and Stage 2 freezes that decoder and fits a dense 2048 × 2048 latent texture per held-out material, evaluated with path-traced rendering.

4. Controlled comparative evidence: a fixed-protocol comparison of Stage-1 training sources (RoboCloth vs. Bonn/UBOFAB19 vs. MERL) plus an analytic Disney principled BRDF fitting baseline, together with a dataset-scale ablation.

Main Findings

• RoboCloth is the only dataset with all five properties compared: The paper's comparison table checks cloth specificity, real-world measurements, spatially varying appearance, multiple views, and flexible coverage. RoboCloth (500 materials) is the only entry marked as satisfying all five; the closest cloth-specific alternative is UBOFAB19 with 378 fabrics, which lacks flexible coverage because its view and light directions follow a preset fixed hardware layout.

• Strongest cross-dataset transfer to UBO2014: On 12 randomly selected UBO2014 materials, the RoboCloth-trained decoder averages 36.76 dB PSNR, versus 33.05 dB for the PBR baseline, 32.27 dB for Bonn, and 30.64 dB for MERL. It achieves the highest PSNR on every one of the 12 materials, leading the next-best baseline (PBR) by 3.7 dB and the strongest competing decoder (Bonn) by 4.5 dB on average — despite never seeing a UBO2014 sample during Stage 1.

• Best in-domain reconstruction: On 5 held-out materials from RoboCloth's own test set, the RoboCloth decoder averages 29.63 dB, ahead of Bonn (27.32 dB), PBR (26.99 dB), and MERL (25.84 dB), a margin of +2.31 dB over the best competitor.

• Asymmetric cross-domain transfer: On 5 held-out Bonn materials, the Bonn-native decoder wins (24.47 dB), with RoboCloth second (23.43 dB), then PBR (22.99 dB) and MERL (21.77 dB). The gap is asymmetric: RoboCloth trails the native Bonn model by only about 1.0 dB on Bonn's distribution, whereas Bonn trails RoboCloth by about 2.3 dB on the RoboCloth distribution — more than twice the gap. The ranking Bonn > RoboCloth > PBR > MERL holds on all five materials.

• Better qualitative appearance after relighting: When held-out UBO2014 materials are relit in a common scene with identical mesh, UV tiling, lighting, and camera, the MERL-trained decoder loses weave structure and drifts in color, the Bonn-trained decoder shifts overall hue, and the PBR baseline misses fine sheen; only the RoboCloth-trained decoder matches ground-truth color while preserving both the specular highlight at folds and fuzzy weave detail.

• Dataset scale helps: Retraining the Stage-1 decoder on random 100 / 300 / 500-material subsets (80 / 240 / 400 training materials) and re-running the UBO2014 transfer shows PSNR rising consistently with more data and beginning to saturate at the full corpus. The conclusion separately notes that transfer PSNR is still improving at the full corpus, i.e., learning had not saturated at 500 materials.

• Grazing-angle regularization matters: Since captures do not cover near-grazing incidence, the paper compares its decay loss against "ZeroExact" (pinning the raw BRDF to zero exactly at 90°), "NearZeroBRDF" (pinning the BRDF to zero over cos θᵢ ∈ [0.005, 0.03]), and no grazing supervision. Only the proposed loss yields reflectance lobes that taper smoothly to zero at grazing; the alternatives either leave the neighborhood free or impose physically wrong constraints on the raw BRDF rather than on the product f_r cos θᵢ.

• Reported evaluation is a subsample: Because fitting a dense latent grid can take up to one day of training on an RTX 6000 Ada, evaluation uses randomly picked subsamples: 12 materials from UBO2014, 5 from the RoboCloth test set, and 5 from the Bonn test set.

Methodology in Plain English

Capture hardware. Two UFACTORY X-ARM 6 robot arms hold a camera and a calibrated LED area light inside a matte-black enclosure. A ferromagnetic turntable adds a third controlled degree of freedom: rotating the sample re-orients it beneath the arm pair, so each arm configuration is reused at several sample-relative azimuths, effectively tripling the variety of camera-light configurations without moving the arms. Cloth is flattened with corner magnets, and a ChArUco fiducial beside the sample supports calibration and per-session drift checks. Each material takes roughly 20 minutes in one automated session, producing about 580 raw HDR frames with robot-logged camera pose, light pose, and turntable angle.

Geometry. A one-time hand-eye calibration gives the camera-to-flange transform, and a separate procedure fits the turntable rotation axis in the robot base frame. Combining these with the arm pose read from the robot controller places every camera in a common metric "0°-turntable" world frame. COLMAP is run on low-dynamic-range copies of the frames to get a sparse point cloud, which is then aligned to the metric frame in closed form via Umeyama using matched camera centers; frames with alignment residual above 16 mm are discarded, and the cloud is cropped to the measured sample footprint and depth-filtered with the Tukey IQR rule.

Radiometry. Raw Bayer mosaics are de-mosaiced into linear-light HDR RGB with no gamma or tone mapping. The authors calibrate a per-exposure camera scalar converting pixel values to physical radiance, and measure a 90-entry polar table for the LED's directional falloff from a near-Lambertian reference card. A 3 × 3 color correction matrix is estimated from a ColorChecker capture and released, but images ship as white-balanced sensor RGB without the matrix applied so users keep colorimetric control.

Observation tensor. Instead of reloading frames during training, each retained sparse point is projected into every retained HDR frame and bilinearly sampled, producing a tensor of shape K_m × V_m × 3 with zeros marking missing observations. Each material yields on average about 1.6 × 10⁸ valid RGB observations, replacing roughly 6–7 GB of per-material HDR frames with a single approximately 1.0 GB indexed tensor.

Stage 1 — learning a prior. A single MLP decoder is shared across all training materials. Each sparse surface point owns an optimizable latent code (24-dimensional), normal, and tangent, updated by the same optimizer step as the decoder weights. A shared coordinate-transform operator builds a tangent-bitangent-normal frame, rotates incident and outgoing directions into it, encodes local directions and normals with degree-3 real spherical harmonics, and concatenates them with the latent. Training draws sparse (material, point, view) triplets across many materials per iteration. Two losses are used: a log-relative reconstruction loss that weights dark diffuse regions and bright specular highlights comparably across HDR's four-to-five orders of magnitude of radiance, and a decay loss that penalizes f_r cos θᵢ in the under-constrained near-grazing region using a stop-gradient anchor, so the BRDF itself need not vanish.

Renderer. Both stages use the same differentiable single-bounce path tracer (from FIPT), with four Monte Carlo samples per surface point. Indirect illumination is omitted because the sample sits in a matte-black enclosure with no secondary sources in the camera's dynamic range. Only the BRDF is predicted by the network; cosine, geometry, emitter radiance, and inverse-square falloff are computed analytically from calibrated quantities, so the frozen decoder ports directly to standard physically based renderers.

Stage 2 — dense fitting. For one target material the decoder is frozen and a 2048 × 2048 latent-geometry grid is optimized, with progressive decaying Gaussian blur acting as a coarse-to-fine regularizer. A parallax-aware texture lookup, following NeuMIP, uses a small geometry MLP to predict view- and light-dependent texture-space displacement so that mesostructure height does not cause incorrect sampling. For sparse baseline datasets without a shared surface parametrization, Stage 2 reverts to Stage 1's per-point representation.

Comparison protocol. The decoder architecture, renderer, and Stage 2 procedure are held fixed across all comparisons, so differences are attributable to the Stage-1 training source: RoboCloth, Bonn (UBOFAB19), or MERL. An analytic Disney principled BRDF baseline is fit with the same renderer and optimizer, using anisotropic GGX specular, Burley diffuse, and sheen lobes plus a normal map at the same 2048 × 2048 resolution.

Why This Matters

Impact on research. The paper reframes material-dataset design around a downstream capability — how well a prior trained on a dataset transfers to unseen materials and to other capture systems — rather than around raw image counts. Its cross-dataset results, including the asymmetric transfer between RoboCloth and Bonn, and its dataset-scale ablation give the community a concrete measurement of how much flexible view-light coverage and corpus size contribute to generalization. The released observation tensors and splits also lower the barrier to training neural BRDF decoders without reprocessing terabytes of raw captures.

Real-world applications:

• Virtual try-on and digital fashion, where garments must be rendered convincingly on moving bodies under changing light.
• Visual effects and animation, where fabrics need shader-ready material maps that drop directly into physically based pipelines.
• Interior and product rendering, illustrated by the paper's own path-traced scene with fabric on a sofa, curtains, pillows, and a carpet.
• E-commerce and configurator visualization, where a scanned swatch could be turned into a relightable material rather than a flat photograph.

Industry relevance. The two-stage scheme is operationally attractive: because the decoder is frozen at deployment and only a dense latent texture is fit per new material, a production pipeline can amortize one shared prior across many fabrics. The fact that the learned decoder transfers to UBO2014 without modification, and that cosine and geometric terms are computed analytically, means the resulting representation plugs into existing physically based renderers rather than requiring a bespoke neural renderer.

Future Directions

• Scaling capture throughput. The authors note each material takes about 20 minutes plus human mounting, that they have roughly 5000 more cloth samples in practice, and that throughput is bounded by operator effort; adding more robot arms or switching capture modality are proposed remedies.

• Active capture. With a prior available from previous scans, an active-capture agent could choose the next-best view-light configuration per sample, replacing the roughly 580 uniformly scheduled frames with a small adaptive set.

• Beyond planar textiles. The dataset is explicitly cloth-specific and tailored to near-planar samples; extending the protocol and evaluation to three-dimensional objects and non-textile materials is stated as a limitation rather than a solved problem.

• Scene-level integration. The paper leaves open how to integrate the recovered material priors into scene-level 3D Gaussian Splatting reconstruction.

• Further scale. The ablation suggests transfer PSNR was still improving at 500 materials, so the corpus size was a practical stopping point rather than a measured optimum.

Target Audience

Researchers and practitioners in inverse rendering, material capture, and physically based rendering who need large, calibrated, cloth-specific reflectance data or a pretrained neural BRDF prior. It is also relevant to graphics engineers building relightable material pipelines for fashion, VFX, and product visualization, and to dataset builders interested in robotic acquisition design and in evaluation protocols that test cross-dataset prior transfer rather than within-dataset reconstruction alone. Readers seeking per-frame image collections rather than tensorized, shader-oriented training records should look elsewhere, and those working on non-planar or non-textile materials are outside the paper's stated scope.

Authors’ abstract

We present RoboCloth, a dataset of 500 real cloth materials captured under flexible, calibrated camera and lighting configurations, where both are robotically controlled during acquisition. Each material is imaged in approximately 580 high-dynamic range frames spanning a wide view-light hemisphere; camera poses are refined via structure-from-motion and aligned to the robot frame, yielding per-material sparse surface geometry and structured per-point reflectance observations ready for neural material training. We demonstrate the value of this data through a two-stage neural reconstruction pipeline: Stage 1 trains a shared MLP decoder on our corpus to learn cloth material priors; Stage 2 then freezes the decoder and fits a dense 2048x2048 latent texture to each target material. Evaluating on held-out materials from our own dataset and on the independent UBO2014 benchmark, we show that a decoder trained on our captures outperforms decoders trained on existing measured reflectance datasets and analytic PBR fitting, transferring to unseen acquisition systems without modification. The dataset, calibration files, observation tensors, and train/test splits are released under CC BY 4.0 on the Hugging Face Hub.