Phys-Liquid: A Physics-Informed Dataset for Estimating 3D Geometry and Volume of Transparent Deformable Liquids

AI summary

Overview

Research area: Computer vision and physics-informed simulation, specifically 3D geometry and volume estimation of transparent, deformable liquids for robotics. Technical level: Advanced. The paper assumes familiarity with fluid simulation, multi-view diffusion models, neural 3D reconstruction (triplanes), and mesh-based evaluation metrics. Scope: The paper introduces a large physics-based simulation dataset of transparent liquids deforming under container rotation, plus a four-stage reconstruction pipeline that turns a single RGB image into a real-world-scaled 3D liquid mesh.

What This Paper Is About

Robots that handle liquids in laboratories — pouring, pipetting, mixing — need to know how much liquid is in a container and what shape it currently takes. This is very hard because the containers and liquids are transparent, and moving a container sloshes the liquid into complex, constantly changing shapes. The authors build a large simulated dataset (Phys-Liquid) where liquids follow real fluid physics under systematic container rotations, and then show that a model trained on this data can reconstruct liquid shape and volume accurately from images, including on real-world photos it has never seen.

Key Contributions

1. Phys-Liquid dataset. A physics-informed simulation dataset of 97,200 images and 8,100 annotated 3D liquid meshes, covering 20 transparent laboratory containers, 5 lab scenes, 8 lighting conditions, 5 liquid colors, and 6 distinct rotation modes over 81 time frames each — capturing liquid deformation as a 4D (space plus time) phenomenon.
2. Navier-Stokes–driven generation workflow. A Blender/Mantaflow pipeline that solves the fluid momentum equation and incompressibility condition, then renders six orthographic views per frame with ground-truth meshes in real-world dimensions, validated against real captured liquid behavior.
3. A four-stage reconstruction and estimation pipeline. Segmentation, multi-view mask generation via a fine-tuned diffusion model, triplane-based 3D mesh reconstruction, and a learned mesh-scaling module that maps reconstructions to real-world units and thus to volume.
4. Benchmarking and generalization evidence. Quantitative comparisons against a liquid-specific baseline and two general single-image 3D reconstruction methods, plus transfer tests on a real-world liquid dataset (DTLD).

Main Findings

• Physical realism validated against reality: For ten representative containers, the angle between the liquid's top surface and container walls in simulation closely matched the same angles measured in real physical experiments under comparable rotations.
• Strong improvement over the liquid-specific baseline: Against Eppel et al., RMSE dropped from 0.0842 to 0.0192, Chamfer Distance from 0.0412 to 0.0079, Volume IoU rose from 0.1216 to 0.4748, and F-Score from 30.91% to 75.38%.
• Outperforms general 3D reconstruction models: Versus InstantMesh and TripoSR, the pipeline achieved Chamfer Distance 0.0085 (vs. 0.0189 and 0.0275), Volume IoU 0.6236 (vs. 0.2794 and 0.2275), and F-Score 78.57% (vs. 46.18% and 38.06%).
• Fine-tuning the diffusion model matters a lot: Multi-view mask generation accuracy improved from 74.38% to 90.05% IoU after fine-tuning on Phys-Liquid, and reconstruction Volume IoU nearly doubled.
• Generalizes to real photographs: Trained only on simulation, the method reached RMSE 0.0266, Chamfer Distance 0.0172, Volume IoU 0.3861, and F-Score 62.43% on the real-world DTLD dataset.
• Temporally stable reconstructions: Across 100 sequences, RMSE variance was 0.00038038 (standard deviation 0.00643858), indicating the pipeline tracks liquid deformation consistently frame to frame.
• More camera views do not automatically help: Single-view input achieved RMSE 0.0120, two-view 0.0138, and three-view 0.0129 — comparable or slightly worse, suggesting the fine-tuned multi-view generation already supplies sufficient geometric cues.
• Ablation highlights where accuracy comes from: Replacing the mesh reconstruction and mesh scaling modules with ground-truth simulation data produced the largest gains, identifying them as the performance-critical stages.
• Ancillary model performance: The SAM2/YOLO-world segmentation stage averaged 0.8694 IoU across 20 objects; the ViT-based scaling model reached 1.75% MAPE on training and 6.26% on test data, with L2 loss converging most stably.

Methodology in Plain English

The authors start by building a virtual laboratory inside Blender. They place transparent containers — flasks, bottles, tubes, cones, cubes, cylinders — in five different lab scenes with eight different indoor lighting setups, fill them with colored or colorless liquid, and then rotate each container along one or more axes from 0 to 80 degrees over 81 time steps.

The liquid itself is not animated by hand. It is simulated as a particle system governed by the Navier-Stokes equations, the same physics used to model real fluids: pressure pushes the fluid, viscosity resists its internal sliding, gravity pulls it down, and the fluid is forced to be incompressible so it does not artificially compress or expand. Each frame produces a mesh describing the liquid's surface, saved with accurate real-world dimensions.

Everything is rendered from six virtual orthographic cameras (top, bottom, front, back, left, right), producing aligned multi-view images, liquid masks, and meshes. The authors check the dataset's realism by filming real liquids in real containers under similar rotations and comparing the surface angles.

To show the dataset is useful, they build a four-step pipeline. First, SAM2 with YOLO-world guidance segments the liquid region from a single RGB image. Second, a multi-view diffusion model (CRM) fine-tuned on Phys-Liquid predicts what the liquid would look like from all six camera views, using canonical coordinate maps to keep the views spatially consistent. Third, a triplane-based reconstruction network aggregates those masks into a 3D mesh. Fourth, a multi-view Vision Transformer regresses a single scaling factor that converts the mesh into real-world dimensions, which also yields the liquid's volume. The scaling factor is computed at training time by comparing reconstructed axis lengths against the simulation's ground-truth dimensions.

Why This Matters

Impact on research. Existing transparent-object datasets (ClearGrasp, ClearPose, TODD) ignore what is inside the container, and liquid datasets (DTLD, the Gautham et al. dataset) capture only static liquid states. Phys-Liquid is the first large-scale dataset to model dynamic liquid deformation driven by container motion, moving the problem from 3D perception to 4D spatiotemporal perception. It also provides a physics-grounded benchmark where ground-truth meshes and volumes are exactly known, which is nearly impossible to obtain with real photography of transparent fluids.

Real-world applications:

• Automated laboratory science: Robots performing dispensing, aspiration, and mixing in chemistry, biology, and pharmaceutical research need to know how much reagent is in a vessel and how it shifts as the vessel is tilted.
• Pharmaceutical and clinical liquid handling: Accurate dosing and transfer in drug preparation and diagnostics, where over- or under-measurement has safety consequences.
• Food and beverage quality control: Estimating fill levels and volumes of transparent liquids in bottles and vats on production lines.
• Household and service robotics: Robots that pour drinks, refill containers, or clean up spills must perceive transparent liquids they cannot see with depth sensors alone.
• Industrial chemical processing: Monitoring and manipulating transparent fluids in containers where lidar and structured-light depth sensing fail due to refraction.

Industry relevance. Laboratory automation is a fast-growing market, and the bottleneck for fully autonomous labs is often perception rather than actuation. Because Phys-Liquid is simulation-based, it can be scaled and extended to new container shapes, lighting, and fluids without expensive physical data collection — attractive to robotics and lab-automation companies. The paper also notes the Blender pipeline can render additional modalities such as surface normals and refractive flow, which are essentially unobtainable from real transparent-liquid photography.

Future Directions

1. Closing the simulation-to-reality gap further. Generalization to real data (62.43% F-Score on DTLD) is meaningful but clearly below in-simulation performance (75.38%), so domain adaptation, real-data fine-tuning, or sim-to-real transfer techniques remain open.
2. Scaling dataset diversity. The dataset covers 20 containers, 5 colors, 5 scenes, and 6 rotation modes. Extending to more complex container geometries, opaque or partially reflective glass, opaque liquids, multiple liquids in one vessel, and pouring or mixing events would broaden applicability.
3. Improving the accuracy-critical stages. The ablation shows mesh reconstruction and mesh scaling dominate error. Better triplane reconstruction or direct metric-scale prediction without a separate scaling module could yield substantial gains.
4. Exploiting unrendered modalities. The authors explicitly point to surface normals and refractive flow as available but unused, raising the question of whether multi-modal supervision improves physics-aware visual reasoning and 4D fluid representation learning.
5. Tighter integration with manipulation. Results could be closed into a robot control loop where volume estimates inform pouring and aspiration decisions in real time, testing whether temporal consistency holds under closed-loop physical interaction.

Target Audience

This paper is most valuable to computer vision researchers working on transparent object perception, physics-informed simulation, or single-image 3D reconstruction; robotics researchers building laboratory automation and liquid-handling systems; and simulation engineers who need benchmarks with exact ground-truth geometry and volume. Practitioners in lab automation, pharma robotics, and embodied AI who need perception for transparent fluids will find the dataset and pipeline directly reusable, while students entering 4D fluid perception will find a well-structured benchmark and baseline.

Authors’ abstract

Estimating the geometric and volumetric properties of transparent deformable liquids is challenging due to optical complexities and dynamic surface deformations induced by container movements. Autonomous robots performing precise liquid manipulation tasks, such as dispensing, aspiration, and mixing, must handle containers in ways that inevitably induce these deformations, complicating accurate liquid state assessment. Current datasets lack comprehensive physics-informed simulation data representing realistic liquid behaviors under diverse dynamic scenarios. To bridge this gap, we introduce Phys-Liquid, a physics-informed dataset comprising 97,200 simulation images and corresponding 3D meshes, capturing liquid dynamics across multiple laboratory scenes, lighting conditions, liquid colors, and container rotations. To validate the realism and effectiveness of Phys-Liquid, we propose a four-stage reconstruction and estimation pipeline involving liquid segmentation, multi-view mask generation, 3D mesh reconstruction, and real-world scaling. Experimental results demonstrate improved accuracy and consistency in reconstructing liquid geometry and volume, outperforming existing benchmarks. The dataset and associated validation methods facilitate future advancements in transparent liquid perception tasks. The dataset and code are available at https://dualtransparency.github.io/Phys-Liquid/.