Physically Guided Visual Mass Estimation from a Single RGB Image

AI summary

Overview

Research area: Computer vision and physically grounded robot perception — specifically, estimating how heavy an object is from a single RGB photograph.

Technical level: Intermediate. The paper uses standard deep learning building blocks (depth estimation, point clouds, CLIP, gated fusion), but the framing is physical rather than purely architectural, so readers need some familiarity with monocular depth estimation and vision-language models.

Scope: The paper proposes and evaluates a single-image mass estimation framework that decomposes mass into a volume-related latent factor guided by monocular depth geometry and a density-related latent factor guided by VLM-derived material semantics, trained with mass supervision only.

What This Paper Is About

Estimating an object's mass from a picture is hard because mass depends on two things that pixels do not directly expose: how big the object is in three dimensions (volume) and what it is made of (density). Because many combinations of volume and density produce the same appearance, the problem is ill-posed. The authors' goal is to make this ambiguous problem more tractable by deliberately wiring each physical factor to the visual evidence best suited to inform it — depth-derived geometry for volume, language-model material descriptions for density — instead of letting one appearance-driven network untangle everything implicitly.

Key Contributions

1. A physically structured decomposition of mass. The framework grounds prediction in the relation m = V·ρ, producing separate volume-related and density-related latent factors from two dedicated regression heads, trained under mass-only supervision.

2. Explicit alignment of visual cues to physical factors. Rather than treating geometry, semantics, and appearance as interchangeable inputs, the method ties monocular-depth-derived point clouds to volume-oriented inference and VLM material descriptions to density-oriented reasoning, with appearance as complementary context.

3. An instance-adaptive gated fusion mechanism. Scalar per-modality weights let the model decide per object whether geometry, semantics, or appearance should dominate; the paper compares this against concatenation and self-attention fusion.

4. Use of coarse material semantics instead of numeric density prediction. The VLM (Qwen2.5-VL) answers only "What is this object primarily made of?" and the resulting text is embedded as a soft prior for density, avoiding the numeric brittleness of prompting models for physical values.

Main Findings

• Best single-view results on image2mass. Gated fusion achieves ALDE 0.519, APE 0.665, MnRE 0.647, and Q 0.722 on the 1,435-instance test set, and is best across all four metrics. For comparison, the image2mass baseline scores ALDE 0.655, APE 1.066, MnRE 0.579, Q 0.614, and the RGB baseline scores ALDE 0.843, APE 1.459, MnRE 0.517, Q 0.514.

• Geometry dominates the learned weighting. The gated fusion learns modality weights of image 0.13, geometry 0.49, text 0.36, confirming that geometry and material cues carry most of the signal.

• Better than the geometry-augmented baseline. The RGB+Depth method (Cardoso and Moreno) reaches ALDE 0.567, APE 4.886, MnRE 0.615, Q 0.686 on image2mass, versus 0.519 / 0.665 / 0.647 / 0.722 for gated fusion, and 0.528 / 0.681 / 0.639 / 0.717 for plain concatenation.

• VLMs are poor numeric mass estimators. On image2mass, LLaVA (direct) reaches ALDE 1.944, APE 63.167, MnRE 0.302, Q 0.245; Qwen-VL (direct) reaches ALDE 2.133, APE 46.618, MnRE 0.402, Q 0.398; reasoning prompts improve stability but remain far behind learned regressors (Qwen-VL reasoning: ALDE 1.074, APE 39.550, MnRE 0.496, Q 0.478).

• Competitive with a multi-view system on ABO-500. The single-view gated model gets ADE 8.321, ALDE 0.998, APE 0.980, MnRE 0.472. The multi-view NeRF2Physics pipeline (30 RGB views) gets ADE 8.730, ALDE 0.771, APE 1.061, MnRE 0.552 — so the proposed method performs better on ADE and APE but trails on ALDE and MnRE.

• Self-attention wins on some ABO-500 metrics. On ABO-500, the self-attention variant achieves ADE 7.530 and APE 0.900, while gated fusion achieves the best ALDE (0.998) and MnRE (0.472). Concatenation reaches ADE 10.161, ALDE 1.034, APE 1.639, MnRE 0.407.

• Generalization holds across novel categories. Only 24.7% of test categories are seen in training, yet seen categories score ALDE 0.485, APE 0.543, MnRE 0.656, Q 0.747 over 355 instances while unseen categories score ALDE 0.531, APE 0.708, MnRE 0.634, Q 0.712 over 1,080 instances — a moderate drop rather than a collapse.

• Learnable material semantics beat numeric and rule-based density. Direct VLM numeric density prediction (*NeRF2Physics with a CLIP encoder) gives ALDE 1.319, APE 6.686, MnRE 0.387, Q 0.255; a rule-based table lookup gives ALDE 1.301, APE 6.180, MnRE 0.390, Q 0.249; ULIP-encoder variants give ALDE 1.323 / 1.294. All are far behind the proposed method's ALDE 0.519, APE 0.665, MnRE 0.647, Q 0.722.

• All three cues are needed. The full Image+Density+Volume model achieves ALDE 0.519, APE 0.665, MnRE 0.647, Q 0.722. Volume alone is the strongest single cue (ALDE 0.641, APE 0.887, MnRE 0.587, Q 0.615), density alone the weakest (ALDE 1.062, APE 2.001, MnRE 0.437, Q 0.390), and image alone is ALDE 0.779, APE 1.199, MnRE 0.546, Q 0.570.

• Multi-material prompts do not help. Replacing the single-material prompt with one enumerating all visible materials gives ALDE 0.532, APE 0.651, MnRE 0.640, Q 0.704 versus the single-material ALDE 0.519, APE 0.665, MnRE 0.647, Q 0.722 — a mixed, not consistently better, outcome.

• Works on unconstrained backgrounds in a small probe. The authors collected six household objects with ten images each under varying viewpoints, using Lang-SAM (GroundingDINO + Segment Anything) for preprocessing. On a qualitative sample the ground truths 0.014, 0.020, 0.075 kg were estimated by the method as 0.017, 0.023, 0.072 kg, compared with image2mass at 0.065, 0.059, 0.105 kg and RGB+Depth at 0.056, 0.036, 0.031 kg.

Methodology in Plain English

The approach starts by decomposing mass into volume times density, then asks a different visual source to inform each half.

For volume, the system predicts a pixel-wise depth map from the input image and lifts it into a point cloud — an explicit 3D shape rather than a flat appearance. The authors fine-tune GLPDepth on ShapeNetSem so the depth model handles isolated object photos, since off-the-shelf depth models are trained on indoor and outdoor scenes and tend to produce imprecise object boundaries, background leakage, or inconsistent global depth scale. Depth values are normalized by the bounding box diagonal length to keep scale consistent across objects.

For density, the system asks Qwen2.5-VL "What is this object primarily made of?" and takes the text answer as a coarse material description. This text is deliberately not treated as a numeric density; it only narrows the plausible range and gives the downstream model something semantic to reason over.

Each cue gets its own encoder: DenseNet-121 for the RGB image, PointNet for the point cloud, and a frozen CLIP text encoder for the material description, all producing 512-dimensional features followed by Layer Normalization. The image and geometry encoders are fine-tuned; the text encoder stays fixed.

Fusion is where the object-dependence is handled. The authors test concatenation into a 1,536-dimensional vector, self-attention over the modality features, and gated fusion that predicts a scalar weight per modality and combines them as a weighted sum. From the fused representation, two separate regression heads predict the volume-related factor (Sigmoid activation to keep it non-negative) and the density-related factor (a small MLP with a custom activation matching the empirical density distribution of image2mass).

The final prediction is the product of the two factors, and the only training signal is ground-truth mass, using an Absolute Log Difference Error loss that compares predictions on a logarithmic scale. All learned models were trained for 25 epochs with Adam at a learning rate of 1×10⁻⁴. Because supervision is mass-only, the predicted factors are explicitly described as physically guided latent factors rather than calibrated volume and density — any rescaling (αV̂, ρ̂/α) yields the same mass.

Why This Matters

The paper reframes a messy regression problem as a structured physical inference problem, showing that how visual evidence is organized matters as much as how much of it is collected. That is a useful lesson for other physical-property estimation tasks (friction, stiffness) that face the same geometric-plus-material ambiguity.

Real-world applications:

• Robot grasping and manipulation. Knowing an object's mass before contact lets a robot set grasp force, limit torque, and avoid crushing fragile items or overloading joints, without a preliminary test lift or force-torque sensing.

• Logistics and warehousing. Estimating package weight from a camera could support sorting and routing decisions where weighing each item is slow or impractical.

• E-commerce and product listing analysis. The image2mass benchmark itself is built from Amazon product listings, suggesting uses in validating or inferring shipping weight information from catalog images.

• Recycling and material-aware handling. Material semantics inferred alongside mass could help automated systems sort or handle items by composition.

Industry relevance: the method needs only an RGB sensor, which matters because depth cameras add hardware cost and fail on reflective or transparent surfaces. The reliance on off-the-shelf components (GLPDepth, Qwen2.5-VL, CLIP, DenseNet, PointNet) means the pipeline is buildable from existing pretrained parts rather than requiring new large-scale annotated datasets of physical properties, which are expensive and rarely available at scale.

Future Directions

• Explicit multi-material modeling. The current formulation predicts a single scalar density-related factor, which cannot represent objects that are mixtures of materials. The authors suggest long-context text encoders such as Long-CLIP to handle longer VLM material descriptions.

• Making the volume/density split identifiable. Because training uses only mass, the decomposition has a scaling ambiguity, so the two factors are not metrically calibrated. Adding any form of supervision or constraint that pins down the scale is an open problem.

• Reducing dependence on upstream foundation models. The authors note that errors in monocular depth estimation and VLM material description propagate through the pipeline, and that the system may inherit biases from those components.

• Scaling evaluation beyond curated benchmarks. The real-world test so far covers only six household objects with ten images each and manually measured masses; a larger unconstrained evaluation would test the segmentation-dependent preprocessing under more varied conditions.

Target Audience

This paper is most useful to computer vision and robotics researchers working on physical property estimation, non-contact perception, or multimodal fusion of geometry and language cues. It also suits engineers building manipulation or logistics systems who need mass estimates from ordinary cameras rather than force sensors or depth hardware. Readers interested in how vision-language models should and should not be used for physical quantities will find the negative VLM results and the density-representation comparison particularly informative.

Authors’ abstract

Estimating object mass from visual input is challenging because mass depends jointly on geometric volume and material-dependent density, neither of which is directly observable from RGB appearance. Consequently, mass prediction from pixels is ill-posed and therefore benefits from physically meaningful representations to constrain the space of plausible solutions. We propose a physically structured framework for single-image mass estimation that addresses this ambiguity by aligning visual cues with the physical factors governing mass. From a single RGB image, we recover object-centric three-dimensional geometry via monocular depth estimation to inform volume and extract coarse material semantics using a vision-language model to guide density-related reasoning. These geometry, semantic, and appearance representations are fused through an instance-adaptive gating mechanism, and two physically guided latent factors (volume- and density-related) are predicted through separate regression heads under mass-only supervision. Experiments on image2mass and ABO-500 show that the proposed method consistently outperforms state-of-the-art methods.