EgoPHI: Estimating Contact and Force from Egocentric Vision

AI summary

Overview

• Research area: Egocentric computer vision, hand-object interaction understanding, and physically grounded 3D perception.
• Technical level: Advanced. The paper assumes familiarity with 3D hand/object meshes (MANO-style hand models, template object meshes), graph neural networks, Vision Transformers, rigid-body physics simulation, and contact/force evaluation metrics.
• Scope: The paper introduces EgoPHI, a vision-based method that jointly estimates dense per-vertex contact maps and 3D force distributions on both hands and manipulated (rigid or articulated) objects from a single egocentric RGB image plus object geometry, trained on physics-simulated force supervision and validated on simulated, out-of-distribution, and real-world instrumented objects.

What This Paper Is About

Recent vision models can reliably detect where hands touch objects, but not how hard they press. EgoPHI addresses this gap by inferring the magnitude and 3D direction of forces at every vertex of the hand and object meshes, not just 2D pressure heatmaps or forces on flat surfaces. The core challenge is that dense ground-truth force annotations are essentially unavailable for natural hand-object interaction, so the authors generate them through physics simulation and then test whether the learned model transfers to real, non-planar objects.

Key Contributions

1. EgoPHI, described as the first vision-based method that estimates 3D force distributions and contact regions over both hand and object meshes during manipulation (previous work covers hands only, or objects not at all).
2. A physics-based force simulation module that converts existing hand-object datasets (with hand and object meshes and recorded motion) into dense per-vertex contact and force supervision, using a penalty-based soft-spring formulation in the SOFA simulator with a constant stiffness of k = 10.
3. An instrumented real-world dataset of ground-truth hand-object contact and 3D forces, captured with two fabricated objects (a cube and a cylinder) from eight participants during touch and grasp interactions, to test sim-to-real transfer.
4. Extending force estimation beyond flat surfaces to rigid and articulated rigid objects, where a Graph-Based Interaction Block performs intra-mesh and inter-mesh 3D reasoning and an Iterative Refinement Module corrects object pose under occlusion.

Main Findings

• Force accuracy improvement over prior work: On ARCTIC (in-distribution), EgoPHI reaches hand force MAE of 4.03 N and RMSE of 5.06 N, versus 6.62 N and 7.29 N for HACO. The abstract reports this as reducing the mean error by approximately 2 N. On H2O (out-of-distribution), EgoPHI's hand force MAE is 5.16 N and RMSE 6.62 N versus HACO's 6.37 N and 7.13 N.

• Outperformance of a specialized 2D force baseline: Against PressureVision on projected 2D metrics, EgoPHI achieves F1 0.271 and IoU 0.157 on ARCTIC (versus 0.128 and 0.068) and F1 0.053 and IoU 0.027 on H2O (versus 0.005 and 0.003).

• Only method reporting object contact and force: HACO is marked "not supported" for object contact and force. On ARCTIC, EgoPHI reports object force MAE 4.42 N, RMSE 5.28 N, vIoU 2.0, and object contact precision .033, recall .517, F1 .060, IoU .032. On H2O, object force MAE 3.88 N, RMSE 4.18 N, vIoU 0.7.

• Iterative Refinement Module (IRM) matters: Removing it drops 2D ARCTIC IoU from 0.157 to 0.021 and volumetric IoU from 15.4 to 1.6; on H2O, IoU falls from 0.027 to 0.010 and volumetric IoU from 2.7 to 0.8. The full model improves hand and object contact and object force, with the single minor exception of H2O hand forces, where the ablated model is better by roughly 10%.

• Pose refinement reduces alignment error: Mean Per-Vertex Error drops by over 70 percent across refinement iterations, from 64 cm to 19 cm on ARCTIC and from 48 cm to 10 cm on H2O.

• Contact remains hard: The paper notes absolute contact scores (F1 < 0.2, IoU < 0.15 on ARCTIC) reflect the inherent difficulty of the task, where ground-truth contact regions are sparse. HACO attains higher hand-contact recall, which the authors attribute to its pretraining on 14 hand contact datasets.

• Real-world sim-to-real transfer works: On the instrumented objects, EgoPHI achieves object contact F1 0.151 and IoU 0.085 on the cube, with force MAE 0.48 N and RMSE 1.54 N; on the cylinder, contact F1 0.121, IoU 0.067, force MAE 1.35 N, RMSE 3.24 N. Force errors are lower than on ARCTIC.

• Simulation fidelity validated: SOFA-simulated forces are close to real FTIR measurements. For the cylinder, mean simulated force is 1.02 ± 0.59 N versus 1.21 ± 0.50 N measured, per-frame MAE 0.36 ± 0.18 N, Spearman ρ .760. For the cube, 1.61 ± 0.39 N versus 1.67 ± 0.39 N, MAE 0.25 ± 0.16 N, Spearman ρ .604.

• Hand pose domain gap is a tradeoff: Fine-tuning with HAMER-estimated hand poses improves hand force (MAE 4.03 to 3.32 N, RMSE 5.06 to 4.08 N, vIoU 2.2 to 2.9) but degrades hand contact (F1 0.190 to 0.128, recall 0.500 to 0.212) and slightly worsens object force (MAE 4.42 to 4.73 N, vIoU 2.0 to 1.8), while object contact stays comparable (F1 0.060 to 0.061).

Methodology in Plain English

The system takes three things as input: one egocentric RGB image, the 3D coordinates of the left and right hand meshes, and the 3D coordinates of the object mesh. A Vision Transformer (ViT-B/16) extracts image features, and each mesh vertex is projected into the image so its visual feature can be sampled and combined with its 3D position.

Because hands occlude the object in egocentric views, the model cannot trust the object's initial pose. It therefore runs an Iterative Refinement Module for three iterations: the hands stay fixed and the object's features are updated using its current 2D projection, its template coordinates, and its geometry relative to each hand's center. A stack of Graph-Based Interaction Blocks does the reasoning — Graph Attention Networks capture structure within each mesh, cross-attention lets each mesh attend to the other two, and a feed-forward network refines the features. A small decoder then predicts an incremental rotation (in 6D representation) and translation, which are composed onto the current estimate.

With the refined object pose, a second stack of Interaction Blocks predicts, for every vertex on the left hand, right hand, and object, a contact probability and a force. Forces are decomposed into magnitude and unit direction for numerical stability, then multiplied by the predicted contact probability so non-contact vertices are suppressed.

Training labels come from simulation, since real per-vertex force annotations do not exist at scale. Hand and object meshes are loaded into the SOFA physics simulator as triangle surface models with uniform mass. Interactions are modeled as soft virtual springs that activate when vertices enter a contact proximity zone, producing a restoring force proportional to penetration depth along the surface normal, with stiffness fixed at k = 10. A binary contact mask filters out spurious non-contact forces. Forces are normalized by the maximum observed magnitude per mesh.

The model is trained with Adam (learning rate 1×10⁻⁴, batch size 8) on ARCTIC only, using weighted binary cross-entropy for contact plus vertex-level class-balanced, dataset-level contact, and smoothness regularizers, and L2-style losses for force magnitude, force direction, and pose. At inference, ground-truth hand poses are replaced with HAMER estimates.

For the real-world study, the team built an 8 mm acrylic cube and cylinder. An LED strip inside each object causes light to escape only where a finger touches the surface, and brighter reflections correspond to stronger pressure — providing an optical proxy for force. A fisheye camera (8 MP Sony IMX179, near-180° lens) below each object captured contact areas, and ArUco markers tracked object pose. Eight participants wore a Logitech C920 webcam (78° diagonal FOV) and performed two-, three-, four-, and five-finger grips, full-hand grasps, and individual touches and presses, about ten repetitions per gesture with each hand and both hands, yielding roughly 2000 synchronized pairs of egocentric and contact+force frames. Light intensity was calibrated to force using a Beurer scale with 1 g resolution, pressing in 50 g increments up to 1000 g (9.81 N) and averaging ten frames per step.

Why This Matters

Egocentric perception has largely been about recognizing contact and pose. This work argues the next layer is physical: forces reveal how an object is held, pressed, stabilized, or controlled, and therefore carry information about intent, stability, and manipulation control that contact alone does not. It also offers one of the first experimental demonstrations that a vision-only system can estimate 3D force distributions on non-planar real objects without tactile sensing at test time.

Real-world applications:

• Robotics and imitation learning: Robots observing human manipulation could infer grasp forces and learn stable grasping or object handling strategies from video.
• AR/VR and telepresence: Force-aware hand-object understanding could drive more physically plausible interaction feedback and haptic rendering without instrumented gloves.
• Skill assessment and training: Applications concerned with how firmly a tool, instrument, or object is held or pressed could use force estimates as feedback signals.
• Benchmark and dataset infrastructure: The instrumented-object dataset gives the community a way to measure sim-to-real transfer for force-aware perception, where ground truth was previously unobtainable.

Industry relevance. The work targets any pipeline where egocentric cameras are already present but tactile sensors are not — assistive and wearable computing, human-robot collaboration, manufacturing and assembly analysis, and surgical or minimally invasive settings where force cues matter but instrumenting the environment is expensive.

Future Directions

• Temporal modeling: EgoPHI processes frames independently and discards temporal structure. Since forces are inherently dynamic, a sequence-aware model could enforce temporal consistency and recover force trajectories a per-frame approach cannot produce.
• Removing the mesh template requirement: The model needs the object mesh template as input. Integrating feed-forward mesh reconstruction models such as SAM3D could make it applicable to fully unconstrained egocentric video.
• Material-aware force simulation: The simulation applies uniform stiffness across all objects. The authors note that force ratios between vertices are k-independent under a linear penalty, so spatial force distribution is preserved, but this limits accuracy for objects with heterogeneous or compliant material properties.
• Tangential forces and global equilibrium: EgoPHI predicts only normal contact forces, matching the simulation and FTIR sensing setup. Extending to tangential forces would require additional sensing, friction-aware simulation, and learning from weaker visual cues; one proposed route is post-hoc inference combining predicted normal forces with object motion via Newton's second law. The model also lacks object properties such as mass needed to enforce global force equilibrium.
• Truly curved test objects: The two real-world objects evaluated are a cube (piecewise planar) and a cylinder (developable lateral surface). The authors suggest their fabrication-based approach for retrofitting existing 3D objects with surface-conforming capacitive sensing as a first step toward ground truth on more complex geometries.

Target Audience

Researchers and graduate students in computer vision, human-computer interaction, and robotics who work on hand-object interaction, egocentric perception, contact estimation, or physically grounded scene understanding. It is also relevant to practitioners building force-aware systems for AR/VR, teleoperation, or imitation learning, and to those interested in sim-to-real transfer and the design of instrumented sensing hardware for dataset collection. Readers without a background in 3D mesh representations, graph attention networks, or physics simulation will find the method sections demanding.

Authors’ abstract

Understanding hand-object interaction from egocentric vision is essential for modeling how people physically engage with the surrounding world. Yet reasoning about physically grounded interaction requires estimating the forces acting on hands and objects, beyond localizing contact. We present EgoPHI, the first method that jointly estimates dense contact maps and 3D force distributions on hand and object meshes from a single monocular RGB image and object geometry. To address the lack of scalable ground-truth force annotations, we introduce a physics-based simulation pipeline that augments existing hand-object datasets with dense per-vertex force supervision. EgoPHI then learns dense 3D contact and force on interacting hand and articulated object meshes, extending vision-based force estimation beyond image-space or planar settings. Our evaluation on in-distribution and out-of-distribution benchmarks shows that EgoPHI improves force estimation over existing approaches while generalizing to unseen datasets. To evaluate sim-to-real transfer, we constructed two physical objects that capture dense object contact and force magnitude and used them to record a dataset of interactions from eight participants across diverse touch and grasp types. Our results demonstrate that EgoPHI recovers meaningful 3D contact and force distributions in simulated, out-of-distribution, and real-world settings, advancing egocentric hand-object understanding from contact localization toward physically grounded interaction reasoning.