Shoe Style-Invariant and Ground-Aware Learning for Dense Foot Contact Estimation

AI summary

Overview

• Research area: Computer vision, specifically dense human foot–ground contact estimation from a single RGB image (3D human pose/contact understanding).
• Technical level: Advanced. The paper combines adversarial style randomization, Vision Transformer backbones, and geometric supervision (pixel height maps, ground normals) and assumes familiarity with SMPL-X meshes, AdaIN, and dense contact estimation benchmarks.
• Scope: The paper introduces FECO, a framework that predicts dense per-vertex foot contact from one image using shoe style-invariant and ground-aware learning, plus a new in-the-wild annotation dataset called COFE.

What This Paper Is About

Most prior work reduces foot contact to a coarse joint-level signal, often derived from zero-velocity constraints rather than from what the image actually shows. The authors argue that dense, spatially distributed foot contact is largely unexplored from a single RGB image, and that two factors make it hard: shoes have highly diverse appearances that create spurious correlations with contact patterns, and ground surfaces are visually monotonous and offer weak cues. The goal is a dedicated dense foot contact model that is robust to shoe style and explicitly reasons about the ground as a physical surface.

Key Contributions

1. FECO framework. The authors propose FECO (FEet COntact estimation), described as the first method dedicated to dense foot contact estimation from a single image, targeting both the shoe style diversity problem and the foot–ground reasoning problem.
2. Shoe style–content randomization. Using the external shoe image dataset UT Zappos50K, they enforce style-invariant but content-preserving features, with a content-randomized branch used for adversarial training and a style-randomized branch that perturbs shoe appearance during training.
3. Ground-aware learning. They supervise the model with pixel height maps and ground normals to encode fine-grained and global geometric properties of the ground, then fuse the ground feature with the shoe style-invariant feature via spatial attention.
4. The COFE dataset. They manually annotate binary foot contact labels on foot keypoints from OpenPose, InstaVariety, PennAction, and MPII, producing an in-the-wild dataset that is used both for training and for evaluating joint-level foot contact.

Main Findings

• Low-level randomization helps. On MMVP, adding progressive low-level style randomization (Pro-RandConv) raises precision from 0.544 to 0.563, recall from 0.584 to 0.613, and F1 from 0.555 to 0.577, which the authors report as increases of 3.5% precision, 5.0% recall, and 4.0% F1.
• Both content and style randomization are needed. On MMVP, neither branch alone (0.531 F1 with content only, 0.554 F1 with style only) matches the combination (F1 0.577); content randomization alone gives the highest recall (0.651) but lowers precision to 0.504.
• Ground-aware learning compounds gains. On MMVP, the baseline without ground supervision scores F1 0.506; adding ground normals gives 0.527, adding pixel height maps gives 0.569, and adding spatial attention gives 0.577.
• COFE data improves in-the-wild performance. Training with 3D mocap datasets alone yields precision 0.494, recall 0.464, F1 0.450 on the COFE test set; adding COFE training data improves all three to 0.553, 0.516, and 0.515.
• FECO outperforms prior dense contact methods on MMVP. FECO reaches precision 0.563, recall 0.613, F1 0.577, versus POSA (0.276/0.308/0.255), BSTRO (0.436/0.538/0.464), and DECO (0.374/0.511/0.409).
• FECO beats feature-level style randomization alternatives. On MMVP, BIN scores 0.505/0.351/0.396, MixStyle 0.437/0.463/0.448, SagNets 0.451/0.564/0.511, and a re-implemented LatentDR 0.534/0.574/0.542, all below FECO's 0.563/0.613/0.577.
• Single-image FECO surpasses temporal joint-level methods. On the COFE test set (video sequences only, toe and heel joints), Footskate Reducer scores 0.399/0.271/0.301 and WHAM scores 0.347/0.431/0.363, while FECO—the only compared method without temporal information—scores 0.553/0.516/0.515.
• OpenPose was excluded from the test set. The authors found it predominantly contains fully contacted feet; predicting all joints as contact there yields precision 0.712, recall 0.858, F1 0.760, so the test set was built from InstaVariety instead.
• Reported scale of data used. The paper assembles 10 datasets (Table 1), ranging from PROX at 0.2K images to MotionPRO at 12.4M, and uses UT Zappos50K (50,025 images across shoes, sandals, slippers, and boots, with open, pointy, sporty, and comfort attributes) as the style source.
• Not reported. The truncated content does not include the conclusion of the supplementary sections S1–S10, so results for additional datasets, alternative backbones, and computational cost are not available here.

Methodology in Plain English

FECO is trained with three synchronized inputs per sample: one clean image and two images passed through progressive random convolutions that perturb low-level texture (random convolution weights, deformable offsets, and affine parameters). All three go through a ViT backbone, initialized from ImageNet-pretrained ViT-Huge weights, to produce image features, and a foot segmentation mask is predicted with a DPT decoder to localize the foot.

From the image features, two parallel branches are built using random shoe images from UT Zappos50K, which are encoded by a frozen ViT. In the content randomization branch, the shoe feature's channel statistics are replaced by the input image's statistics, so the shoe's content is expressed in the input's style; this branch feeds an adversarial loss that pushes its contact predictions toward a uniform distribution, training only the small residual adapters (a 3×3 convolution with zero-initialized weights scaled by a learnable parameter initialized to 0.02). In the style randomization branch, channel statistics of the input and shoe features are mixed with a random interpolation weight drawn from a uniform distribution, then applied via AdaIN, producing a shoe style-invariant representation.

Both branches feed a ground feature encoder that outputs four levels of features. A DPT decoder predicts a dense pixel height map (scaled by the maximum image side length so outputs are in pixel units, inspired by Depth Anything's depth scaling factor), and the final-level feature, with the foot region masked out to prevent shortcut learning from foot orientation, is pooled to predict a unit-length ground normal through two fully connected layers with hidden dimension 128 and L2 normalization. A spatial attention module concatenates the randomized and ground features, reduces channels to 256, applies dropout at rate 0.2, and produces two softmax-normalized weights per location that blend the two feature sources. A Transformer-based foot contact decoder adapted from HACO maps the fused feature to vertex-level logits over 265 foot vertices (HACO predicts 778 vertices on the MANO hand mesh). Logits are projected to coarser levels with N = 3 and vertex counts in {V, 11, 3}: the full SMPL-X foot region, 11 foot regions (five toes, heel, front, bottom, left, right, back), and 3 OpenPose foot keypoints via a custom SMPL-X-to-OpenPose regressor.

Training uses a total loss combining a binary cross-entropy main loss, a style-branch loss whose gradients are restricted to that branch's contact decoder, the style adversarial loss, a mask loss averaging binary cross-entropy and Dice loss, and a ground loss made of mean absolute error on pixel height plus cosine similarity on ground normals. Optimization uses AdamW at learning rate 10⁻⁵ with mini-batch size 4, learning rate decayed by 0.9 after the 5th and 10th epochs, 10 epochs total on a single NVIDIA A6000 GPU. Evaluation reports precision, recall, and F1 after thresholding predicted probabilities, excluding samples with no positive contact, with MMVP as the main evaluation dataset.

Why This Matters

• Impact on research: The paper argues that foot contact has been approximated by zero-velocity heuristics and joint-level labels, and that dense body contact models like POSA, BSTRO, and DECO remain inaccurate on feet. It extends the pattern seen in hand contact work (HACO) that a body-part-specific dense contact model can beat general body models, and it introduces a new in-the-wild annotated dataset (COFE) to support that line of work.
• Real-world applications (implications drawn from the paper's stated motivations for modeling human–environment interaction):
  • Physically consistent human motion capture and animation, where feet must not slide or float relative to the environment.
  • Robotics and humanoid balance/control, where knowing where the foot contacts a surface informs stability reasoning.
  • AR/VR and avatar systems that must place virtual humans plausibly on real floors, carpets, or other surfaces.
  • Sports and biomechanics analysis, where foot–ground interaction patterns are relevant to movement quality.
• Industry relevance: The method is designed for single-image, in-the-wild input and is released as open code, which makes it applicable to settings where multi-camera or pressure-sensor setups are unavailable—domains such as gaming, film, telepresence, and motion analysis from ordinary video or photos.

Future Directions

• Extending evaluation beyond the truncated content. The supplementary sections promise quantitative results on more datasets and on different backbones, which the visible text does not include; those comparisons are open questions from this excerpt.
• Joint dense and joint-level reasoning. FECO already projects dense predictions to OpenPose foot keypoints and beats temporal joint-level methods without using time; combining dense contact with temporal cues is a natural extension the paper's comparison implies.
• Handling annotation ambiguity. The COFE annotation rules classify snowboards as contact, skis as non-contact unless touching a surface, and clothing as non-contact, and images with fewer than two valid foot joints were removed—suggesting that ambiguous contact categories remain a boundary case.
• Reducing reliance on external style sources. The method depends on UT Zappos50K for shoe style images and on 3D mocap datasets plus COFE for training; how far the style-invariance generalizes to shoe types and materials outside those sources is not established in the visible content.

Target Audience

Researchers and graduate students in computer vision working on 3D human pose and shape, human–scene interaction, and dense contact estimation; practitioners building motion capture, animation, or AR/VR avatar systems who need foot contact from ordinary images; and dataset builders interested in annotation protocols for in-the-wild contact labels. The paper is also relevant to robotics researchers studying foot–ground interaction, though the technical machinery assumes a strong background in vision transformers and mesh-based human models.

Authors’ abstract

Foot contact plays a critical role in human interaction with the world, and thus exploring foot contact can advance our understanding of human movement and physical interaction. Despite its importance, existing methods often approximate foot contact using a zero-velocity constraint and focus on joint-level contact, failing to capture the detailed interaction between the foot and the world. Dense estimation of foot contact is crucial for accurately modeling this interaction, yet predicting dense foot contact from a single RGB image remains largely underexplored. There are two main challenges for learning dense foot contact estimation. First, shoes exhibit highly diverse appearances, making it difficult for models to generalize across different styles. Second, ground often has a monotonous appearance, making it difficult to extract informative features. To tackle these issues, we present a FEet COntact estimation (FECO) framework that learns dense foot contact with shoe style-invariant and ground-aware learning. To overcome the challenge of shoe appearance diversity, our approach incorporates shoe style adversarial training that enforces shoe style-invariant features for contact estimation. To effectively utilize ground information, we introduce a ground feature extractor that captures ground properties based on spatial context. As a result, our proposed method achieves robust foot contact estimation regardless of shoe appearance and effectively leverages ground information. The codes are available at https://github.com/dqj5182/FECO_RELEASE.