Reproducible Multimodal Affordance Prediction

AI summary

Overview

Research area: Multimodal affordance prediction in computer vision and robotics, with a focus on reproducibility, benchmarking transparency, and deployment-oriented evaluation.

Technical level: Intermediate.

Scope: The paper diagnoses why affordance prediction models are hard to compare and reproduce, and proposes a structured documentation template — the Affordance Sheet — to standardise reporting of task formulation, datasets, methods, experimental setups, performance measures, and real-world validation.

What This Paper Is About

Affordance prediction asks what actions an agent can perform on an object, using inputs such as images, depth, language, point clouds, audio, or tactile data. The field has fragmented into different formulations (classification, segmentation, grounding, grasping detection, hand pose estimation) built on datasets with inconsistent annotations and incompletely reported training and evaluation protocols, which makes fair comparison and reliable deployment difficult. The paper's goal is to expose these reproducibility gaps and provide a practical documentation tool that forces the missing information into the open.

Key Contributions

1. The Affordance Sheet. A structured Open Science document, inspired by Model Cards, detailing for each method: task formulation and input modalities; datasets; the proposed method and its code and weights; the experimental setup; performance measures; and validation through generalisation, robustness, robot deployment, and human safety.

2. A named taxonomy of reproducibility challenges (RC1–RC5). RC1: data availability for benchmarking. RC2: availability of a method's implementation. RC3: availability of trained models. RC4: details of experimental setups. RC5: details of performance measures for evaluation. Each section of the Affordance Sheet is mapped to the challenge it addresses.

3. A systematic survey of datasets and benchmarks. A table of affordance datasets grouped by formulation, recording input modality (vision, language, point cloud), number of images, object categories, affordance categories, and whether real-world data, transparency, and hand occlusion are considered.

4. Four worked Affordance Sheets for state-of-the-art methods. EgoTopo (functional classification), ACANet (affordance segmentation), Multi-FinGAN and HUG (hand pose estimation), used to show concretely how reporting practices differ across published work.

Main Findings

• No existing dataset was collected for benchmarking under varied in-the-wild conditions (RC1). The paper states that none of the surveyed datasets tests models under different illumination, clutter, or hand-occlusion conditions.

• Annotation cost caps dataset scale. Precise affordance annotation is manual and time-consuming, which limits most datasets to fewer than 50,000 images and fewer than 10 affordance categories. The largest entries in the survey are HANDAL (308,000 images, 17 object categories, 1 affordance category) and CHOC-AFF (138,240 images, 3 object categories, 3 affordance categories), both of which scaled annotation via synthetic data or weakly/self-supervised approaches.

• Datasets are formulation-specific and rarely reused across tasks. Each affordance dataset is collected for one formulation and is not reused across different affordance tasks, which limits fair comparison. Cross-dataset evaluation is mostly avoided because affordance and object classes only partially overlap and are labelled with different conventions; datasets such as UMD, IIT-AFF, and Multi-View share some classes but use different labelling conventions.

• Controlled laboratory conditions dominate. Most datasets place a single unoccluded object on a planar surface (for example a tabletop) with fixed camera viewpoints and illumination, varying only object category or instance. UMD and Multi-View, for instance, were collected with constant lighting and rotating platforms and a nearly constant camera–object distance.

• Training setups on the same dataset are not comparable. Table 2 compares affordance detection and segmentation methods on UMD: AffordanceNet and BPN used 1000×600 resolution, CNN and DRNAtt 320×240, RANet 224×224, and GSE 400×400. Augmentation differed too: AffordanceNet, CNN, DRNAtt, and RANet did not consider flipping, scaling, rotating, or colour jittering; GSE considered flipping and scaling but not rotating or jittering; BPN considered all four. Resize procedures at test time differ (centre-crop, sliding window, or unknown).

• Undocumented implementation details break reproducibility (RC2–RC4). AffordanceNet and BPN did not report image resize during training and testing; DRNAtt, RANet, and GSE omitted these details for the testing phase; Gu et al.'s work omitted optimizer parameters. Apicella et al.'s work showed that missing setup details led to unfair comparisons.

• Proxy comparisons appear in the literature. AffordanceDiffusion was compared against the coarse hand prediction of GanHand because object pose annotation is missing from the dataset, and because part of the architecture and training procedure is missing, the result is only a proxy for the unknown true performance of GanHand.

• Evaluation is inconsistent across loosely related formulations (RC5). Affordance grounding works do not compare against affordance segmentation works despite similar problem formulations. STRAP was measured with IoU instead of F^w_beta, which most other methods use. Mean IoU itself is ambiguous, since it can be the average of per-image IoUs or the IoU over the full set of predictions and annotations.

• Benchmark practice differs from mainstream computer vision. COCO releases only training and validation sets and keeps the test set private. Affordance benchmarks such as UMD and IIT-AFF release both training and testing data, which risks tuning methods to test-set scores rather than advancing the field.

• Reporting quality varies sharply across the four profiled methods. ACANet gives the most complete description: training (89,856 images), validation (17,280), and two test splits (13,824 and 17,280), plus extensive hyperparameters (batch size 2, learning rate 0.001, SGD with momentum 0.9, weight decay 0.0001, resize [1, 1.5], flip 0.5, centre-crop to 480×480). EgoTopo omits training and validation split sizes (reporting only 1,155 test images) and several preprocessing parameters. Multi-FinGAN reports only a 3,000-image training split and leaves several training hyperparameters unspecified.

• Metric definitions are often absent. ACANet and HUG give formal definitions (per-class Jaccard index, and fingertip contact error FC = 1/2(|d_thumb| + min over non-thumb fingers of d_f)); EgoTopo and Multi-FinGAN only describe their metrics without a mathematical formulation. No method documented the limitations of its chosen performance measures.

• Robot deployment is rarely reported. Neither EgoTopo nor ACANet describes robotic deployment. Multi-FinGAN reports a Franka Emika Panda with a Barrett hand and an Intel RealSense D435 camera at a 45-degree viewpoint, generating 20 grasps per object and executing the first physically reachable grasp with the lowest intersection and highest quality metric; HUG reports a 7-DoF xArm with a 6-DoF Ability hand driven by an external ZED camera, with MANO grasps mapped to the robot hand.

• Generalisation is tested by only some methods. HUG was tested in real and simulated environments on 90 unseen objects spanning 5 geometric categories (cylindrical, spheroidal, prismatic, appendaged, amorphous) and 3 size bins (small, medium, large). ACANet and Multi-FinGAN were tested on real images different from their synthetic or mixed-reality training data.

• No method assessed robustness or human safety. The paper reports that none of the four profiled methods evaluated robustness to occlusions or sensing disturbances, or evaluated safety for humans.

Methodology in Plain English

The authors begin by surveying the landscape rather than proposing a new model. They catalogue the datasets used in affordance prediction, grouped by how the affordance problem is formulated, and record modality, image counts, object and affordance category counts, and whether real-world, transparency, and hand-occlusion conditions are covered. They then inspect published methods and their papers to see what is missing — code, trained weights, preprocessing steps, hyperparameters, metric definitions — and condense those gaps into five reproducibility challenges.

From these gaps they design a reporting template. The Affordance Sheet borrows its structure from Model Cards and Datasheets for datasets but adds the task-specific detail those tools lack: how the affordance is defined, which modalities are used, whether a human is in the scene, how performance metrics are mathematically defined, and whether the method was validated for generalisation, robustness, robot deployment, and human safety. To keep the template comparable across work, the authors unify related formulations — affordance detection and segmentation plus affordance grounding become "functional segmentation", affordance classification becomes "functional classification", and grasping detection is treated as a case of hand pose estimation.

Finally, they fill in the template for four published methods and compare the results side by side. The comparison is the evidence: differences that are invisible when reading each paper alone become explicit when the fields are lined up. The authors also note practical points, such as that publicly available software without a license is automatically protected by copyright and therefore cannot legally be reused by other researchers to reproduce results.

Why This Matters

Non-reproducible affordance models are not merely an academic accounting problem: these models can drive physical interaction with the world, so unverified claims about robustness, generalisation, and safety carry direct physical risk. The paper argues that reproducibility underpins credibility of findings and enables reliable, safe interaction with humans in real environments.

Real-world applications the paper identifies as benefiting from affordance prediction:

• Human-robot collaboration, where inaccurate affordance estimation around a person's hands may cause unsafe or unintended interactions.
• Wearable robotics for human-to-human collaboration.
• Immersive tutorials, where an agent must recognise what a person can do with an object.
• Robotic grasping and manipulation, demonstrated by Multi-FinGAN on a Franka Emika Panda with a Barrett hand and HUG on a 7-DoF xArm with a 6-DoF Ability hand.

Industry relevance: teams deploying affordance or grasping models on real robots need to know whether a published result transfers to their camera setup, object set, and occlusion conditions. The Affordance Sheet gives evaluation and procurement teams a checklist for what a model claim actually covers, and gives dataset and model publishers a licensing and metadata structure (persistent identifiers, versioning, licenses) that supports long-term reuse. The paper also notes the tool is generic enough to extend to Vision-Language-Action Models, affordance foundation models, and tasks beyond affordance prediction.

Future Directions

• Benchmarks with held-out test sets and in-the-wild conditions. The authors recommend that future benchmarks release clear usage and visualisation instructions and evaluate models under generalisation across unseen object instances, novel object categories, camera viewpoints, lighting conditions, clutter, partial occlusions, and different sensing modalities — and follow the COCO-style practice of keeping the test set private.

• Multi-metric evaluation with documented limitations. They recommend evaluating models with more than one performance measure and avoiding single aggregated scores, while explicitly documenting the assumptions and failure cases of each metric, which no profiled method did.

• Re-implementation and release of missing models. Because trained weights and implementations are often unavailable, the authors encourage the community to re-implement, retrain, and release models — as previous work such as Apicella et al. and the ACANet authors did — together with a license that permits reuse.

• Deployment and safety validation as standard practice. Robot-platform validation, including platform, end-effector, setup, task, and success criteria, plus standards-based tests for human safety, should become part of reported results rather than an occasional extra.

• Open question on cross-formulation comparison. The paper leaves open how to definitively compare methods across related formulations — for example whether using the action as input, as affordance grounding does, provides any advantage over affordance segmentation — suggesting conversion between confidence maps and binary masks as one route.

Target Audience

Researchers and graduate

Authors’ abstract

Affordance prediction is the identification of potential actions an agent can perform on a target object from multimodal inputs. Affordance prediction methods are difficult to evaluate and compare due to heterogeneous problem formulations, inconsistent dataset annotations, incomplete reporting of experimental protocols, and limited information about deployment conditions. These limitations challenge fair benchmarking and performance comparison. To promote transparency, we propose the Affordance Sheet, a documentation detailing task formulation with its input modalities, model architectures and training information, datasets, and experimental protocols. Affordance Sheets enable reproducible benchmarking and reliable evaluation of affordance models for real-world scenarios, including generalisation to novel conditions and human safety.