TouchScale: 500 Hours of Human Vision and Touch for Visual-Tactile Learning

AI summary

Overview

Research area: Computer Vision / Embodied AI — egocentric visual–tactile datasets and their use for perception and robot manipulation.

Technical level: Intermediate. The core ideas (synchronized sensing, contact prediction, policy mid-training) are accessible, but the evaluation protocol draws on tactile sensor layouts, representation pretraining, and vision–tactile–language–action policies.

Scope (one sentence): The paper introduces TouchScale, a 500-hour human visual–tactile dataset recorded with one consistent wearable pipeline, and measures what scaling such data buys for zero-shot tactile prediction, action-recognition representations, and real-world robot manipulation.

What This Paper Is About

Egocentric video datasets record how human hands move, but not where they press or how hard — the contact and pressure signals that matter for physical interaction. Existing visual–tactile datasets supply that missing signal, yet most contain fewer than 30 hours of interaction, and the largest ones pool recordings from different sensors and collection procedures, which makes the effect of data scale impossible to isolate. TouchScale fills this gap with 500 hours collected through a single unified wearable setup, and the paper asks whether that scale translates into better tactile perception, better visual representations, and better robot policies.

Key Contributions

1. A 500-hour dataset with one sensing pipeline. TouchScale contains approximately 87K episodes and approximately 2K task descriptions, recorded with a unified wearable rig: a head-mounted Orbbec Gemini 345Lg stereo RGB-D camera, two wrist-mounted Intel RealSense D405 RGB cameras, and bimanual tactile gloves with 880 taxels per hand. All recordings share the same hardware generation and synchronization pipeline, so visual and tactile signals stay consistent across the dataset.

2. Diversity at scale. The collection spans 1.5K+ objects, 9 high-level collection settings (laboratory, kitchen, workbench, office, bedroom, medical, packing, among others), and 800+ distinct scene configurations, with approximately 20 participants contributing variation in grasp choice, contact sequence, and trajectory. Reconstructed 3D object models are also provided.

3. An evaluation from perception to control. Models trained on TouchScale transfer zero-shot to an unseen tactile sensor; tactile-supervised pretraining on TouchScale yields visual representations that transfer to action recognition on three benchmarks; and TouchScale used as action-free visual–tactile mid-training of the N₀-VTLA robot policy raises real-world success on four contact-rich tasks from 22.5% to 57.5%.

4. A controlled scaling study. With sensor and collection protocol fixed and only the amount of training data varied, both zero-shot tactile prediction and real-world manipulation success trend upward as more TouchScale data is used.

Main Findings

• Zero-shot tactile transfer improves (size-matched comparison). With the same TouchAnything architecture, a ~16 h TouchScale subset reaches 0.181 contact IoU on EgoTactile versus 0.134 for the 16.2 h EgoTouch training split — relative gains of 35.1% in cIoU and 31.4% in vIoU (0.185 to 0.243). Center-of-pressure localization error drops from 0.444 to 0.416 and regional pressure magnitude error from 0.184 to 0.148.

• In-distribution performance is also slightly better. On each source dataset's unseen split, TouchScale reaches cIoU 0.422 versus EgoTouch's 0.403, CoP error 0.261 versus 0.272, and pressure magnitude error 0.061 versus 0.095, while vIoU is comparable (0.299 versus 0.302).

• Tactile prediction scales with data. Going from 10% to 100% of TouchScale raises zero-shot cIoU on EgoTactile from 0.311 to 0.383 and vIoU from 0.252 to 0.276, while CoP error falls from 0.332 to 0.317 and pressure magnitude error from 0.108 to 0.105.

• Better visual representations for action recognition. Pretraining Hiera-B with the same tactile prediction objective on TouchScale gives the highest accuracy on MECCANO, Something-Something V2, and Ego-Exo4D under both linear probing and fine-tuning: 24.45% average linear-probe accuracy and 49.62% average fine-tuning accuracy, compared with OpenTouch (22.97% / 48.28%), EgoTouch (8.99% / 21.75%), and FEEL (7.85% / 13.29%).

• Real-world robot success more than doubles. Using TouchScale only for visual–tactile mid-training of N₀-VTLA — without retargeting human hand motion to the robot — lifts the average success rate across four tasks from 22.5% to 57.5%, an absolute gain of 35 percentage points. The largest jump is Soft/Hard Sorting (10% to 60%); Bottle-Cap Removal goes 40% to 70%, Test-Tube Transfer 30% to 60%, and Whiteboard Wipe 10% to 40%.

• Robot performance scales with mid-training data. Average success rises from 22.5% with no mid-training to 30.0%, 32.5%, 50.0%, 50.0%, and 57.5% at 20%, 40%, 60%, 80%, and 100% of TouchScale respectively.

• Future-tactile prediction on robot data improves modestly. Accuracy on robot data unseen during mid-training rises from 0.18% without mid-training to 2.87% with 20% of TouchScale and 3.31% with the full dataset — a real but modest gain from 20% to 100%.

• Tactile and action errors are only weakly linked. After post-training, the Pearson correlation between per-step future-tactile prediction error and action prediction error is r = 0.119 (p = 0.020) with mid-training, versus r = 0.003 (p = 0.953) for the baseline.

• Dataset scale comparison. Among the datasets in the paper's comparison table, TouchScale reports the most recorded hours (500) and the densest glove-based sensing (880 taxels per hand), versus e.g. EgoTouch at 20 h, EgoTactile at 5.82 h, OpenTouch at 5.1 h, and DeskTask-Tac at 37.2 h.

Methodology in Plain English

Capture. A person wears a head-mounted RGB-D camera, two wrist cameras, and tactile gloves on both hands. The head camera records the overall scene at 1280×720 RGB and 640×480 depth at 30 Hz; the wrist cameras record close-up RGB at 1280×720 and 30 Hz, covering views the head camera misses when a hand occludes contact. Each glove has 880 pressure-sensing taxels spread over five fingers and palm at under 2 mm resolution, recording normal pressure across the whole hand. Because the same rig is reused across participants and environments and needs no manual action annotation, the collection process is meant to be repeatable at larger scale.

Task design. Each recording is tied to a natural-language instruction that names the action, objects, and target state — for example "Swirl a glass Erlenmeyer flask with both hands" or "Seal the center seam on top of a box with a tape dispenser." Repeated takes vary object instances, positions, orientations, and scene layout, and different participants produce different grasps and contact patterns for the same instruction.

Quality control. An automated pipeline cross-checks tactile signals against wrist video. Gemini 3.7 Flash watches the wrist video without seeing the tactile data and identifies grasp intervals, release events, visually confirmed contacting fingers, and contact-free periods; the tactile stream is then compared against that independent visual reference. Recordings with whole-hand signal loss during confirmed interaction, or persistent multi-finger failures, are rejected; single-finger silences go to manual review.

Evaluation, three ways. (1) Tactile prediction: train the same model on a size-matched TouchScale subset versus the EgoTouch training split, then test zero-shot on EgoTactile. Because the three gloves have different native layouts (TouchScale 32×44 with 880 valid cells, EgoTouch 21×21 with 217, EgoTactile 17×19 with 137), all layouts are pooled into 12 shared anatomical regions — five fingertips, five finger mid/base regions, palm, and thumb base — for scoring only, never for training. Metrics are contact IoU, volumetric IoU, center-of-pressure localization error, and regional pressure magnitude error. (2) Visual representation learning: pretrain Hiera-B with the same tactile prediction objective on TouchScale versus OpenTouch, FEEL, and EgoTouch, from the same checkpoint and with the same budget, then linear-probe and fine-tune on three action recognition benchmarks. (3) Robot learning: take the released N₀-VTLA vision–tactile–language–action policy, optionally mid-train it on TouchScale using its future-tactile prediction stage, then post-train on 50 teleoperated demonstrations per task and evaluate 20 trials per task on an xArm6 arm with a BrainCo Revo 2 hand.

Why This Matters

For research, TouchScale decouples two things that prior work conflated: data volume and sensing consistency. Because every hour comes from the same hardware and synchronization pipeline, an observed gain can be attributed to scale rather than to a mixture of sensors or annotation procedures — a cleaner experimental handle than pooling heterogeneous datasets. The paper also shows that human tactile data can help a robot policy without hand-motion retargeting or matched human–robot action spaces, which lowers the barrier to using human recordings.

Real-world applications:

• Contact-rich manipulation — tasks where force matters more than position, such as sorting soft from hard objects or wiping a surface with sustained contact.
• Dexterous and prosthetic hands — dense full-hand pressure data for grasp control and slip detection on hardware that lacks its own tactile arrays.
• Human–robot skill transfer — training contact-aware policies from wearable human demonstrations instead of exhaustive robot teleoperation.
• Wearable sensing and AR/VR — pressure-aware interaction models for gloves and hand-tracking systems.

Industry relevance spans robotics hardware and foundation-model teams, tactile sensor manufacturers (the zero-shot result crosses to an unseen sensor), and companies building data-collection pipelines for embodied AI, since the paper argues its wearable rig is a practical route to further scaling.

Future Directions

• Broaden robot evaluation. The authors note their robot results cover only one platform (xArm6 with a BrainCo Revo 2 hand) and four tasks; testing across embodiments and more contact-rich tasks is left as future work.
• Add action labels. TouchScale contains no action labels, which currently restricts it to mid-training rather than direct policy learning from human data.
• Clarify the role of future-tactile prediction. Only a weak association between future-tactile and action prediction errors was found (r = 0.119), and the authors explicitly call the role of future-tactile prediction in robot control an open question.
• Push the scaling curve further. The trends are upward across the tested range but the paper does not report where they saturate; whether the modest future-tactile gain from 20% to 100% mid-training data is a ceiling or a measurement artifact is unresolved.

Target Audience

Robotics and embodied-AI researchers who need contact-rich training data; computer vision researchers working on egocentric video, hand–object interaction, and cross-modal (vision-to-touch) prediction; tactile sensing and hardware engineers interested in sensor layouts and cross-sensor generalization; and practitioners building vision–tactile–language–action policies who want a concrete recipe — and measured numbers — for using human tactile data as a mid-training stage.

Authors’ abstract

Large-scale egocentric human interaction data is becoming an important source of physical supervision for embodied learning, yet video alone leaves the contact and pressure that characterize physical interaction unrecorded. Recent visual-tactile datasets provide this missing supervision, but their synchronized tactile data remain far smaller in volume than human video. Moreover, the largest resources often merge recordings from different sensors or annotation procedures, which makes the effect of data scale difficult to isolate. We therefore introduce TouchScale, a 500-hour dataset of contact-rich human interaction recorded with a single unified wearable setup. Its approximately 2K predefined task descriptions span everyday activities and structured manipulation, and each recording temporally aligns egocentric RGB-D video with wrist RGB video and dense full-hand bimanual tactile measurements. Compared with prior tactile data, training on the full TouchScale raises zero-shot contact IoU on data from an unseen tactile sensor from 0.134 to 0.383. Pretraining a visual encoder on TouchScale also yields the highest action recognition accuracy on three benchmarks among the compared visual-tactile datasets. Used for visual-tactile mid-training of a robot policy, TouchScale improves the average real-world success rate across four contact-rich manipulation tasks from 22.5% to 57.5%. With the sensor and collection protocol held fixed, both zero-shot tactile prediction and robot success show an overall upward trend as more TouchScale data is used. These results suggest that human visual-tactile data collected at scale with consistent sensing benefits both perception and robot manipulation. We will publicly release TouchScale, including all synchronized visual-tactile recordings and reconstructed object models, to support future research on scalable visual-tactile learning.