Tactile-JEPA: Topology-Aware Self-Supervised Representation Learning for Distributed Tactile Sensors

AI summary

Overview

• Research area: Robot learning — self-supervised representation learning for distributed tactile sensors (electronic skins / e-skins), with downstream evaluation in force and pose estimation, object and action classification, and visuo-tactile policy learning.
• Technical level: Advanced. The paper assumes familiarity with joint-embedding predictive architectures (I-JEPA, V-JEPA), masked autoencoders (MAE), self-distillation (DINO/DINOv2), BYOL, transformer encoders, exponential moving average (EMA) target networks, and standard robot-learning benchmarks.
• Scope, in one sentence: The paper proposes Tactile-JEPA, a self-supervised pre-training method that samples masks over a taxel connectivity graph at two spatial scales to produce topology-aware per-taxel embeddings for distributed tactile sensors, and shows consistent gains over BYOL-, MAE-, and DINO-based tactile encoders and end-to-end training across three datasets.

What This Paper Is About

Robots that need touch to handle contact-rich manipulation usually get tactile signals from distributed sensors whose individual sensing elements (taxels) are sparse and irregularly arranged, not laid out on a neat image grid. Most existing self-supervised tactile encoders were designed for vision-based sensors (e.g., GelSight-style cameras) and either reshape e-skin time series into an image or treat each taxel as an independent token, so the actual wiring/adjacency of the sensing surface is not represented.

The goal is a pre-training method that needs nothing beyond the sensor's own multivariate time-series output and its known layout, learns representations that capture both fine local contact detail and the overall contact state of the whole hand or body, and transfers to many downstream tasks without per-dataset tuning.

Key Contributions

1. Tactile-JEPA, a self-supervised representation learning method for spatially distributed tactile sensors that operates directly on multivariate time-series output and produces topology-aware, multi-scale per-taxel embeddings.
2. Consistent gains over prior tactile representation learning methods across multiple embodiments, sensor types, and downstream tasks, including force and pose estimation, object and action classification, and tactile-conditioned policy learning.
3. An experimental study of mask sampling: masks sampled over the sensor connectivity graph outperform image-style block masks, and their spatial scale controls locality — local or global masks favor particular downstream tasks, while mixing them transfers broadly.
4. A demonstration that no taxel coordinates are required at downstream time: the taxel graph is used only for mask sampling during pre-training.

Main Findings

• Force estimation: Tactile-JEPA reaches 14.05 ± 0.73 cN total RMSE on Sparsh-skin, versus 15.00 ± 0.92 (end-to-end), 23.79 ± 0.96 (BYOL), 21.55 ± 2.06 (MAE), and 16.66 ± 1.17 (DINO). The paper summarizes this as a 6.3% reduction in force estimation error over the strongest baseline.
• In-hand orientation: θ-RMSE falls to 5.48 ± 0.39 degrees, compared with 7.20 ± 0.87 (end-to-end), 6.92 ± 0.36 (BYOL), 8.38 ± 0.70 (MAE), and 8.27 ± 0.44 (DINO) — reported as a 20.8% reduction.
• In-hand pose accuracy: 95.74 ± 1.05 (x), 93.14 ± 0.99 (y), and 71.75 ± 2.56 (θ), all marked as statistically better than the second-ranked method.
• Object classification: 83.27 ± 3.36 top-1 accuracy, essentially tied with DINO at 83.26 ± 0.79 and well above end-to-end (70.61 ± 2.85), BYOL (64.30 ± 1.38), and MAE (63.44 ± 0.64).
• Tactile socks results: action classification accuracy of 96.91 ± 1.92 (vs 95.63 ± 1.53 for DINO, 92.97 ± 1.26 end-to-end; BYOL and MAE collapsed) and full-body pose RMSE of 13.42 ± 0.33 rad (vs 13.78 ± 0.19 DINO, 14.37 ± 0.30 end-to-end). Reported as +1.28% accuracy and a 2.6% RMSE reduction.
• Policy learning on DECO-50 is not a win for Tactile-JEPA: MAE achieves the lowest RMSE (0.4400 ± 0.0021); Tactile-JEPA is second-best at 0.4634 ± 0.0029, ahead of vision + end-to-end tactile encoder (0.4922 ± 0.0058), vision + random embeddings (0.4969 ± 0.0118), and vision-only (0.5132 ± 0.0103). BYOL and DINO both collapsed in their standard configurations on this dataset.
• Training stability is a distinguishing property: Tactile-JEPA pre-trains stably on all three datasets under the same masking configuration with no per-dataset tuning, while BYOL collapses on both Tactile socks and DECO-50 and MAE collapses on Tactile socks. DINO becomes viable on DECO-50 only after disabling the iBOT loss.
• Efficiency: Tactile-JEPA pre-trains 65–75% faster than DINO across all datasets while matching or exceeding its downstream performance.
• Masking ablation: the default mix of 2 local + 2 global targets gives the best force RMSE (14.05 ± 0.73) and in-hand x-accuracy (95.74 ± 1.05) and ranks second on DECO-50 (0.4635 ± 0.0044), while 4 global masks give the lowest policy error (0.4600 ± 0.0019). Graph-agnostic 4-block I-JEPA masking is outperformed on all three ablated tasks (force 14.99 ± 1.03, x-accuracy 93.50 ± 1.46, policy 0.4959 ± 0.0083).
• Local context hurts generality: swapping the default global context mask for a local one improves force estimation (e.g., 13.28 ± 0.83 for 2L+2G with local context) but degrades pose (x-accuracy 92.83 ± 2.17) and policy (0.4828 ± 0.0102).

Methodology in Plain English

The core idea. Take a tactile sensor made of many sensing elements spread over a surface. Instead of treating its readings as an image, treat each taxel as a token in a sequence and draw a graph connecting taxels that are neighbors in the known physical layout. This graph needs no coordinates and never changes over time; for two-part embodiments such as two hands, it simply has two connected components.

Pre-training. The model hides some taxels (called targets) and keeps the rest visible (the context). A context encoder sees only the visible taxels; a target encoder — the same architecture, updated as an exponential moving average of the context encoder — sees everything and produces the prediction goals. A small predictor receives the context embeddings plus the positions of the hidden taxels and guesses their embeddings. The loss is mean squared error between predictions and target-encoder outputs, with a stop-gradient on the target side, so only the context encoder is updated by backpropagation. Predictions are made in embedding space, avoiding direct prediction of noisy raw sensor signals.

The two distinctive design choices. First, masks are sampled on the taxel graph rather than a pixel grid. A local mask is grown from a random seed taxel by Dijkstra expansion until it fills its budget, giving a compact connected region that captures localized contact. A global mask picks taxels uniformly at random across the whole graph, reflecting overall contact state (e.g., the whole hand). Second, these two scales are mixed equally: the default configuration samples 1 context region and 4 targets (2 local, 2 global), with 1 local and 1 global target per sensing part in two-part embodiments. Context starts at 50–90% of taxels, each target at 10–18% of its sampling domain, and all target taxels are removed from the context to prevent overlap.

Input handling. Each taxel's short window of readings is flattened and passed through a shared affine projection plus layer normalization to a d-dimensional embedding — one semantic unit per contact event. Positional embeddings are learned per taxel and added before the transformer. The encoder follows a ViT-Tiny configuration (12 blocks, d = 192, 3 heads, MLP ratio 4); the predictor uses the same configuration with 4 blocks so that predictive information stays in the encoder rather than the predictor.

Temporal setup. Windows are chosen so that T_w ≈ 0.1 s, motivated by human slip-response latency, giving τ = 10 frames on Sparsh-skin and τ = 3 on DECO-50; the Tactile socks sample at 14 Hz, so τ = 5 is used instead (T_w ≈ 0.36 s). Pose estimation uses 10 consecutive windows on Sparsh-skin and 12 on Tactile socks. Pre-training does not model temporal structure across windows — that is left to the downstream head, letting the encoder be called at control-loop rate. For DECO-50, 4–5 adjacent taxels are grouped into a single token to keep the sequence length manageable (N = 1062 × 2 = 2124 taxels).

Downstream use. The target encoder is frozen and reused. An attentive probe (one-layer, 3-head transformer with a learned query token plus a two-layer MLP) handles window-level tasks; a one-layer transformer decoder over pooled windows handles sequence-to-sequence pose; and for DECO-50, pooled tactile embeddings, ResNet-18 visual embeddings, and a learnable action token form a sequence processed by a two-layer bidirectional transformer with a two-layer MLP mapping to an action chunk. Baselines are BYOL (T-DEX style, middle frame), MAE (Sparsh-skin style, masked reconstruction with taxel coordinates concatenated), DINO (Sparsh-skin style, taxel-wise masking), and end-to-end training from random initialization.

Evaluation protocol. Three encoders per SSL method with different seeds, three downstream heads per encoder on Sparsh-skin and four on Tactile socks and DECO-50, yielding 9 or 12 runs per task; end-to-end baselines use three or four independent runs. Means and sample standard deviations are reported, and the top two methods are compared with a one-sided Welch t-test at p ≤ 0.05 without multiple-comparison correction.

Why This Matters

Impact on research. The paper reframes tactile sensing quality as a problem of encoder pre-training rather than sensor hardware alone, and it isolates a concrete mechanism — graph-based, dual-scale mask sampling — that outperforms image-style block masking when the underlying masks are otherwise identical. It also identifies a practical failure mode of transferring vision SSL recipes to e-skins: BYOL and MAE collapse on some tactile datasets, while graph-based pre-training is stable across all three without per-dataset tuning. A public code release accompanies the work.

Real-world applications:

• Dexterous manipulation with thin, wide-coverage e-skins on robotic hands, where contact is spread over the palm and fingers rather than confined to bulky fingertip cameras.
• Wearable sensing for human motion, as demonstrated by action classification over 9 activities and full-body pose estimation over 19 joint angles from sensorized socks.
• Contact-rich assembly, where the DECO-50 task involves plugging a socket held in one hand with a plug held in the other, driven by a visuo-tactile policy.
• Manipulation under visual occlusion and contact with fragile objects, which the paper cites as the settings where vision alone is a limiting factor.

Industry relevance. The method reuses a single frozen backbone across sensors, embodiments, and tasks, keeping downstream heads lightweight, and it needs only the sensor's own unlabeled signal plus its layout — no task labels and no privileged simulation geometry, which the paper explicitly identifies as a limitation of HyperTaxel. Pre-training costs are modest on the reported hardware (four NVIDIA A100 80 GB GPUs; 6.1 h, 0.7 h, 2.0 h, and 5.5 h for the Sparsh-skin, Tactile socks action, Tactile socks pose, and DECO-50 encoders), and pre-training is 65–75% faster than DINO. The honest caveat for practitioners: on DECO-50 policy learning, a MAE-pretrained encoder achieved lower RMSE than Tactile-JEPA, so the advantage is not uniform across every task.

Future Directions

• Closing the policy-learning gap. Tactile-JEPA ranked second to MAE on DECO-50 (0.4634 ± 0.0029 vs 0.4400 ± 0.0021), while 4 global masks alone gave the best policy error in the ablation. The paper does not report a configuration that wins on both policy learning and the other tasks simultaneously.
• Temporal reasoning. The pre-training objective operates strictly within a short window and deliberately leaves temporal structure to the downstream decoder. Whether extending the predictive objective across windows improves contact-rich tasks is not reported.
• Co-design of masking with task scale. The ablation shows that mask scale biasses which downstream tasks improve. The paper does not report an adaptive or learned mask sampler that selects scale from the task.
• Broader sensor and embodiment coverage. Results span magnetic (Xela uSkin) and piezoresistive (knitted fabric, Inspire FTP) sensors, with a capacitive option only mentioned in related work. Generalization to capacitive skins, other taxel counts, and other layouts is not reported.
• Scale of the taxel-grouping step. For DECO-50, 4–5 adjacent taxels were grouped to reduce the 2124-token sequence. The paper does not report how grouping size affects accuracy, nor an alternative that avoids grouping.

Target Audience

• Robotics and embodied-AI researchers working on tactile sensing, dexterous manipulation, or VLA pipelines who need a reusable tactile encoder.
• Self-supervised learning researchers interested in how JEPA-style joint-embedding prediction behaves when the input is neither an image nor a language sequence, but an irregular graph of sensor channels.
• Engineers building e-skin hardware who want to know how much of the performance gap is attributable to encoder pre-training versus sensor design.
• Practitioners familiar with masked autoencoders and self-distillation who want a concrete, code-backed recipe and its failure modes when ported to sparse, irregular, non-visual signals.

Authors’ abstract

Tactile sensing is an essential modality for robots performing contact-rich, dexterous manipulation, particularly under visual occlusion. While pre-trained image encoders are standard in robot learning pipelines, tactile encoders are still commonly trained from scratch from raw, noisy signals, which might limit their expressivity. Existing self-supervised learning (SSL) approaches focus predominantly on vision-based tactile sensors, leaving distributed electronic skins largely unaddressed. These sensors, however, have a distinctive property: their sensing elements are sparse and irregularly arranged over the surface they cover, which makes direct reuse of visual SSL methods suboptimal. We present Tactile-JEPA, an efficient self-supervised pre-training method that uses the spatial arrangement of tactile sensors to learn topology-aware representations. Specifically, it is trained to predict the embeddings of masked sensing elements from the unmasked remainder, using the sensor connectivity graph to guide spatial masking. Our analysis shows that effective tactile representations require capturing both local contact details and the global state of the tactile surface, which we achieve through dual-scale masking. Across three diverse datasets spanning magnetic and piezoresistive sensors, different robot embodiments, and single- and paired-sensor configurations, Tactile-JEPA reduces force estimation error by 6.3% and in-hand orientation error by 20.8% over the prior state-of-the-art, with consistent gains in other downstream applications, including policy learning. Overall, our results demonstrate that the benefit of tactile sensing depends critically on the quality of encoder pre-training, a problem which Tactile-JEPA addresses directly. Code is available at https://github.com/E-Kovtun/tactile.