TouchFormer: A Robust Transformer-based Framework for Multimodal Material Perception

AI summary

Overview

Research area: Multimodal machine learning for robotic material perception, combining touch (force, friction, acceleration) and hearing (sound) in place of vision.

Technical level: Advanced — the paper assumes familiarity with transformer architectures, cross-modal attention, gating mechanisms, and contrastive representation learning.

One-sentence scope: The paper introduces TouchFormer, a transformer-based fusion framework that classifies surface materials from noisy, incomplete, and temporally misaligned non-visual sensor streams, and validates it on two benchmarks plus a simulated fire-scenario robot sorting task.

What This Paper Is About

Vision-based material recognition breaks down when vision is unavailable — in fire scenes, fog, or dark factories — so robots must fall back on touch and hearing. Existing non-visual multimodal methods fuse their inputs naively: they assume every modality is equally reliable and temporally aligned, which fails when sensors are noisy, missing, or differ in importance for a given material. TouchFormer's goal is to adaptively weight, fuse, and regularize these imperfect modality streams so that material classification remains accurate when real-world data is degraded.

Key Contributions

1. A robust multimodal representation learning framework (TouchFormer) that takes noisy or incomplete multimodal sequences as input and does not require explicit temporal alignment between sensors.
2. Three functionally complementary modules: Modality-Adaptive Gating (MAG), which dynamically scores and re-weights each modality and discards those below a threshold gate_th; intra- and inter-modal Transformer fusion, which injects source-modality features into target modalities via cross-modal attention and then models within-modality temporal structure; and Cross-Instance Embedding Regularization (CER), a contrastive loss that tightens intra-class and separates inter-class embeddings.
3. Benchmark results showing accuracy gains of at least 2.48% on SSMC and 6.83% on USMC over existing non-visual multimodal approaches, plus best-in-class robustness under injected modality noise.
4. A new dataset and robotic demonstration: the Fire Incident Sound and Haptic Material (FISHM) dataset, collected with a custom multimodal fingertip, and a blindfolded simulated fire material-sorting task on a Realman RM65-B 6-DoF arm with a TESOLLO Gripper-3F three-fingered hand.

Main Findings

• SSMC gains: On the LMTHM dataset's 8-class Seen Surface Material Classification task, TouchFormer reaches 95.19% accuracy (G_mean 0.95) versus DDQN at 92.71% (0.93), MMUSR at 79.7% (0.80), and MMUSR with visual input at 93.8% (0.94) — a reported improvement of 2.48%.
• USMC gains: On the 8-class Unknown Surface Material Classification task with fivefold cross-validation, TouchFormer reaches 94.38% (G_mean 0.94) versus MTCNN at 87.55% (0.88), LSTM at 74.23% (0.73), and DALNet at 46.67% (0.47) — a reported improvement of 6.83%.
• Fine-grained subclass classification: On the challenging 193-class fine-grained SSMC task the paper introduces (for example, separating softwood from hardwood), TouchFormer reaches 89.77% (G_mean 0.90) versus MTCNN at 80.21% (0.80), LSTM at 65.75% (0.66), and DALNet at 42.62% (0.43) — a 9.56% improvement.
• Vision-free comparison: The paper states that MMUSR attains 97.2% when integrating eight modalities drawn from six different sensors (a camera, multiple tactile sensors, an infrared surface reflectance sensor, and a metal detection sensor); TouchFormer operates without any visual input.
• Robustness to missing modalities: In the USMC task, removing sound, normal force, friction force, or acceleration individually still leaves TouchFormer above all other methods using full-modal input — for example, 87.92% with sound removed, 88.55% without normal force, 89.12% without friction force, and 89.84% without acceleration, versus 94.38% with all four modalities.
• Robustness to noise: Under Gaussian noise injected into randomly selected modalities with corruption ratio p ∈ {0.0, 0.1, …, 1.0} during both training and testing, TouchFormer consistently outperformed baselines across noise combinations and intensities.
• Ablation: Starting from a baseline using only the intra- and inter-modal Transformer fusion, accuracy was 91.32% / 90.17% / 83.53% on SSMC / USMC / fine-grained. Adding MAG raised this to 93.15% / 92.56% / 84.88%, and adding CER raised it further to 95.19% / 94.38% / 89.77%, with CER contributing +2.04%, +1.82%, and +4.89% respectively.
• Robotic deployment: On the FISHM dataset, TouchFormer achieved 91.47% (SSMC), 90.26% (USMC), and 90.05% (fine-grained) without noise, and 89.54%, 88.03%, and 87.76% with noise.
• Dataset scale: LMTHM contains 965 samples from 193 distinct surface materials with five samples per material, organized into eight major classes and several subclasses. FISHM comprises 7 daily material categories, with metal and stone labeled Non-combustible and the rest Combustible; the paper does not report FISHM sample counts in the main text.

Methodology in Plain English

The system takes four streams as input: sound from a microphone, normal force and friction force from a uSkin tactile sensor, and tri-axial acceleration from a 12-DoF IMU. Each stream first passes through a lightweight gating network that produces a reliability score between 0 and 1. Scores are normalized with a softmax into importance weights, and any modality scoring below the gate_th threshold is discarded outright so it cannot pollute the fused representation.

The surviving modalities go through a one-dimensional temporal convolution (so each time step sees its local neighbors) plus positional embeddings. Because the sensors have different sampling rates, the model never forces the streams onto a common timeline. Instead, cross-modal attention lets each target modality pull relevant information from each source modality, and the resulting attention output is scaled by the source's importance weight — so an unreliable source contributes less. A residual connection folds this back into the original representation, which then passes through self-attention within the same modality. The four refined streams are re-weighted by their importance scores and concatenated into one fused vector.

To make the embedding space more discriminative, a contrastive loss (CER) operates across all samples in a batch: embeddings from the same class are pulled together and embeddings from different classes pushed apart, using a temperature parameter. Training minimizes classification loss plus this contrastive loss, weighted by a factor lambda, in a single stage. Experiments used A100 GPUs and PyTorch, batch size 32, the Adam optimizer with weight decay 0.1, an initial learning rate of 0.1 decaying to 0 via cosine annealing, and 50 epochs. SSMC used a 7:3 train/test split; USMC used fivefold cross-validation matching MTCNN's protocol. Evaluation metrics were mean classification accuracy and G_mean. For the robot task, the model trained on LMTHM was deployed on the arm and fine-tuned on FISHM for domain adaptation.

Why This Matters

Impact on research: The paper reframes multimodal fusion as a robustness problem rather than a simple concatenation problem, showing that dynamic modality weighting and cross-instance regularization matter more than adding sensor types. It also introduces a fine-grained 193-class subclass benchmark and the FISHM dataset, giving the field a harder test for material perception and a vision-free evaluation setting.

Real-world applications:

• Emergency response, where smoke or flames eliminate camera input and robots must still identify flammable versus non-flammable materials to clear safe pathways.
• Industrial automation in dark, dusty, or occluded factory environments where fixed cameras cannot track parts reliably.
• Inspection and sorting in harsh conditions where vision-based tactile sensors would degrade but low-cost multi-contact sensors like uSkin remain durable.
• Human-robot or teleoperated manipulation where a robot must infer object properties without line of sight.

Industry relevance: The work targets exactly the failure mode that blocks deployment of perception models in the field — sensors that misalign, drop out, or produce corrupted readings. The use of low-cost, durable tactile hardware and a fine-tuning path from a public dataset (LMTHM) to a task-specific dataset (FISHM) maps onto realistic deployment pipelines. Funding from the National Natural Science Foundation of China (Grant 62236007) and the institutional affiliations — the Institute of Automation, Chinese Academy of Sciences, the University of Chinese Academy of Sciences, and Nanyang Technological University — indicate a robotics-focused research context.

Future Directions

• Bridging perception and manipulation: The authors note a remaining gap between material perception and manipulation; the current robot experiment uses fixed-parameter grasp programs because no visual feedback is available, so closed-loop control driven by the perception model is an open problem.
• Expanding the FISHM dataset: The main text reports only 7 material categories and does not state sample counts, so scaling it to more materials, more fire conditions, and more sensors would strengthen the robustness claims.
• Tuning and understanding the gating behavior: The paper defers analysis of gate_th and the loss weighting factor lambda to an appendix, leaving open how sensitive performance is to these choices and how stable the learned modality importance weights are.
• Narrowing the remaining gap to multi-sensor systems: TouchFormer is vision-free and approaches, but does not match, the 97.2% reported for MMUSR with eight modalities from six sensors, raising the question of which additional non-visual sensors would close that gap.

Target Audience

This paper is best suited to robotics and multimodal machine learning researchers working on tactile sensing, sensor fusion, and robust perception; graduate students looking for a concrete example of combining gating, cross-modal attention, and contrastive regularization in one framework; and applied engineers building vision-free or vision-degraded robotic systems for emergency response, industrial inspection, or harsh-environment automation. Practitioners focused on tactile hardware selection will also find the comparison of vision-based tactile sensors against multi-contact sensors like uSkin relevant to durability and cost trade-offs.

Authors’ abstract

Traditional vision-based material perception methods often experience substantial performance degradation under visually impaired conditions, thereby motivating the shift toward non-visual multimodal material perception. Despite this, existing approaches frequently perform naive fusion of multimodal inputs, overlooking key challenges such as modality-specific noise, missing modalities common in real-world scenarios, and the dynamically varying importance of each modality depending on the task. These limitations lead to suboptimal performance across several benchmark tasks. In this paper, we propose a robust multimodal fusion framework, TouchFormer. Specifically, we employ a Modality-Adaptive Gating (MAG) mechanism and intra- and inter-modality attention mechanisms to adaptively integrate cross-modal features, enhancing model robustness. Additionally, we introduce a Cross-Instance Embedding Regularization(CER) strategy, which significantly improves classification accuracy in fine-grained subcategory material recognition tasks. Experimental results demonstrate that, compared to existing non-visual methods, the proposed TouchFormer framework achieves classification accuracy improvements of 2.48% and 6.83% on SSMC and USMC tasks, respectively. Furthermore, real-world robotic experiments validate TouchFormer's effectiveness in enabling robots to better perceive and interpret their environment, paving the way for its deployment in safety-critical applications such as emergency response and industrial automation. The code and datasets will be open-source, and the videos are available in the supplementary materials.