Adding internal audio sensing to internal vision enables human-like in-hand fabric recognition with soft robotic fingertips

AI summary

Overview

Research area: Robotics, tactile sensing, multimodal perception (vision-based and audio-based soft robot fingertips), machine learning for material recognition.

Technical level: Intermediate. The paper assumes familiarity with tactile sensing hardware (camera-based and microphone-based sensors), transformer architectures, and standard classification metrics, but the core ideas are explained with enough context for a reader who knows basic machine learning and robotics.

Scope in one sentence: The paper introduces a soft microphone-based fingertip sensor called Minsound, pairs it with the camera-based Minsight sensor on a four-fingered robot hand, and shows through dynamic in-hand rubbing of 20 fabrics that internal audio is the single most useful tactile modality for fabric recognition.

What This Paper Is About

Humans identify fabrics by rubbing them between their fingers, combining low-frequency skin deformation with high-frequency vibrations. Most robot tactile sensors cannot do both, because camera-based sensors give high spatial resolution but are limited to camera frame rates, while accelerometers and microphones give high temporal resolution but are hard to embed in soft, compliant fingertips. The paper's goal is to build a soft fingertip that senses touch-generated sound from inside the silicone shell, pair it with an existing internal-camera tactile sensor, and measure how much each modality contributes to recognizing fabrics using a human-like enclosing and rubbing motion.

Key Contributions

1. Minsound: a low-cost, soft, audio-based fingertip sensor for high-frequency touch perception, using a SPH0645 MEMS microphone inside the same soft shell design as Minsight.
2. A modality study comparing the relative importance of vision-based touch, audio-based touch, and proprioceptive data for fabric recognition during dynamic in-hand exploration on a robot hand.
3. A multimodal dataset of 20 common fabrics plus a no-fabric class, containing high-resolution images, internal and external audio, and proprioception, gathered over about eight minutes of rubbing per fabric, with the full dataset available on the project website.
4. Generalization experiments showing the learned representations transfer to fabric properties (stretchiness, roughness, thickness) and to fabrics never touched during training.

Main Findings

• Internal audio dominates single modalities: Using only Minsound's internal microphone achieved 95.06 ± 1.85% test accuracy, far above Minsight images alone (41.15 ± 5.74%), optical flow alone (47.16 ± 5.12%), proprioception alone (47.10 ± 3.72%), or the external microphone alone (59.22 ± 2.43%).

• Best accuracy comes from internal audio plus external audio: The top result in the ablation table was 97.75 ± 1.27% with Minsound audio plus external audio, which the authors report as significantly higher than internal audio alone (Wilcoxon signed-rank test, p = 0.0254). The abstract states a maximum fabric classification accuracy of 97% on the 20-fabric dataset.

• Vision-based tactile images overfit on small data: Minsight images produced poor test accuracy (41.15%) despite the sensor's high spatial resolution, which the authors attribute to overfitting on a dataset with only roughly 80 samples per class. The paper notes that Minsight still matters operationally because its force calibration holds the normal force constant across fabrics.

• Bandwidth of 15 kHz is optimal: Truncating the power spectral density to different maximum frequencies gave test accuracies of 89.87 ± 3.31% at 2.5 kHz, 93.07 ± 3.33% at 10 kHz, 95.06 ± 1.85% at 15 kHz, and 91.94 ± 2.68% at 24 kHz. Exceeding the microphone's specified 15 kHz bandwidth caused overfitting.

• Transformer backbone beats a TCN: The same three-modal input (Minsight images, Minsound audio, proprioception) reached 75.41 ± 11.2% with a temporal convolutional network, versus 94.63 ± 2.78% with the transformer.

• Noise robustness requires the extra microphone: Under coffee-shop-like noise, internal audio alone fell to 67.51 ± 5.05%. Adding proprioception raised it to 87.51 ± 2.66% (p < 0.0001), and adding the external microphone raised it further to 90.30 ± 2.18% (p < 0.0098) in Table III; the body text reports the same configuration as 91.57%. Lab noise near the external microphone was 29 to 31 dB, while the coffee-shop noise was 47 to 55 dB, measured with a Sauter SU 130 sound meter.

• Properties generalize to unseen fabrics: On five holdout fabrics, stretchiness was classified at 76.15% with Minsound alone and 82.95% with Minsound + Minsight (chance 50.00%); thickness at 70.56% and 90.47% (chance 33.33%); roughness at 40.23% and 33.94% (chance 20.00%). On training fabrics, all properties were classified above 82%.

• Learned embeddings capture fabric similarity: A t-SNE plot of mean latent features shows fabrics with similar properties cluster together, and holdout fabrics embed near similar training fabrics. Notably, holdout real silk (fabric 21) embedded near a smooth synthetic fabric (fabric 4) rather than the real silk (fabric 12) in the training set, which the authors suggest may be due to different weave structures.

• Fast inference: Running the transformer over a sequence of 200 samples takes 5.58 ms on an A100 GPU.

Methodology in Plain English

The team mounted two soft fingertip sensors on a low-cost four-fingered ISyHand V2. The middle finger carries the open-source Minsight sensor, which uses an internal camera to image the deformation of an opaque silicone shell, sampled at 50 Hz to match the hand's joint command rate; images are 308 × 410 pixels and can be downsampled to 60 × 80. The thumb carries the new Minsound sensor, which is structurally identical — same dimensions, same aluminum skeleton, same overmolded silicone shell — but replaces the camera with a SPH0645 MEMS microphone mounted at the base, with a bandwidth from 50 Hz to 15 kHz and audio sampled at 48 kHz with 16-bit precision through a Raspberry Pi 4. A second identical microphone sits on the side of the palm to record ambient noise.

To collect data, the robot closes its thumb and middle finger around a folded, hanging fabric until Minsight measures a force of 0.1 N, then rubs back and forth. Each trial consists of 50 rubbing motions and yields roughly 1500 time steps. For every camera frame, the system saves the six involved joint angles, currents, and velocities (an 18-dimensional vector) and a 2048-sample window of audio from both microphones, corresponding to 42.67 ms before image capture, so consecutive audio windows overlap by about 50%. Audio is converted to a power spectral density using Welch's method with 512 frequency bins; images are also converted to dense Farneback optical flow filtered to the top 0.1 percentile of magnitude and averaged into a 2 × 102 × 137 tensor.

The classifier processes sequences of N = 200 time steps (4 seconds). Each modality goes through its own encoder: a CNN for images, and multi-layer perceptrons for audio and proprioception, with the two audio encoders sharing weights. Features are concatenated, normalized to [0, 1] along the time axis, given a position embedding, and passed through a sequential backbone of three multi-head attention layers, followed by two fully connected layers and a softmax. Image encoders use random rotations from −90 to 90 degrees as augmentation. Data came from twelve training trials and two testing trials per label, collected on different days in random order, giving 1588 training samples and 273 testing samples across 21 classes (20 fabrics plus a no-fabric class). For property perception, separate classification heads were trained jointly with equal loss weighting for two levels of stretchiness, five levels of roughness, and three levels of thickness.

Why This Matters

This paper is a proof of concept that a microphone buried inside a soft fingertip shell — the same shell already optimized for an internal camera — can carry most of the information needed to identify materials. It challenges the assumption that high spatial resolution is the key to fine texture perception in robots, and it shows a practical route to robust, cheap, high-frequency tactile sensing that survives repeated contact better than thin deformable sensor skins.

Real-world applications:

• Textile and apparel handling: automated sorting, quality inspection, or inventory tasks where a robot must distinguish silk, wool, tulle, microfiber, or knitwear by feel alone.
• Household and service robots: recognizing clothing and linens while folding laundry, where fabrics are freely hanging or folded rather than mounted on a rigid board.
• Recycling and waste sorting: identifying material types from touch when visual information is occluded, dirty, or ambiguous.
• Manipulation research platforms: providing a low-cost, open-design multimodal fingertip for studying sensor fusion in grasping and slip detection.

Industry relevance: the sensor is built from commodity components (a MEMS microphone, a Raspberry Pi 4, a silicone shell, and an aluminum skeleton), and the paper explicitly compares it against a rigid accelerometer approach that tears out of soft structures under load. That combination of low cost, robustness, and strong classification accuracy is attractive for robotics companies building soft or compliant end effectors, and for anyone collecting real-world tactile data where simulated fabric contact is not realistic.

Future Directions

• Unify the two sensors: the conclusion states the authors plan to combine vision-based and audio-based sensing into a single soft fingertip the same size as Minsight.
• Replace preprogrammed exploration: the limitations section notes the rubbing motion is preprogrammed, explores only one direction, and requires manual tuning of the force threshold and finger speed, so it may miss anisotropic fabric properties; teleoperation or more diverse motion generation is an open problem.
• Scale beyond the current fabric set: the dataset uses 20 fabrics plus a no-fabric class and three holdout fabrics, which the authors acknowledge is not representative of the variety of existing fabrics.
• Improve property generalization: roughness on holdout fabrics stayed close to chance (40.23% with Minsound, 33.94% with Minsound + Minsight, against a 20.00% chance level), leaving room for better transfer of learned fabric representations.

Target Audience

Robotics researchers working on tactile sensing, soft sensors, and multimodal perception; engineers designing end effectors for manipulation of deformable materials; machine learning practitioners interested in sensor fusion with small real-world datasets; and textile, apparel, or recycling automation teams evaluating touch-based material recognition. The paper is suited to readers with intermediate background knowledge, though the high-level findings about audio versus vision in touch sensing are accessible to a broader technical audience.

Authors’ abstract

Distinguishing the feel of smooth silk from coarse cotton is a trivial everyday task for humans. When exploring such fabrics, fingertip skin senses both spatio-temporal force patterns and texture-induced vibrations that are integrated to form a haptic representation of the explored material. It is challenging to reproduce this rich, dynamic perceptual capability in robots because tactile sensors typically cannot achieve both high spatial resolution and high temporal sampling rate. In this work, we present a system that can sense both types of haptic information, and we investigate how each type influences robotic tactile perception of fabrics. Our robotic hand's middle finger and thumb each feature a soft tactile sensor: one is the open-source Minsight sensor that uses an internal camera to measure fingertip deformation and force at 50 Hz, and the other is our new sensor Minsound that captures vibrations through an internal MEMS microphone with a bandwidth from 50 Hz to 15 kHz. Inspired by the movements humans make to evaluate fabrics, our robot actively encloses and rubs folded fabric samples between its two sensitive fingers. Our results test the influence of each sensing modality on overall classification performance, showing high utility for the audio-based sensor. Our transformer-based method achieves a maximum fabric classification accuracy of 97 % on a dataset of 20 common fabrics. Incorporating an external microphone away from Minsound increases our method's robustness in loud ambient noise conditions. To show that this audio-visual tactile sensing approach generalizes beyond the training data, we learn general representations of fabric stretchiness, thickness, and roughness.