Tactile Memory with Soft Robot: Robust Object Insertion via Masked Encoding and Soft Wrist

AI summary

Overview

Research area: Robotics — contact-rich manipulation, tactile sensing, soft robotics, and representation learning for robot control.

Technical level: Advanced. The paper assumes familiarity with Transformer architectures, masked token prediction, retrieval-based (non-parametric) control, tactile sensor arrays, and compliance in robot hardware.

Scope: The paper presents TaMeSo-bot, a peg-in-hole insertion system that pairs a mechanically compliant soft wrist with a database of tactile experiences encoded by a masked bidirectional Transformer (MAT³), and evaluates it in real-robot experiments across seven peg shapes and several perturbation conditions.

What This Paper Is About

Robots struggle with contact-rich tasks like inserting a peg into a hole when the exact position, object shape, or friction conditions are uncertain. This paper builds a "tactile memory" system that stores past teleoperated insertion demonstrations as encoded tactile representations and, at execution time, retrieves the most similar past experience and replays its action, so the robot adapts to unseen pegs and conditions without being trained on them. The key technical claim is that a masked bidirectional Transformer, trained to reconstruct missing sensor tokens, learns tactile representations that support this retrieval better than prior tactile encoders or a parametric policy trained on the same data.

Key Contributions

1. TaMeSo-bot, a system that combines soft robotics (a soft wrist with three parallel coil springs providing 6D deformation) with a tactile memory database, enabling safe contact exploration during data collection and robust retrieve-and-replay execution for contact-rich peg-in-hole manipulation.
2. MAT³ (Masked Tactile Trajectory Transformer), a masked bidirectional Transformer encoder that jointly models spatiotemporal interactions across distributed taxels, robot actions, force-torque measurements, and proprioceptive signals via masked token prediction, without requiring explicit subtask segmentation or manual annotation.
3. A non-parametric control policy that stores encoded tactile-action pairs in a database and retrieves the k most similar representations by L2 distance using an HNSW graph-based approximate nearest neighbor index, running at 50 Hz, with one neighbor sampled uniformly from the retrieved set.
4. An extensive real-robot evaluation on seven 3D-printed peg shapes (two seen, five unseen) and three unseen perturbation conditions, comparing MAT³ against a Tactile Transformer baseline, a no-masking ablation, and the Action Chunking Transformer (ACT).

Main Findings

• Seen pegs: MAT³ reached 88.8% total success (71/80) and MAT³ without masking reached 90% (72/80), while the Tactile Transformer reached 22.5% (18/80) and ACT reached 63.75% (51/80). Per shape: square 35/40 for MAT³, 36/40 for the no-mask variant, 8/40 for the Tactile Transformer, 13/40 for ACT; cylinder (⌀40) 36/40, 36/40, 10/40, and 38/40 respectively.

• Unseen pegs: MAT³ achieved 85% (170/200), versus 71% (142/200) for MAT³ without masking, 17.5% (35/200) for the Tactile Transformer, and 53% (106/200) for ACT. The gap between MAT³ and its no-mask ablation on unseen shapes (85% vs. 71%) is the paper's main evidence that masked modeling improves generalization.

• Unseen conditions: Across unseen starting positions, increased friction, and a 5° tilted grasp, MAT³ maintained 57.5% (69/120) success, compared with 29.2% (35/120) for the no-mask variant, 14.17% (17/120) for ACT, and 7.5% (9/120) for the Tactile Transformer. Breakdown: 20/40, 20/40, and 29/40 for MAT³ versus 9/40, 7/40, and 19/40 for the no-mask variant.

• Smoother retrieved actions: In the per-timestep action-change comparison, MAT³ produced lower values than the Tactile Transformer across most shapes. For total seen pegs, MAT³ showed 0.2 ± 0.1 degrees rotation change and 1.7 ± 1.0 mm position change versus 0.3 ± 0.2 degrees and 2.0 ± 1.0 mm for the Tactile Transformer; for total unseen pegs, 0.2 ± 0.1 degrees and 1.6 ± 0.7 mm versus 0.3 ± 0.1 degrees and 2.1 ± 0.9 mm.

• Failure modes differ between paradigms: The dominant ACT failure mode is applying downward force before proper alignment, triggering protective stops. The authors hypothesize that retrieval constrains actions to the demonstration dataset, providing implicit phase-consistent regularization, whereas parametric policies can extrapolate beyond that distribution.

• Embedding space encodes subtask structure: With a database built from 80% of demonstrations and queries drawn from the remaining 20%, t-SNE visualizations overlaid with distance colormaps for the subtasks fit, align, and insert show connected yet distinct regions, indicating that MAT³ encodes subtask-related structure despite never being given subtask labels.

• Other reported failure cases: Incorrect movement direction and partial insertion, where the peg slips past the hole or gets caught, causing gradual deviation of the gripper pose.

Methodology in Plain English

The team mounted a 3×3 PapillArray tactile sensor (nine taxels, each measuring a 3D force) on one finger of a parallel gripper, attached that gripper to a robot arm through a soft wrist made of three parallel coil springs, and tracked gripper pose with a motion capture tracker. They teleoperated the robot to perform peg insertions, collecting 64 successful demonstrations (32 on a square peg, 32 on a cylinder peg), each lasting roughly 150–300 timesteps at 50 Hz, starting from one of four poses around the hole. Peg and hole tolerances were 2 mm.

Each moment of data becomes a set of tokens: nine tactile tokens, one action token (3D position displacement plus Euler rotations), and auxiliary tokens for force-torque (6D), arm pose (7D), and gripper pose (7D). The tactile tokens get 2D sinusoidal position embeddings tied to their grid location, the action token sits at the grid center, and auxiliary signals are added into every base token as a weighted sum. A 4-layer bidirectional Transformer with 8 attention heads and hidden dimension 512 processes a window of H timesteps. During training, between 0% and 60% of tokens are randomly masked and the model must reconstruct them, with an MSE loss split into tactile and action terms and optimized with Adam for 30 epochs at batch size 128 and learning rate 1.0 × 10⁻⁴. The encoder uses 248-dimensional token embeddings plus 8-dimensional spatial embeddings, giving 256-dimensional inputs; the pair (256, 512) was selected from candidate embedding/hidden-size pairs because it gave the best action retrieval accuracy.

After training, every demonstration window is turned into a single vector by average pooling and stored alongside its action, forming a database. At run time, the current sensor window is encoded with the current action token masked out (since it is unknown), the k nearest stored vectors are found by L2 distance using an HNSW index, and one is sampled uniformly for execution. The wrist's mechanical compliance absorbs contact forces, which both prevents protective stops during data collection and protects the tactile sensor during insertion. Baselines are a Tactile Transformer adapted from the Visuo-Tactile Transformer (with learned 1D position embeddings and separate tokens per modality), the same MAT³ architecture trained without masking, and ACT as a parametric policy trained on the same 64 demonstrations.

Why This Matters

Impact on research: The paper argues that tactile representation learning methods typically encode space and time separately (for example CNNs or ViT-style Transformers plus LSTMs) or fuse modalities by simple concatenation, missing joint spatiotemporal correlations between distributed tactile signals and concurrent actions. It offers masked bidirectional modeling with tactile-centric weighted soft-concatenation as a lightweight alternative to retrieval frameworks that need learned alignment modules or residual policies, and it supplies a controlled comparison of retrieval-based versus parametric policies on identical demonstration data.

Real-world applications (as implied by the task setting and stated future work):

• Robotic assembly and insertion of parts where clearances are tight and the exact pose of a part is uncertain.
• Contact-rich finishing or handling tasks such as grinding, scooping, and wiping, which the authors name as future extensions.
• Insertion tasks with deformable or variable-geometry objects, since the system generalizes to unseen peg shapes such as oval, hexagon, and pentagon without per-task parameter tuning.
• Tasks requiring safe physical interaction, where a compliant wrist prevents the protective stops that rigid arms trigger during contact.

Industry relevance: The approach needs no manual annotation of subtask phases such as fit, align, and insert, and no per-task tuning, which reduces engineering effort for deploying insertion skills. It uses off-the-shelf components named in the paper (UR5e arm, Robotiq Hand-e gripper, PapillArray tactile sensor, HTC VIVE tracker), and the authors note that replacing the external motion tracker with visual pose estimation would simplify setup and improve reproducibility.

Future Directions

• Extending the framework to other contact-rich tasks beyond peg-in-hole, specifically grinding, scooping, and wiping.
• Addressing the fundamental limit that retrieval cannot extrapolate beyond the stored demonstration distribution, for example by combining retrieval with parametric policy learning.
• Leveraging large-scale open datasets for tactile pretraining to enrich the representation beyond the 64 collected demonstrations.
• Replacing the external motion capture tracker with visual pose estimation to simplify the system and improve reproducibility.
• Investigating the two leading failure modes further: incorrect movement direction and partial insertion, where the peg slips past the hole or becomes caught.

Target Audience

Researchers and engineers working on robot manipulation with tactile sensing, contact-rich assembly, soft robotics, and imitation- or retrieval-based learning. It is most useful to readers already comfortable with Transformer architectures, masked modeling, and real-robot experimental methodology. Practitioners focused on industrial insertion or assembly automation will find the baseline comparisons and the practical hardware description relevant, while those new to the area would need background reading on tactile representation learning and retrieval-based control first.

Authors’ abstract

Tactile memory, the ability to store and retrieve touch-based experience, is critical for contact-rich tasks such as key insertion under uncertainty. To replicate this capability, we introduce Tactile Memory with Soft Robot (TaMeSo-bot), a system that integrates a soft wrist with tactile retrieval-based control to enable safe and robust manipulation. The soft wrist allows safe contact exploration during data collection, while tactile memory reuses past demonstrations via retrieval for flexible adaptation to unseen scenarios. The core of this system is the Masked Tactile Trajectory Transformer (MAT$^\text{3}$), which jointly models spatiotemporal interactions between robot actions, distributed tactile cues, force-torque measurements, and proprioceptive signals. Through masked token prediction, MAT$^\text{3}$ learns rich spatiotemporal representations by inferring missing sensory information from context, autonomously extracting task-relevant features without explicit subtask segmentation. We validate our approach on peg-in-hole tasks with diverse pegs and conditions in real-robot experiments. Our extensive evaluation demonstrates that MAT$^\text{3}$ achieves higher success rates than the baselines over all conditions and shows remarkable capability to adapt to unseen pegs and conditions.