End-to-end example-based sim-to-real RL policy transfer based on neural stylisation with application to robotic cutting

AI summary

Overview

Research area: Robotics — reinforcement learning, sim-to-real transfer, domain adaptation, and contact-rich manipulation (specifically robotic cutting/destructive disassembly).

Technical level: Advanced. The paper assumes familiarity with reinforcement learning (PPO, behavioural cloning), variational autoencoders, neural style transfer, generative adversarial networks, and statistical hypothesis testing.

Scope: The paper proposes and evaluates a neural style transfer framework that adapts a simulation-trained RL cutting policy to real-world materials using unpaired, unlabelled real-world trajectory data.

What This Paper Is About

Reinforcement learning policies for robotic cutting work well in simulation but degrade when deployed on physical robots, because simulated cutting processes differ from real ones (motor backlash, tool wear, chattering, mismatched process parameters). Collecting real-world data to fix this is expensive and slow, and in robotic cutting the real-world reward signal is unavailable. The paper's goal is to transfer an expert simulation policy to real hardware without any labelled or reward-supervised real-world data, by restyling simulated trajectories using real-world examples the robot has merely observed.

Key Contributions

1. A latent-space pairing mechanism for matching content (simulated) and style (real-world) trajectory windows that operates without paired examples and without retraining.
2. A neural style transfer framework for trajectory-level domain adaptation, repurposing style transfer from image processing into a sim-to-real policy transfer mechanism for robotic skills, requiring no labelled or reward-supervised target-domain data.
3. A VAE-based self-supervised feature representation jointly used for style transfer and for generating weakly paired source-target trajectory windows to improve physical realism of synthesised trajectories.
4. Empirical validation on robotic cutting, a contact-rich task where conventional RL pipelines are difficult to apply due to absence of a real-world reward signal, with comparisons against the unadapted Expert policy, the authors' previous GP-based behavioural cloning work (BC), CycleGAN, and CVAE-based time series translation.

Main Findings

• Faster task completion in every case study: Style transfer achieved significant reductions in task completion time relative to comparison methods across planar, depth-of-cut offset, and curved-surface case studies. In the planar case, the largest effect was −1.00 s relative to GAN and the smallest was −0.329 s relative to the Expert. In the curved case, effects were −1.00 s relative to GAN and −0.413 s relative to the Expert.
• No significant difference from BC on completion time: Style transfer did not significantly differ from BC on task completion time in any case study (planar Tukey HSD; offset p = 0.0694; curved p = 0.529), but outperformed all other methods.
• Path deviation: In the planar case, style transfer significantly differed from the Expert (−1.50 mm, p = 0.000196) and GAN (0.451 mm, p = 0.005074), but results were inconclusive against BC (p = 0.560) and CVAE (p = 0.109). In the offset case, only GAN differed significantly from BC (p = 0.0047) and Expert (p = 0.0125).
• Lower average tool load in the planar case: Style transfer significantly outperformed the Expert (−1.273 N, p = 0.0001) and BC (−0.651 N, p = 0.0352), with no significant difference versus CVAE (p = 0.867) or GAN (p = 0.611). On curved surfaces, style transfer significantly outperformed GAN only (−1.25 N, p = 0.0001).
• Material removed volume unaffected: MRV showed no conclusive difference between strategies in any case study (planar Kruskal H = 2.87, p = 0.578; offset H = 2.61, p = 0.624; curved H = 2.15, p = 0.708).
• Action similarity to the source expert: Style transfer showed large improvements over the Expert (g = 0.875) and GAN (g = 2.18) and a moderate improvement over CVAE (g = 0.575) in the planar case, with a small reduction relative to BC (g = −0.370); all post-hoc comparisons were highly significant (p ≤ 3.65 × 10⁻¹⁷). A similar pattern held in the offset and curved cases.
• Qualitative behaviour: Style transfer reduced feed rate prior to material engagement more correctly than BC, maintained more consistent depth of cut than the unadapted Expert, and suppressed transverse TCP path defects (low-frequency "random walk" on low-stiffness materials and higher-frequency "wobble" on stiff materials) seen in the Expert and GAN trials.
• Pairing coverage: Roughly 50% of real-world embedding points were matched at least once, with matched windows dispersed throughout the latent space. The authors note matched real-world embeddings clustered at intersections between simulation and real-world clusters — behaviour they compare to mode collapse in GANs.
• Offset robustness: Insetting the path by 1 mm into the material reduced the influence of strategy; path deviation was more consistent across trials for style transfer, CVAE, and GAN, implying better tolerance of path planning offsets.

Methodology in Plain English

The framework works in three stages.

Stage one — simulation. A physically-informed cutting simulation is used to train an "expert" policy with proximal policy optimisation (PPO) over 32,000 episodes with domain randomisation of material properties. Separately, a variational autoencoder (VAE) with strided convolutional encoder and decoder (3 layers, batch normalisation) is trained on a mixture of 680 on-policy and off-policy simulated trajectories. Each trajectory is chopped into overlapping windows of 100 samples (2 seconds), which also augments the training data substantially. The VAE learns a compressed latent representation of trajectory windows.

Stage two — style transfer. Trajectories are treated like images: simulated windows supply the "content" and real-world windows supply the "style". The VAE encoder is used as the feature extractor for both content and style losses, with style represented by Gram matrices of intermediate encoder layers (layer indices 2, 5, and 7) and content taken from an early layer. Because the real-world windows are unpaired and unlabelled, the authors build a pairing matrix from pairwise cosine similarity between real-world and simulated window embeddings, and pair each simulated content window with its closest real-world match. Generated windows are initialised as copies of the content windows and optimised by gradient descent with Adam for 1000 iterations at a learning rate of 0.01, with a content-to-style weight ratio of 0.02 chosen by grid search.

Stage three — policy adaptation. The translated real-world-like state windows are paired with the corresponding expert actions from the source windows to form a labelled dataset, and a target-domain policy initialised from the expert is trained on it by behavioural cloning.

Experiment. Fine-tuning used 50 episodic source-domain expert trajectories as content and 148 off-policy real-world trajectories as style. Real cutting trials were performed on a KUKA LBR iiwa R820 14 kg robot with a wrist-mounted motorised slitting saw (tooth count doubled relative to simulation to introduce further mismatch), controlled through the Fast Research Interface at 500 Hz, with force measured by an FT-AXIA 80 wrist sensor and gravity-compensated. The policy outputs translational stiffness, a depth-of-cut offset, and feed rate relative to a nominal trajectory (nominal feed rate 0.75 m/min). Five trials per material per strategy were run on polyurethane foam, cardboard, corrugated plastic, mica, and aluminium, with trials interleaved to mitigate drift. Metrics were task completion time, average path deviation, average tool load, material removed volume, and normalised dynamic time warping distance to the source expert's actions. Statistical testing used one-way ANOVA (with Shapiro-Wilk and Levene assumption checks, Box-Cox transforms where needed), Kruskal-Wallis where assumptions failed, and Tukey HSD or Dunn post-hoc tests, at α = 0.05.

Why This Matters

Impact on research: The paper pushes neural style transfer beyond expressive motion synthesis and dataset augmentation into use as a trajectory-level domain adaptation mechanism for robotic skill transfer. It offers a route to sim-to-real transfer that needs no paired data, no labels, no reward signal in the target domain, and no retraining of encoders or classifiers for new scenarios — distinguishing it from adversarial approaches such as CycleGAN, which the authors note are difficult to train due to mode collapse and vanishing gradients. The finding that MRV did not differ between strategies despite differing action profiles is an open result the authors themselves flag as surprising.

Real-world applications:

• Nuclear decommissioning, where cutting operations occur on unknown or degraded materials and reprogramming is expensive.
• Robotic disassembly of complex products for recycling and re-use, a low-volume high-mix setting with high uncertainty.
• Demolition with robotised equipment, where the precise cutting sequence may not be known in advance.
• Surgical or other contact-rich cutting tasks where the environment cannot be fully modelled.

Industry relevance: The method targets low-volume, high-mix processes where traditional high-tolerance manufacturing assumptions do not apply — tolerances of a few millimetres rather than micrometres, and unknown cutting sequences. If policies can be adapted from a simulation plus a modest set of observed real trajectories (148 off-policy trajectories here), the cost of reprogramming and reconfiguring robots for each new workpiece could fall substantially.

Future Directions

• Extending the imitation learning algorithm: The authors note the behavioural cloning step could be replaced by alternative imitation learning algorithms such as DAgger, provided the translation function can be inferred during generation of source windows.
• Sensor generality: The method is stated to be applicable to sensors other than the FT-AXIA 80 used here, including sensors built into the iiwa, provided real-world examples are collected with them.
• Unknown surface geometry: The curved-surface case study assumed the reference path relative to the surface was already known, derived from a 3-point arc estimation; the authors point to existing milling path-planning work for the case where surface geometry is unknown.
• Poorly matched pairings: Windows whose pairings diverged substantially from the content introduced mean shifts and encoding artefacts, and roughly half of real-world embeddings were never matched. Improving pairing coverage and reducing this parasitic matching behaviour is an unresolved issue.
• Reward-free limits: Because real-world reward information is unavailable, the framework can only reproduce expert-like behaviour; whether genuine improvement beyond the simulation expert is achievable in the real domain is left open.

Target Audience

Robotics and machine learning researchers working on sim-to-real transfer, domain adaptation, and reinforcement learning for contact-rich manipulation. It is also relevant to engineers deploying learning-based control in disassembly, decommissioning, or other low-volume high-mix industrial settings, and to readers interested in creative reuse of generative modelling techniques (VAEs, style transfer) for non-vision domains.

Authors’ abstract

Whereas reinforcement learning has been applied with success to a range of robotic control problems in complex, uncertain environments, reliance on extensive data - typically sourced from simulation environments - limits real-world deployment due to the domain gap between simulated and physical systems, coupled with limited real-world sample availability. We propose a novel method for sim-to-real transfer of reinforcement learning policies, based on a reinterpretation of neural style transfer from image processing to synthesise novel training data from unpaired unlabelled real world datasets. We employ a variational autoencoder to jointly learn self-supervised feature representations for style transfer and generate weakly paired source-target trajectories to improve physical realism of synthesised trajectories. We demonstrate the application of our approach based on the case study of robot cutting of unknown materials. Compared to baseline methods, including our previous work, CycleGAN, and conditional variational autoencoder-based time series translation, our approach achieves improved task completion time and behavioural stability with minimal real-world data. Our framework demonstrates robustness to geometric and material variation, and highlights the feasibility of policy adaptation in challenging contact-rich tasks where real-world reward information is unavailable.