Semi-Supervised Cross-Domain Imitation Learning

AI summary

Overview

Research area: Machine learning — specifically imitation learning, offline reinforcement learning, and cross-domain transfer (the intersection of distribution correction estimation, domain adaptation, and behavior cloning).

Technical level: Advanced. The paper builds on MDP occupancy measures, convex duality, regularized distribution matching (DemoDICE), Bellman flow constraints, normalizing flows, and a convergence proof for a non-convex-style density-ratio estimator. The prose is readable, but the machinery assumes familiarity with offline RL and density-ratio estimation.

One-sentence scope: The paper defines a new "semi-supervised cross-domain imitation learning" (SS-CDIL) setting and proposes AdaptDICE, an offline algorithm that transfers source-domain knowledge to a target domain using only a small number of labeled target expert trajectories plus unlabeled imperfect target trajectories.

What This Paper Is About

Imitation learning normally assumes the expert demonstrations and the environment you want to act in share the same domain, but in practice you often have plenty of cheap simulated data and almost none of the expensive real-world data. Cross-domain imitation learning tries to bridge that gap, yet existing methods either need costly paired or proxy supervision, or they go fully unsupervised and rely on strong assumptions such as domain isomorphism — which can produce ambiguous alignments and unstable transfer. This paper introduces a middle ground, semi-supervised CDIL, and gives the first algorithm for it with theoretical justification: use source-domain (imperfect and expert) data as a prior, add just a handful of labeled target expert trajectories plus unlabeled target trajectories, and learn a stable target-domain policy entirely offline.

Key Contributions

1. A new problem setting, Semi-Supervised CDIL (SS-CDIL). The paper formalizes a setting where the imitator has a scarce target-domain expert dataset (as few as one trajectory) together with an imperfect target-domain dataset of unknown optimality, plus an auxiliary source-domain dataset — with no proxy tasks and no paired trajectories across domains.

2. AdaptDICE, the first algorithmic framework for SS-CDIL. The algorithm extends single-domain distribution correction estimation (DICE) to the cross-domain case, and is designed to handle discrepancies in transition dynamics, state space, and action space.

3. A novel cross-domain mapping loss and a hybrid density ratio. Mapping functions G: S_tar → S_src and H: S_tar × A_tar → A_src are learned by minimizing a Bellman-consistency error evaluated under the pre-trained source critic, so that source-domain density ratios can be transferred to target-domain state-action pairs. The target policy is then extracted by weighted behavior cloning using a hybrid (cross-domain) density ratio that mixes the mapped source ratio and the learned target ratio.

4. An adaptive weighting scheme with a convergence guarantee. The mixing weight β(t) is set as a smooth function of the relative estimation errors of the two density ratios, and the paper proves an upper bound on the cross-domain density ratio error that decays polynomially and is no worse than the better of the two domain estimators.

Main Findings

• Consistent empirical gains: Experiments on MuJoCo and Robosuite show AdaptDICE achieving consistently better performance than the baselines, including in challenging offline cross-domain scenarios with as few as only 1 labeled target-domain expert trajectory.

• Theoretical convergence guarantee: Theorem 1 bounds the pointwise error |w_cross^(t)(s,a) − w*_tar(s,a)| by β(t)Δw_src^(t)(s,a) plus a target-domain term that decays as 1/√t (containing ||ν_tar^(0) − ν*_tar||₂ / √t), holding for learning rates η ≤ 1/L_f, where L_f is the smoothness constant of L_DICE.

• Error bounded by the better domain: With the hard-switching rule β(t) = 0 if Δw̄_tar^(t) ≤ Δw̄_src^(t) and β(t) = 1 otherwise, the expected cross-domain density ratio error satisfies a bound of min(Δw̄_src^(t), Δw̄_tar^(t)). This means AdaptDICE cannot do worse — in expectation — than whichever of the source or target estimators is currently more accurate.

• Smooth weighting is preferred in practice: The paper argues that the hard switch above can cause oscillations when the two error estimates are comparable, so the implementation uses a continuous β(t) that interpolates inversely with the two expected errors: when the source ratio is more accurate, β(t) > 0.5 and rises toward 1; when the target ratio is more accurate, β(t) falls toward 0. This smooth transition is stated not to alter the convergence rate, but to improve stability.

• Ablation on hybrid density ratios: The paper reports an ablation study demonstrating the benefits of the hybrid density ratio for effective cross-domain transfer.

• Concrete outcome unstated: The truncated paper content does not report specific numerical scores, success rates, or per-task numbers; only that gains are consistent across the MuJoCo and Robosuite benchmark tasks.

Methodology in Plain English

The starting point is a well-known offline imitation learning trick: instead of directly optimizing a policy, you learn how much more often the expert visits each state-action pair than an imperfect dataset does, a quantity called a density ratio. That ratio then acts as an importance weight in behavior cloning, nudging the policy toward what the expert did.

AdaptDICE takes this idea and runs it across two domains at once. First, before any training, the source domain is used to pre-train two things offline: a pseudo Q-function (a critic) and a source density ratio, using an existing method, DemoDICE. In the target domain, a discriminator is trained to tell target expert state-action pairs apart from the full target dataset; its output is converted into a pseudo reward that approximates the target log density ratio.

Inside the training loop, three things get updated repeatedly. Mapping networks G and H translate target states and actions into the source domain, and they are trained so that the source critic's values satisfy Bellman consistency on the mapped, target-data transitions. Because the source and target state and action spaces may genuinely differ in shape, the outputs of G and H are passed through a normalizing flow that is pre-trained offline on source-domain feasible samples and then frozen, so mappings always land in a valid source region. Meanwhile, the target pseudo value function ν_tar is updated by gradient descent on the DICE loss, which is the standard regularized distribution-matching objective using the target pseudo reward.

Policy learning then combines the two worlds: a cross-domain density ratio is formed as a weighted average of the source ratio evaluated on mapped state-action pairs and the target ratio evaluated on the original pairs. The weight β(t) is recomputed each iteration from the empirical estimation errors of the two ratios, so the algorithm automatically leans on whichever estimator is currently more trustworthy. Finally, the policy is updated by weighted behavior cloning with this hybrid ratio. Everything is offline — no online interaction with either environment is required.

Why This Matters

Impact on research. The paper carves out a middle setting between fully supervised CDIL (accurate but data-hungry) and unsupervised CDIL (cheap but assumption-laden and unstable), and supplies both an algorithm and a convergence bound for it. It also shows how DICE-style density-ratio estimation, previously single-domain, can be extended across domains by learning mappings that preserve Bellman consistency — a template that could be reused beyond imitation learning. The adaptive mixing rule offers a principled alternative to hand-tuned domain weights.

Real-world applications (from the motivating examples and setting):

• Robot control, where simulated or proxy demonstrations are abundant but physical demonstrations are expensive and risky to collect.
• Autonomous driving, where target-domain expert data (real road driving) is costly and hazardous while simulation is cheap.
• Visual navigation, where agents must transfer navigation skills between environments with different dynamics and observation spaces.
• Any setting with a dynamics mismatch between a training simulator and the deployed system, where only a handful of expert runs on the real system are affordable.

Industry relevance. The data-efficiency claim is the practical hook: with as few as only 1 labeled target-domain expert trajectory, the method still reports gains. That matters for organizations that can simulate at scale but cannot label real demonstrations at scale — robotics fleets, autonomous vehicle programs, and simulated-to-real deployment pipelines. The release of code at https://github.com/NYCU-RL-Bandits-Lab/CDIL and the OpenReview record (https://openreview.net/forum?id=WARXnbJawZ) lower the barrier to reproducing and building on the method.

Future Directions

• Beyond the tabular analysis. The convergence guarantee is derived in the tabular case; extending the bound to the function-approximation regime used in the experiments (which relies on normalizing flows and neural networks) remains open.
• Relaxing the offline constraint. The paper deliberately restricts SS-CDIL to offline data; whether a small online interaction budget in the target domain could be combined with the adaptive weighting to improve transfer is not addressed.
• Stress-testing the mapping assumption. The method learns mappings from target to source state-action spaces; how it behaves when no reasonable mapping exists, or when the flow pre-training data poorly covers the target region, is not characterized in the content available.
• Sensitivity to the unlabeled data. The setting assumes imperfect target trajectories are readily available, but the paper content does not report how performance degrades as the quality or quantity of these unlabeled trajectories varies.

Target Audience

Researchers and graduate students working on imitation learning, offline reinforcement learning, and domain adaptation or transfer learning, who want either a new problem formulation to build on or a concrete algorithmic recipe for cross-domain policy learning under minimal supervision. Practitioners in robotics and autonomous systems with a simulator-to-reality gap and very limited expert data will also find the setting and the as-few-as-one-trajectory claim most relevant, though they should expect to engage with the density-ratio and duality machinery to implement it.

Authors’ abstract

Cross-domain imitation learning (CDIL) accelerates policy learning by transferring expert knowledge across domains, which is valuable in applications where the collection of expert data is costly. Existing methods are either supervised, relying on proxy tasks and explicit alignment, or unsupervised, aligning distributions without paired data, but often unstable. We introduce the Semi-Supervised CDIL (SS-CDIL) setting and propose the first algorithm for SS-CDIL with theoretical justification. Our method uses only offline data, including a small number of target expert demonstrations and some unlabeled imperfect trajectories. To handle domain discrepancy, we propose a novel cross-domain loss function for learning inter-domain state-action mappings and design an adaptive weight function to balance the source and target knowledge. Experiments on MuJoCo and Robosuite show consistent gains over the baselines, demonstrating that our approach achieves stable and data-efficient policy learning with minimal supervision. Our code is available at~ https://github.com/NYCU-RL-Bandits-Lab/CDIL.