Cross-Domain Policy Optimization via Bellman Consistency and Hybrid Critics

AI summary

Overview

• Research area: Reinforcement learning, specifically cross-domain reinforcement learning (CDRL) and transfer learning for sequential decision making.
• Technical level: Advanced. The paper combines tabular policy-gradient convergence analysis with a deep RL implementation built on soft actor critic and normalizing flows.
• Scope: The paper proposes a framework called Q-Avatar that transfers knowledge from a pre-trained source-domain Q function to a target domain with different state and action spaces, using a notion the authors call cross-domain Bellman consistency and a weighted "hybrid critic."

What This Paper Is About

Cross-domain RL tries to make learning in a target domain cheaper by reusing data or models from a similar source domain, such as a simulator. Two problems get in the way: the two domains may have different state and action spaces (so a model cannot be copied directly), and it is hard to know in advance whether the source model will actually help rather than hurt target-domain performance. The paper's goal is a transfer method that delivers the sample-efficiency benefit when the source model is useful and falls back safely to ordinary target-domain learning when it is not.

Key Contributions

1. The Q-Avatar framework. A CDRL method that transfers knowledge between two domains with distinct state and action spaces by combining source-domain and target-domain Q functions with an adaptive, hyperparameter-free weight function.
2. A tabular prototype with convergence theory. The authors first present Direct Q Transfer (DQT) and then a prototypical Q-Avatar algorithm, and establish average sub-optimality bounds for Q-NPG, DQT, and Q-Avatar under the tabular and approximate-Q settings plus an exploratory initial-distribution assumption.
3. A practical deep RL implementation. Q-Avatar is extended via the connection between natural policy gradient and soft policy iteration, integrated with soft actor critic (SAC), and combined with a normalizing flow model that keeps the outputs of the state and action mapping functions inside feasible regions.
4. Experiments and an ablation study. The authors report that Q-Avatar outperforms CDRL benchmark algorithms on locomotion, robot arm manipulation, and goal navigation tasks. Details of the ablation study are not included in the available content.

Main Findings

• Cross-domain Bellman consistency measures transferability. A source critic is defined as δ-Bellman-consistent under a target policy if there exist inter-domain mappings φ and ψ such that the cross-domain Bellman error under the target-domain visitation distribution is at most δ. Smaller δ means a better-transferring source model; the sub-optimality bound scales with δ.
• DQT alone is fragile. Under DQT, the average sub-optimality bound contains a term proportional to the cross-domain Bellman error. If the source critic transfers poorly to the target domain, that term dominates and transfer fails.
• The hybrid critic bounds the damage. The Q-Avatar bound is a weighted sum of the cross-domain Bellman error and the ordinary target-domain TD error. When the cross-domain error is small (for example, zero under an ideal mapping), the weight α(t) goes to one and the transfer term vanishes; when the source critic is only δ-Bellman-consistent with large δ, α(t) is approximately zero and the bound reduces to the standard TD error.
• The weighting rule is hyperparameter-free. Instead of a noisy indicator function, the paper uses α(t) equal to the target TD error divided by the sum of the cross-domain Bellman error and the target TD error. The authors state this design is hyperparameter-free and incurs minimal deployment overhead.
• Data efficiency improves over learning from scratch. The paper reports that Q-Avatar improves data efficiency over SAC throughout training in all tasks, and that on the reported threshold metric Q-Avatar requires about 44% of the environment steps that SAC needs in the best case.
• Benchmark comparisons. CAT-SAC achieves moderate results on MuJoCo but transfers slowly to other tasks, which the authors attribute to reliance on parameter-based transfer that assumes shared feature representations. Fine-tuning improves data efficiency over SAC on MuJoCo but is slow in Robosuite because of dissimilar state-action representations across robot arms. CMD generally performs poorly and is unstable (for example, in Ant), which the authors attribute to its unsupervised, adversarial mapping module that ignores target-domain rewards. The visible content does not report detailed results for PAR, and it does not report numerical values from the threshold table beyond the 44% figure.
• Experimental protocol. All reported results are averaged over 5 random seeds, and all methods use the same source-domain models (policy and corresponding Q-networks pre-trained with SAC) for a fair comparison.

Methodology in Plain English

The authors start from a standard policy-gradient view of RL: alternately estimate a Q function for the current policy and improve the policy in its direction. They write down how far the resulting policy can be from optimal, and split that gap into a learning term that shrinks with more iterations and an approximation term that reflects how inaccurate the Q function is. With limited target-domain data, the approximation term is the problem.

Their first idea, DQT, is to replace the target-domain Q function with a pre-trained source-domain Q function, after passing target states and actions through learned mapping functions φ (state) and ψ (action) so that the source Q function can be evaluated on them. This works only if the source Q function actually matches the target-domain transition and reward structure. The paper quantifies that match with the cross-domain Bellman error: the residual of the target Bellman equation when the source Q function is used instead. A small residual means the source model is transferable.

The second idea, Q-Avatar, keeps both critics and mixes them. Each iteration: collect on-policy target samples, fit a target-domain Q function by the usual TD loss, fit the mapping functions φ and ψ by minimizing the cross-domain Bellman loss, compute the weight α(t) from the two measured errors, and update the policy with a natural-policy-gradient-style step using the weighted combination. The weight automatically favors whichever critic currently has the smaller error, so a bad source model is effectively switched off rather than trusted.

For the practical version, the authors move past the tabular setting by exploiting the known connection between soft policy iteration and natural policy gradient, and build on SAC. Because state and action spaces in practice are bounded, the mapping functions could otherwise output infeasible values, so a normalizing flow is trained to map their outputs into the feasible regions, following prior work in action-constrained RL. This choice replaces the adversarial learning commonly used for inter-domain mappings, which the authors describe as unstable.

Why This Matters

• Research impact. The paper reframes transferability in CDRL as a measurable quantity (cross-domain Bellman consistency) rather than an assumption, and provides sub-optimality bounds for a transfer scheme that degrades gracefully. It also shows that a reward-aware cross-domain Bellman loss can be used to learn inter-domain mappings, and the appendix reportedly gives a toy example where cycle consistency fails but the Bellman-like loss learns a better mapping.
• Sim-to-real robot control. A controller trained in a simulator could be reused on a differently shaped physical robot without retraining from scratch.
• Cross-embodiment manipulation. The paper's robot arm tasks transfer from a Panda arm to a UR5e arm on door opening and table wiping, which mirrors reusing skills across different hardware.
• Cross-platform navigation. The CarGoal0 to DoggoGoal0 transfer in Safety-Gym illustrates moving a learned goal-reaching behavior from one robotic platform to another.
• Low-data industrial settings. In any setting where target-domain interaction is expensive but a related simulator or older system is available, the hybrid critic offers a way to use the old model without risking worse performance than learning from scratch.
• Industry relevance. The method targets the common practical situation where a company has a well-trained model for one environment and wants to deploy to a new one with different observation and action dimensions.

Future Directions

• Scaling the guarantees beyond tabular settings. The convergence results are established for the tabular and approximate-Q settings; extending them to the function-approximation regime used in the practical implementation is left open by the analysis as presented.
• Better mapping learning. The paper notes that identification issues and unstable adversarial training motivate the normalizing-flow approach; improving inter-domain mappings, especially when the Bellman loss is ambiguous, remains an open problem.
• Beyond δ-Bellman-consistency for transferability estimation. The paper's transferability measure is defined through the existence of good mappings; how to estimate it reliably and cheaply before committing to transfer is a natural next question.
• Broadening the benchmark coverage. The visible results cover locomotion, robot arm manipulation, and goal navigation; further evaluation across more domain pairs with differing reward functions and dynamics would test how general the hybrid-critic behavior is.

Target Audience

Researchers and practitioners in reinforcement learning and transfer learning who are comfortable with policy gradient methods, Bellman equations, and convergence analysis. It suits readers interested in transfer across differing state-action spaces, in provable transfer guarantees, or in building practical systems on top of SAC where a pre-trained model from a related domain is available. The theoretical sections assume familiarity with occupancy measures, TD error, and natural policy gradient bounds, so the paper is not aimed at newcomers.

Authors’ abstract

Cross-domain reinforcement learning (CDRL) is meant to improve the data efficiency of RL by leveraging the data samples collected from a source domain to facilitate the learning in a similar target domain. Despite its potential, cross-domain transfer in RL is known to have two fundamental and intertwined challenges: (i) The source and target domains can have distinct state space or action space, and this makes direct transfer infeasible and thereby requires more sophisticated inter-domain mappings; (ii) The transferability of a source-domain model in RL is not easily identifiable a priori, and hence CDRL can be prone to negative effect during transfer. In this paper, we propose to jointly tackle these two challenges through the lens of \textit{cross-domain Bellman consistency} and \textit{hybrid critic}. Specifically, we first introduce the notion of cross-domain Bellman consistency as a way to measure transferability of a source-domain model. Then, we propose $Q$Avatar, which combines the Q functions from both the source and target domains with an adaptive hyperparameter-free weight function. Through this design, we characterize the convergence behavior of $Q$Avatar and show that $Q$Avatar achieves reliable transfer in the sense that it effectively leverages a source-domain Q function for knowledge transfer to the target domain. Through experiments, we demonstrate that $Q$Avatar achieves favorable transferability across various RL benchmark tasks, including locomotion and robot arm manipulation. Our code is available at https://rl-bandits-lab.github.io/Cross-Domain-RL/.