Offline Reinforcement Learning of High-Quality Behaviors Under Robust Style Alignment

AI summary

Overview

• Research area: Offline reinforcement learning, style-conditioned policy learning, and diverse behavior generation from pre-recorded datasets.
• Technical level: Advanced. The paper builds directly on Implicit Q-Learning (IQL), expectile regression, advantage-weighted regression, and goal-conditioned RL terminology.
• One-sentence scope: The paper proposes a general definition of behavioral style based on labeling functions over subtrajectories, and instantiates it in SCIQL, an offline RL algorithm that combines style-conditioned value learning with a Gated Advantage Weighted Regression mechanism to improve task performance while preserving style alignment.

What This Paper Is About

Most sequential decision-making research optimizes task performance alone, but many applications need agents that perform a task in a specific way, such as running rather than crawling, or drawing a circle clockwise rather than counter-clockwise. When behavior must be learned purely from a fixed dataset of pre-recorded trajectories, aligning an agent's style with a target style is hard because of distribution shift and because style and task reward often conflict. The paper introduces a unified way to define style and an offline RL method, SCIQL, designed to achieve both high style alignment and high task performance without any further environment interaction.

Key Contributions

1. A general view of stylized policy learning as a generalization of goal-conditioned RL. A style is defined as the labeling of a subtrajectory given a comparison criterion relative to a task, so a style is a particular way of carrying out a task under that criterion. This reframes alignment as a goal-reaching-like problem.
2. An instantiation using data programming with labeling functions on subtrajectory windows. Following the supervised setting of CTVAE (Zhan et al., 2020) and BCPMI (Yang et al., 2025), but applying hard-coded labeling functions to windows of radius w(λ) rather than full trajectories, which captures the multi-timescale nature of styles, reduces labeling cost, improves interpretability, and mitigates credit assignment issues.
3. Style-Conditioned Implicit Q-Learning (SCIQL). An IQL-inspired offline RL algorithm that learns style-conditioned value functions with expectile regression and extracts a policy through advantage-weighted regression, using style relabeling and trajectory stitching to handle state-style distribution shift.
4. Gated Advantage Weighted Regression (GAWR). A policy improvement rule that combines style and task advantages through a gating function, ξ(A^λ, A^r)(s,a,z) = A^λ(s,a,z) + σ(A^λ(s,a,z)) · A^r(s,a), so that task optimization signals are applied mainly when they are unlikely to harm style alignment.

Main Findings

• Style alignment improvements (Table 1): Across all seven reported datasets, SCIQL attains the highest style alignment scores among the compared methods. The numbers below are averages over criteria, labels, and 5 seeds, reported as mean ± standard deviation.
• Circle2d, in-place: BC 29.1 ± 6.3, CBC 58.6 ± 2.3, BCPMI 58.9 ± 2.6, SORL (β=0) 58.9 ± 2.7, SCBC 68.6 ± 2.0, SCIQL 74.6 ± 9.3.
• Circle2d, navigate: BC 29.1 ± 5.3, CBC 58.9 ± 2.7, BCPMI 59.9 ± 2.3, SORL (β=0) 60.0 ± 3.3, SCBC 67.2 ± 1.8, SCIQL 75.5 ± 4.7.
• HalfCheetah, fixed policy: BC 30.0 ± 5.9, CBC 51.2 ± 9.0, BCPMI 58.1 ± 8.4, SORL (β=0) 53.1 ± 10.6, SCBC 58.0 ± 5.3, SCIQL 78.0 ± 1.8.
• HalfCheetah, stitched segments: BC 30.0 ± 6.8, CBC 52.1 ± 7.6, BCPMI 58.9 ± 11.3, SORL (β=0) 48.4 ± 12.5, SCBC 57.4 ± 4.7, SCIQL 78.0 ± 1.1.
• HalfCheetah, varying policy: BC 30.0 ± 4.5, CBC 52.0 ± 12.0, BCPMI 52.6 ± 17.2, SORL (β=0) 46.7 ± 9.5, SCBC 31.7 ± 4.2, SCIQL 78.9 ± 0.7.
• HumEnv simple: BC 50.0 ± 44.4, CBC 89.1 ± 22.0, BCPMI 79.2 ± 26.7, SORL (β=0) 79.4 ± 26.9, SCBC 99.6 ± 0.0, SCIQL 99.6 ± 0.0. Here SCIQL and SCBC are reported at the same value.
• HumEnv complex: BC 33.3 ± 4.0, CBC 47.1 ± 12.8, BCPMI 44.6 ± 18.4, SORL (β=0) 47.7 ± 6.9, SCBC 33.2 ± 3.5, SCIQL 83.5 ± 6.2.
• A design choice that mattered: Among the strategies compared for the style-reward estimator χ, the paper reports (Appendix E.1) that using the indicator χ^λ(s,a,z) = 1{z = z_t^λ} based on the associated window label was among the best performing and was kept for simplicity. Sampling style labels outside the joint distribution p^(λ(D))(s,a,z) is what the authors describe as addressing distribution shift.
• Task-performance results are not reported in the provided text. The experimental section asks whether GAWR helps style-conditioned task-performance optimization and how SCIQL compares to prior work on that objective, but the truncated content ends during the environment descriptions, so no task-performance table, values, or dataset sizes are available here.
• Source of the environment design: Figure 1 is described as illustrating that achieving movement styles such as high-speed running may require standing up and accelerating, and that datasets may not cover all speed styles, requiring trajectory stitching such as slow → medium and medium → fast.

Methodology in Plain English

The authors start by asking what a "style" actually is. Their answer: a style is a label attached to a short window of a trajectory, produced by a simple hand-written rule (a labeling function) rather than by manual annotation or an unsupervised encoder. Each state-action pair in the dataset is marked with the label of the window it belongs to, so styles can change within a single trajectory, and users can write their own rules for training and evaluation.

Because the true style objective depends on a window of past and future states, it is not Markovian and cannot be optimized directly. The authors replace it with a surrogate: a learned label predictor that estimates the probability that a state-action pair sits at the center of a window labeled z, which turns style alignment into an ordinary Markov reward function defined on the dataset.

SCIQL then adapts IQL to this setting. It learns style-conditioned value functions Q and V using expectile regression, plus separate task value functions for the environment reward. The policy is extracted by advantage-weighted regression, where actions are copied with a bias toward those with high advantage. Crucially, style labels are resampled independently of the states they are paired with, which exposes the value functions and policy to state-style combinations that never co-occur in the data and improves robustness.

To avoid letting task optimization destroy style alignment, the authors do not simply add style and task advantages together. Instead, GAWR passes the task advantage through a sigmoid gate driven by the style advantage: when the style advantage is high, the task signal is allowed through strongly; when it is low or negative, the task signal is suppressed. Training is organized in three loops (task value functions, style value functions, policy) that can be run jointly in a single loop.

Why This Matters

This work targets a practical gap in offline RL: producing agents that both perform a task well and do it in a requested way, using only pre-recorded data. It offers an interpretable, low-cost style definition that sidesteps manual labeling and unsupervised latent codes, and it treats style alignment as a first-class objective rather than a constraint to be traded away.

Real-world applications:

• Video games: generating non-player characters or animation that match a designer-specified play or motion style, with Ubisoft La Forge among the author affiliations.
• Robotics and human-robot interaction: training robots from logged demonstrations to move in a legible or safe manner, not just an efficient one.
• Autonomous driving: learning driving policies from recorded data that follow a specified style, such as cautious or assertive, while remaining performant.
• Animation and character motion synthesis: retargeting recorded motion into stylized locomotion such as running, crawling, or varying gait patterns.

Industry relevance: the offline setting means training can reuse existing logged data without expensive or risky live environment interaction, which matters for games, simulation, and safety-critical systems. The method is also model-free and lightweight to integrate, since it is a direct adaptation of IQL, and the authors release code, datasets, and visuals.

Future Directions

• Principled constrained or multi-objective formulations. The authors state that their GAWR gate is a practical approximation to an exact projection onto the set of style-optimal policies, and they explicitly list constrained or multi-objective stylized offline RL as future work, noting the difficulty of choosing a universal alignment threshold.
• Richer labeling functions for complex observations. The paper positions its work as complementary to Lin et al. (2024a), suggesting VQ-VAE state categories could supply semantically meaningful labels in pixel-based observation spaces.
• Better style-reward estimators. The choice of χ was made empirically (Appendix E.1) and simplified to an indicator; learning or improving the label predictor is an open design question.
• Extending beyond the presented environments. The experimental setup covers Circle2d, HalfCheetah, and the (truncated) HumEnv benchmarks, leaving the question of how the framework scales to other task domains and criteria.

Target Audience

Researchers and practitioners in offline reinforcement learning, imitation learning, and goal-conditioned RL who are interested in controllable, style-conditioned behavior; game AI engineers and robotics researchers who need agents trained from logged data that match a specified behavioral style; and readers already familiar with IQL, expectile regression, and advantage-weighted regression who want to see how a style objective can be folded into that machinery.

Authors’ abstract

We study offline reinforcement learning of style-conditioned policies using explicit style supervision via subtrajectory labeling functions. In this setting, aligning style with high task performance is particularly challenging due to distribution shift and inherent conflicts between style and reward. Existing methods, despite introducing numerous definitions of style, often fail to reconcile these objectives effectively. To address these challenges, we propose a unified definition of behavior style and instantiate it into a practical framework. Building on this, we introduce Style-Conditioned Implicit Q-Learning (SCIQL), which leverages offline goal-conditioned RL techniques, such as hindsight relabeling and value learning, and combine it with a new Gated Advantage Weighted Regression mechanism to efficiently optimize task performance while preserving style alignment. Experiments demonstrate that SCIQL achieves superior performance on both objectives compared to prior offline methods. Code, datasets and visuals are available in: https://mathieu-petitbois.github.io/projects/sciql/.