Action-Free Offline-to-Online RL via Discretised State Policies

AI summary

Overview

Research area: Offline and offline-to-online reinforcement learning, specifically the action-free setting where datasets contain only (s, r, s′) tuples.

Technical level: Advanced. The paper assumes familiarity with MDPs, Q-learning, conservative offline RL regularisation, and actor-critic algorithms such as TD3 and SAC.

Scope: The paper introduces OSO-DecQN, a value-based algorithm that pre-trains state policies from action-free offline data using a discretised state-difference transformation, plus a guided online learning mechanism that transfers those state policies into an online agent.

What This Paper Is About

Most offline RL methods assume the dataset contains action labels, but in many real settings actions are missing for privacy, storage, or sensor reasons, leaving only states, rewards and next states. This paper formalises "action-free offline-to-online RL," where an agent must learn from (s, r, s′) tuples alone and then use that knowledge to speed up online interaction. The goal is to learn policies that recommend desirable next-state transitions instead of actions, and to convert those transition recommendations into executable actions during online learning.

Key Contributions

1. A scale-invariant state discretisation transformation. Each state dimension's change is mapped to one of three values (−1, 0, +1) indicating decrease, no meaningful change, or increase, after first applying z-score normalisation per state dimension. This converts unstable continuous next-state regression into a discrete prediction problem.

2. OSO-DecQN (Offline State-Only DecQN). A value-based offline algorithm that adapts the DecQN framework from action dimensions to state-difference dimensions, using an ensemble variant of DecQN, a double Q-learning variant, and a conservative regularisation term adapted to the discrete state-difference domain.

3. A theoretical value bound. Theorem 1 states that with k evenly spaced bins per coordinate in the discretised increment space, the gap between the optimal value functions of the original and discretised MDPs satisfies ‖V* − V_D*‖∞ = O(H√M / k), where M is state dimension and H is the range of per-coordinate mean increments.

4. A guided online learning mechanism. The pre-trained state policy selects a discretised state difference via arg max over Q(s, Δs); a lightweight inverse dynamics model (IDM) trained from scratch, concurrently with the online agent, translates that into an action. A policy-switching rule with hyperparameter β and ζ ∼ U(0,1) blends these guided actions with the online agent's own policy π_on.

Main Findings

• Guided online learning improves both speed and final performance. Across D4RL datasets with TD3 as the base algorithm and Action-factorised DeepMind Suite datasets with an ensemble DecQN variant (DecQN_N) as the base algorithm, guiding online agents with pre-trained state policies improved asymptotic performance and convergence speed over 1M timesteps, measured as mean normalised return across 5 seeds with shading at 1 standard deviation.

• Scalability to high-dimensional states. The approach was applied in environments with up to 78 state dimensions (Quadruped-Walk). Consistent improvement was observed during early learning stages in Quadruped-Walk, which the authors present as evidence of scalability.

• Hopper showed the smallest benefit. The authors attribute this to Hopper being the simplest task with the fewest actions and observation dimensions, so the online agent learns quickly without guided exploration; small gains in asymptotic performance and convergence speed were still observed.

• Continuous next-state targets fail. In Table 1, imitation policies trained to mimic s′ or s′ − s produce poor returns, while policies trained to mimic the discrete Δs produce returns matching an action-labelled imitation policy. For example, on Hopper-medium-replay, BC_s′ scores 5.8 and BC_{s′−s} scores 4.9, versus 29.2 ± 3.7 for BC_Δs.

• Regularisation is critical. A state-only DecQN variant without the regularisation term, DecQN_N, struggles to learn in Table 1 and also fails to predict next states accurately in Table 2, which the authors interpret as evidence of both overestimation bias and unmet state-reachability constraints. OSO-DecQN addresses both.

• OSO-DecQN beats imitation learning and DecQN_N. In Table 1, OSO-DecQN outperforms both action-based BC and discrete-state BC in the reported settings, for instance 65.7 ± 2.6 on Hopper-medium-replay and 110 ± 1.3 on Hopper-medium-expert, versus 7.7 ± 1.3 for DecQN_N on the former.

• The baseline action-free method underperforms. Af-guide (Zhu et al., 2023) consistently underperformed even a from-scratch TD3 baseline across all three environments shown, in contrast to OSO-DecQN. The authors also note their method avoids decision-transformers and therefore substantially reduces pre-training time.

• The IDM is not the bottleneck in tested settings. Ablations over IDM architecture and batch size (Appendix E.1) showed similar performance for smaller and larger architectures and batch sizes, suggesting the IDM has minimal effect on online guidance in the settings tested; sensitivity to β is analysed in Appendix D.3.

• Robustness to discretisation choices. Appendix D reports sensitivity of pre-training performance to the discretisation threshold ε and a comparison between 2-bin and 3-bin discretisation, which the authors state shows robustness across a broad range of ε and granularities.

• Generalises to another online algorithm. Appendix F applies the same method and hyperparameters to SAC with comparable improvements to those obtained with TD3.

Methodology in Plain English

The problem is that you have logs of states, rewards and resulting states, but no record of which action caused which transition. So instead of asking "which action should I take?", the method asks "which way should the state move?".

To make that question learnable, the authors take each state dimension's change between consecutive timesteps, normalise the states first so dimensions with wildly different numeric scales are comparable, and then collapse each change into one of three symbols: it went down a lot, it stayed roughly the same, or it went up a lot (governed by a threshold ε). Now the prediction target is a small discrete label, which lets the authors use Q-learning machinery rather than fragile continuous regression.

They then reuse the DecQN idea — normally used to decompose Q-values across action dimensions so complexity grows linearly rather than exponentially — but decompose across state-difference dimensions instead. An ensemble of critics and a double Q-learning variant add stability, and a conservative regularisation term (a log-sum-exp style penalty over the Q-values) stops the critic from overestimating the value of state differences the data never contained, which also keeps predictions within reachable states.

Because the learned state policy outputs a desired state change rather than an action, it cannot act in the environment directly. The online stage therefore trains a small neural network, the inverse dynamics model, to answer "given this state and this desired state change, what action would produce it?". That IDM is trained with an L1 loss on both online-collected samples and offline samples, where offline actions are imputed using the online agent's own policy. During training, with probability β the agent executes the IDM's translation of the offline state policy's recommendation, and otherwise it uses its own online policy. All transitions go into the standard replay buffer, and the online agent trains normally.

Why This Matters

Impact on research. The paper provides the first demonstration, in the authors' framing, that an action-free offline state-policy learner can be combined with a guided online mechanism to produce consistent gains across diverse control environments. It also offers a reusable offline analysis protocol — including a per-step discrete state difference error metric and an expert-level IDM used only for evaluation — for comparing alternative action-free state policies. The theoretical discretisation bound gives a principled handle on how much value information is lost by the transformation.

Real-world applications.

• Healthcare: learning from patient records where explicit treatment decisions are excluded for legal or confidentiality reasons, leaving only states such as symptoms, vitals and lab results (motivated by Kushida et al., 2012; Yu et al., 2021).
• Finance: settings where exact trades or order flows are withheld to protect proprietary strategies, leaving only partial portfolio states, and where risk makes online exploration costly (Liu et al., 2022; Loukides and Gkoulalas-Divanis, 2012).
• Robotics and industrial control: using sensor logs kept without the precise torque or command signals, either to reduce storage overhead or because that information is proprietary.
• Sensor-failure recovery: exploiting historical logs from periods when action channels were corrupted or never recorded.

Industry relevance. The method avoids transformer-based architectures, which lowers pre-training cost, and it scales to environments with 78 state dimensions while requiring minimal tuning to transfer from TD3 to SAC. The lightweight IDM keeps computational overhead low, which matters for deployment on systems with constrained compute.

Future Directions

• Adaptive discretisation. Adopting mechanisms such as those of Seyde et al. (2024) to see whether finer, adaptive control improves pre-training and online guidance.
• More expressive translation. Replacing the minimal IDM with richer inverse dynamics models to improve robustness where dynamics are strongly discontinuous or inverse mappings are highly multi-modal — the specific failure conditions the authors identify.
• Learning from video. Extending the method to suboptimal video demonstrations, which requires reliably extracting rewards from visual data and converting images into vector representations compatible with the state-difference formulation.
• Cross-entropy value objectives. Combining OSO-DecQN with work that replaces value-function regression by classification over discretised value supports (Farebrother et al., 2024), potentially improving critic stability for larger architectures or more complex domains.
• Reward shaping instead of action switching. The authors deliberately left the intrinsic guiding-reward approach of Zhu et al. (2023) to future work, noting it lacks convergence guarantees.

Target Audience

Researchers and practitioners working on offline RL, offline-to-online transfer, and offline RL with incomplete or privacy-restricted logging. It is most useful to readers already comfortable with Q-learning variants, conservative value regularisation, and actor-critic algorithms such as TD3 and SAC. Engineers in healthcare, finance, robotics, or industrial control who have large state-and-reward logs but no action labels are the primary applied audience. Readers new to RL will find the discretisation intuition accessible but will need background for the theoretical bound and the DecQN decomposition.

Authors’ abstract

Most existing offline RL methods presume the availability of action labels within the dataset, but in many practical scenarios, actions may be missing due to privacy, storage, or sensor limitations. We formalise the setting of action-free offline-to-online RL, where agents must learn from datasets consisting solely of $(s,r,s')$ tuples and later leverage this knowledge during online interaction. To address this challenge, we propose learning state policies that recommend desirable next-state transitions rather than actions. Our contributions are twofold. First, we introduce a simple yet novel state discretisation transformation and propose Offline State-Only DecQN (\algo), a value-based algorithm designed to pre-train state policies from action-free data. \algo{} integrates the transformation to scale efficiently to high-dimensional problems while avoiding instability and overfitting associated with continuous state prediction. Second, we propose a novel mechanism for guided online learning that leverages these pre-trained state policies to accelerate the learning of online agents. Together, these components establish a scalable and practical framework for leveraging action-free datasets to accelerate online RL. Empirical results across diverse benchmarks demonstrate that our approach improves convergence speed and asymptotic performance, while analyses reveal that discretisation and regularisation are critical to its effectiveness.