Learning Human-Like RL Agents Through Trajectory Optimization With Action Quantization

AI summary

Overview

Research area: Reinforcement learning, specifically human-like agent design and offline/imitation-driven RL.

Technical level: Intermediate. Familiarity with standard RL concepts (MDPs, policies, offline datasets) helps, plus a working intuition for variational autoencoders.

Scope: The paper proposes Macro Action Quantization (MAQ), a framework that distills human demonstrations into discrete "macro actions" via a conditional VQVAE and uses them as the action space for off-the-shelf RL algorithms, evaluated on four D4RL Adroit tasks with trajectory-similarity metrics and a 19-person human evaluation study.

What This Paper Is About

Reward-driven RL agents can achieve strong task performance while behaving in ways that look nothing like humans — for example, "glitching" a hand through a door handle instead of gripping it. The paper asks how to design agents that both succeed at a task and move like a human, and proposes formulating human-likeness as a trajectory optimization problem: find an action sequence that stays close to human behavior while still maximizing reward. The authors implement this with MAQ, which replaces primitive actions with short sequences of human-derived actions so the agent plans inside a human-like behavioral space.

Key Contributions

1. A trajectory-optimization formulation of human-likeness. The paper formalizes human-like behavior as maximizing return subject to a constraint that the action sequence lies in a "human manifold" — the set of human-generated action sequences — and adapts receding-horizon control into Human-like Receding-horizon Control (HRC) to make this tractable.

2. Macro Action Quantization (MAQ). A framework that trains a Conditional-VQVAE on state and macro-action pairs from offline human demonstrations, producing a discrete codebook of macro actions (sequences of H consecutive actions). A policy then learns over codebook indices rather than continuous actions.

3. Algorithm-agnostic integration and benchmark evaluation. MAQ is applied on top of three off-the-shelf RL algorithms — IQL, SAC (using Discrete SAC), and RLPD — yielding MAQ+IQL, MAQ+SAC, and MAQ+RLPD, evaluated on four D4RL Adroit tasks: Door, Hammer, Pen, and Relocate.

4. A human evaluation study with 19 evaluators. Participants completed a Turing Test (2AFC, forced choice between a human demo and an agent video) and a human-likeness ranking test across agent pairs.

Main Findings

• MAQ raises trajectory similarity across every task and metric. The paper reports normalized DTW and Wasserstein scores where higher is more human-like. In Door, DTW_s rises from 0.43 ± 0.06 (IQL) to 0.84 ± 0.06 (MAQ+IQL), from -0.39 ± 0.10 (SAC) to 0.80 ± 0.08 (MAQ+SAC), and from -0.06 ± 0.04 (RLPD) to 0.76 ± 0.04 (MAQ+RLPD). In Hammer, WD_a rises from -0.19 ± 0.04 (SAC) to 0.78 ± 0.11 (MAQ+SAC), and from 0.20 ± 0.03 (RLPD) to 0.85 ± 0.03 (MAQ+RLPD).

• The largest gains are on the algorithm that used no human demonstrations. Averaged over the four tasks, DTW_s moves from -0.49 ± 0.48 for SAC to 0.56 ± 0.23 for MAQ+SAC. The authors attribute this to SAC learning purely from environment interaction, unlike IQL and RLPD.

• Task success is largely preserved but not uniformly. Average success rates are BC 0.11 ± 0.19, IQL 0.14 ± 0.19, MAQ+IQL 0.39 ± 0.40, SAC 0.19 ± 0.22, MAQ+SAC 0.28 ± 0.25, RLPD 0.68 ± 0.40, and MAQ+RLPD 0.52 ± 0.32. Individual tasks vary widely: on Hammer, RLPD records 1.00 ± 0.00 while MAQ+RLPD records 0.56 ± 0.37 and MAQ+IQL records 0.00 ± 0.00; on Door, MAQ+IQL scores 0.93 ± 0.04 against RLPD's 0.96 ± 0.07.

• Behavioral cloning achieves moderate similarity but the worst success. BC scores 0.16 ± 0.29 on average DTW_s yet only 0.11 ± 0.19 average success, because it imitates state-action pairs without optimizing for task outcomes.

• Longer macro actions perform better. Sweeping macro action length H from 1 to 9 on MAQ+RLPD across the four Adroit tasks, H = 9 gives the highest similarity scores and the best success rate. WD_a stays relatively stable across all values of H, which the authors link to the Adroit action space consisting solely of Shadow Hand joint positions — suggesting short sequences can look human-like in action space without being effective at the task.

• MAQ agents are the hardest for humans to distinguish. Across the four tasks, Turing Test win rates (the share of questions where the evaluator was fooled) are: MAQ+RLPD 39% > MAQ+SAC 34% > MAQ+IQL 32% > BC 24% = RLPD 24% > SAC 19% > IQL 13%. The paper describes MAQ+RLPD's win rate as 15% higher than non-MAQ agents. Error bars in that figure are 96% confidence intervals.

• The ranking test mirrors the Turing Test. Overall human-likeness rankings are: Human 74% > MAQ+RLPD 71% > MAQ+SAC 61% = MAQ+IQL 61% > BC 43% > RLPD 38% = IQL 38% > SAC 14%. MAQ+RLPD's 71% is close to the human demonstrations' 74%. The only order change versus the Turing Test is that IQL and SAC swap positions.

• Success does not imply human-likeness. SAC wins nearly 0% of Door comparisons despite a 43% success rate. The Pen task produces the highest win rate for all agents, indicating evaluators found it hardest to distinguish agents from humans there.

• Qualitative feedback supports the metrics. In Door, MAQ+RLPD holds, rotates, and pulls the handle in a way evaluators called human-like ("it shows how a human casually opens a door — the grip on the handle is stable"), while RLPD presses the handle backhanded and pulls without properly gripping — described by evaluators as "glitching its hand through the door handle."

Methodology in Plain English

The problem is framed as: at each decision point, choose a short sequence of actions that both earns reward and looks like something a human would do. Optimizing a whole episode at once is intractable, so the authors borrow receding-horizon control — repeatedly solve a small optimization over a short window of H steps, execute it, then replan. To enforce human-likeness, they constrain the search to segments that appear in human demonstrations.

Searching raw human data at every step would be far too slow, so the method quantizes those segments into a finite "menu." A Conditional-VQVAE is trained on pairs of (state, H-step action sequence) drawn from demonstrations via a sliding window. The encoder maps a sequence to a latent vector, which is snapped to its nearest entry in a learned codebook of size K; the decoder reconstructs the sequence from that entry plus the state. The loss combines reconstruction error, codebook loss, and a commitment loss with coefficient β = 0.25.

Once trained, the codebook serves as the agent's action space. The RL policy outputs K logits — one per codebook entry — and the selected entry is decoded into a macro action that the environment executes. This turns the original MDP into a Semi-Markov Decision Process with cumulative rewards over each segment. Because the policy only ever selects human-derived macro actions, the agent is confined to a human-like behavioral space while still optimizing return. Crucially, this wrapper works on top of existing algorithms without modifying their internals.

Experimental setup details: each of the four Adroit tasks includes 25 human demonstration trajectories, split 9:1 into training and testing. All agents train for 10^6 steps, with IQL and MAQ+IQL trained for 10^6 steps in each of their offline and online stages. The VQVAE uses β = 0.25, hidden size 256, batch size 32, and 100 training episodes, and the same trained VQVAE is shared across all MAQ-based agents. Similarity is measured with Dynamic Time Warping (state and action variants, DTW_s and DTW_a) and Wasserstein distance (WD_s, WD_a, computed with the POT library's emd2 solver), normalized per task by 1 - (agent score - human score) / (random score - human score).

Why This Matters

Research impact. Most RL research optimizes reward alone, and the paper argues that interpretability and trustworthiness suffer when agents behave unnaturally. By casting human-likeness as an optimization constraint rather than a handcrafted penalty, MAQ offers an alternative to prior approaches that rely on rule-based behavior costs (e.g., penalizing spinning and shaking) or on behavior cloning, which the paper finds limits task performance. The framework also comes with quantitative human-likeness metrics (DTW, Wasserstein) and a documented human evaluation protocol, which the authors position as a standardized way to measure and compare human-like behavior.

Real-world applications mentioned or implied by the paper:

• Human-robot collaboration. A robot that moves in human-like ways can improve safety and trust and reduce the risk of accidents.
• Robot arm manipulation. The Adroit tasks (opening a door, hammering a nail, twirling a pen, relocating a ball) are direct proxies for manipulation settings where demonstrations are already collected.
• Real-time strategy games. The paper notes agents often act at unnaturally high frequencies to maximize performance; MAQ regulates decision frequency through learned macro actions rather than manually designed delays.
• Integrity risks in competitive play. The authors' broader-impact discussion explicitly flags that human-like agents could be misused for deceptive or manipulative purposes, such as cheating in competitive games.

Industry relevance. MAQ is presented as a drop-in wrapper requiring no changes to the underlying RL algorithm, and it was demonstrated with three different algorithms. The paper reports that human demonstrations already exist and are widely used for RL training in domains such as Atari games, self-driving cars, and robot arm manipulation — meaning the raw material MAQ needs is often already available in industrial pipelines.

Future Directions

• Reducing dependence on human demonstrations. The paper names this as the main limitation: MAQ requires human demonstrations to build the codebook, and demonstration quality affects effectiveness. The authors argue that collecting them is appropriate given the goal of human-likeness, but robustness to limited or noisy demonstrations is left open.

• Scaling to more complex domains. The authors state the results suggest MAQ has potential in harder settings such as real-time strategy games, where the number of possible macro actions and the need for realistic decision frequency are both greater.

• Better metrics for action-space human-likeness. WD_a remained relatively stable across all macro action lengths H = 1 to 9, suggesting it may not discriminate well in Adroit's joint-position action space. Whether this holds in other action spaces is not reported.

• Broadening the algorithm and benchmark coverage. MAQ was tested with IQL, SAC (via Discrete SAC), and RLPD on four Adroit tasks. Whether the same gains transfer to other RL algorithms, other D4RL domains, or discrete-action environments is not reported in the paper content provided.

Target Audience

This paper is most useful for RL researchers working on imitation learning, offline RL, or human-likeness and interpretability in agent behavior; for robotics and human-robot interaction practitioners who need agents whose motions are acceptable to human collaborators; and for game AI developers interested in making agents behave and act at human-like frequencies. Readers evaluating human-likeness methodology — including the design of Turing-Test-style user studies — will also find the evaluation protocol directly relevant.

Authors’ abstract

Human-like agents have long been one of the goals in pursuing artificial intelligence. Although reinforcement learning (RL) has achieved superhuman performance in many domains, relatively little attention has been focused on designing human-like RL agents. As a result, many reward-driven RL agents often exhibit unnatural behaviors compared to humans, raising concerns for both interpretability and trustworthiness. To achieve human-like behavior in RL, this paper first formulates human-likeness as trajectory optimization, where the objective is to find an action sequence that closely aligns with human behavior while also maximizing rewards, and adapts the classic receding-horizon control to human-like learning as a tractable and efficient implementation. To achieve this, we introduce Macro Action Quantization (MAQ), a human-like RL framework that distills human demonstrations into macro actions via Vector-Quantized VAE. Experiments on D4RL Adroit benchmarks show that MAQ significantly improves human-likeness, increasing trajectory similarity scores, and achieving the highest human-likeness rankings among all RL agents in the human evaluation study. Our results also demonstrate that MAQ can be easily integrated into various off-the-shelf RL algorithms, opening a promising direction for learning human-like RL agents. Our code is available at https://rlg.iis.sinica.edu.tw/papers/MAQ.