Selective Growing Q-Networks (SGQN): An Analysis of Adaptive Action Grids in Decoupled Q-Learning

AI summary

Overview

Research area: Reinforcement learning for continuous control, specifically value-based (Q-learning) methods that discretise the action space instead of learning a continuous policy.

Technical level: Intermediate. Readers with basic familiarity with reinforcement learning, Q-learning and discretisation will follow it; the paper is more an empirical component analysis than a theory paper.

Scope: The paper introduces Selective Growing Q-Networks (SGQN), an adaptive action-grid variant of decoupled Q-learning, evaluates it on eight DeepMind Control Suite tasks against five baselines, and then tests in detail whether the adaptive grid is actually responsible for its results.

Note on content: The supplied paper content is truncated inside Section 5.4. The hyperparameter-sensitivity numbers, the conclusion and the appendices are not available here, so results from those parts are not reported below.

What This Paper Is About

Discretising a continuous action space lets you use Q-learning, but the granularity you pick forces a trade-off: a coarse grid explores easily but ends up imprecise, while a fine grid is precise but converges slowly. Growing Q-Networks (GQN) handled this by refining the grid uniformly over training, ignoring which parts of the action range the learned Q-function considers promising. SGQN's goal is to make the grid follow the learned value function instead, by adding bins next to high-Q bins and pruning low-Q bins, and to test honestly whether that idea actually explains the method's performance.

Key Contributions

1. A new method. SGQN, an adaptive-grid variant of decoupled Q-learning with value-guided selective growth, lazy pruning, and ε-greedy selection, evaluated on eight DMControl tasks against five baselines. It has the highest mean return on four tasks and is close to the best method on three more.
2. An analysis of the learned action grids. The paper examines which bins are active over training, how the Q-values of active bins compare to inactive ones, and which bins get pruned. In the reported runs the growth rule activates every candidate bin, so it behaves like uniform growth, and the retained and removed bins have nearly identical Q-values, while pruning is weakly but consistently biased toward low-Q bins.
3. Component ablations and a hyperparameter sensitivity study. Within SGQN, the results attribute performance to starting from a coarse grid and to ε-greedy selection, not to value-guided growth or pruning, and the growth and pruning hyperparameters can be varied widely without changing the return.

Main Findings

• Strong but not dominant returns. SGQN reaches the highest mean final return on four of the eight tasks, is within 1.1% of the best method on three more, and is 3.4% below Bang-Bang MPO on the remaining one (Walker Run). It uses on average 7.5 of the 9 available bins per dimension at the end of training.
• SGQN's wins. Cheetah Run 788 (±37) versus 741 for an earlier SGQN version, 648 for Bang-Bang MPO and 603 for GQN; Fish Swim 715 (±17) versus 662 and 561; Hopper Hop 257 (±11) versus 203 for Bang-Bang MPO and 174 for GQN; Quadruped Walk (12 action dimensions) 924 (±10) versus 901, 812 and 739. On Hopper Hop and Quadruped Walk it also has a much smaller spread across seeds (standard deviations of 11 and 10) than GQN (55) and Bang-Bang MPO (30 on Hopper Hop) and their 208 and 199 on Quadruped Walk.
• The Reacher tasks do not separate methods. On Reacher Easy, SGQN (962), GQN (966), the earlier SGQN version (967) and Bang-Bang MPO (959) are within 1% of each other; on Reacher Hard, SGQN (951), GQN (953) and Bang-Bang MPO (943) are too. All these differences are within about one standard deviation of the seeds.
• Where SGQN loses. Bang-Bang MPO has the highest mean on both Walker tasks: 738 versus 713 for SGQN on Walker Run (3.4% lower) and 967 versus 957 on Walker Walk (1.0% lower). Both gaps are larger than the standard deviations across seeds of either method, so Bang-Bang MPO is better on these two tasks.
• The adaptive grid is not what drives the results. In the runs, growth activates every candidate bin and therefore behaves like uniform growth. The alignment of active bins with Q-values (mean Q of active bins divided by mean Q of the whole grid) is 1.000 on every task and 1.001 on Quadruped Walk. The Q-range across the nine bins of a dimension is a median of only 0.5% to 2.9% of its mean, so almost every bin lies above the growth threshold.
• Pruning is only weakly value-guided. Across 56 runs there were 903 pruning events. No anchor bin was removed. The removed bin was in the lowest fifth of the Q-ranking in 29% of events, against 17.8% expected under random removal given the candidate counts, and in the lowest two fifths in 50% of events against 36% expected.
• The active schedule is fixed, not task-specific. Every task follows the same schedule: three anchor bins until the first growth step at episode 300, seven active bins after it, nine after the growth step at episode 600, then pruning (one bin removed at each of episodes 650 and 700), regrowth at episode 900 and further pruning at episode 950. The schedule is identical across seeds and tasks.
• What does matter: a coarse start. Starting with all 9 bins from the beginning instead of growing into them leaves mean returns of 722 (Fish Swim), 252 (Hopper Hop), 818 (Quadruped Walk) and 684 (Walker Run), and produces individual seeds that collapse — the Quadruped Walk standard deviation is 132.
• What does matter: ε-greedy selection. Replacing ε-greedy with the earlier annealed softmax lowers the return by 23% to 100%: 508 (Fish Swim), 0 (Hopper Hop), 250 (Quadruped Walk) and 547 (Walker Run), against a full method of 718, 254, 927 and 713.
• Growth and pruning ablations change almost nothing. Uniform instead of value-guided growth agrees to the rounded value in 19 of 20 runs, with the remaining Quadruped Walk seed differing by 7 points (917 versus 924). Removing pruning changes the return by between −0.4% and +1.1%, within one standard deviation on all four ablated tasks, while removing about 1.5 of the 9 bins per dimension.
• Hyperparameter insensitivity. SGQN is reported as insensitive to its growth and pruning hyperparameters, including growth thresholds varied up to 0.95, which also leave the return essentially unchanged.
• Confounded baselines. SGQN differs from the GQN and DecQN baselines in other respects, notably one gradient update per environment step instead of one per eight, plus a different ε schedule (0.1 decayed to 0.01 versus a constant 0.1), a different network and a different replay exponent β. The authors did not control for these, so they do not attribute SGQN's advantage to a single component.
• Weak baselines are not informative. DecQN and CQN stay below every other method on every task, mostly by a wide margin; their best results are 389 and 413 on Walker Walk, and they reach 1 and 10 on Hopper Hop. The authors ran both with their publication settings without tuning. The CQN comparison is explicitly not like-for-like because CQN learns from pixels (84×84 images, 3 stacked frames) while every other method observes the proprioceptive state.

Methodology in Plain English

The action grid. For each action dimension the method keeps a fixed full grid of N = 9 uniformly spaced bins covering that dimension's range, plus a binary mask saying which bins are currently active. Only active bins participate in action selection and in the TD target's maximisation. Training starts with n₀ = 3 active bins: the two boundary bins and the centre bin, which are treated as anchors and are never pruned.

Two rules change the grid. Selective growth runs every Δ_g episodes after a warm-up: any active bin whose running mean Q-value exceeds a fraction (τ_Q, set to 0.85 in the main runs) of the best bin of that dimension has its inactive immediate neighbours activated. Lazy pruning runs every Δ_p episodes: among bins that are "mature" (a minimum age), bins whose running mean Q-value falls below a τ_p-quantile of the mature bins are flagged, and then removed only if they are eligible — not anchors, and either selected at least v_min times or having lived through L_max pruning events. A dimension never drops below n_min active bins. Pruned bins are held back from regrowth for a cooldown period.

Action selection. With probability ε, each dimension independently picks an active bin uniformly at random; otherwise it picks the active bin with the highest Q-value. ε starts at 0.1 and is multiplied by 0.995 after every episode down to 0.01.

Why decoupling matters. The Q-function is decomposed into one head per action dimension (as in DecQN), so the joint action space is never enumerated: complexity is O(d × n) rather than O(n^d) in the number of dimensions d and bins n. SGQN keeps the same off-policy training as DecQN and GQN: prioritised experience replay (α = 0.6, constant β = 0.4), double Q-learning with twin critics, 3-step returns, layer normalisation, two hidden layers of 512 units, Adam at 1e-4, γ = 0.99, a replay buffer of 10⁶, batch size 256, target networks hard-updated every 100 gradient updates, and gradient clipping at norm 40.

Evaluation. Eight DMControl tasks spanning 2 to 12 action dimensions (Reacher Easy and Reacher Hard, Hopper Hop, Fish Swim, Cheetah Run, Walker Walk, Walker Run, Quadruped Walk), 1M environment steps each, targeting 7 random seeds per method and task, with the mean episode return over the final 100 episodes reported. A few baseline runs produced no data and are excluded.

The distinctive part: measuring the mechanism. The authors did not stop at benchmark scores. They logged, every 25 episodes, which bins were active, each bin's running mean Q-value, and how often it was selected, across all 56 main runs, so they could check whether resolution actually follows value. They then changed one design choice at a time (no pruning; uniform instead of value-guided growth; start with all 9 bins; annealed softmax instead of ε-greedy) on Quadruped Walk, Hopper Hop, Walker Run and Fish Swim with 5 seeds each, and swept individual hyperparameters on Hopper Hop and Walker Run with 3 seeds.

Why This Matters

Impact on research. The paper is framed by its authors as "above all, an analysis of which ingredients of a growing action grid matter and which do not." It provides a negative result with measurements attached: a value-guided adaptive discretisation, the mechanism that motivated the method, was inert in these runs because converged Q-values across bins sat within a narrow band (0.5% to 2.9% of the mean). That is a caution for anyone designing value-driven representation growth, and the ablation design — changing one component at a time and reporting where the performance actually comes from — is a template for more honest component analysis in this line of work.

Real-world applications. The paper itself does not report real-world deployments; the following are plausible domains suggested by the evaluated tasks.

• Robotic locomotion and whole-body control, where action dimensions range from a few joints to a quadruped's twelve, and where bang-bang boundary behaviour is often near-optimal.
• Low-compute controllers where value-based Q-learning avoids the hyperparameter burden and compute cost of actor-critic training.
• Sim-to-real pipelines where a small, discrete action set is easier to certify and schedule than a continuous policy output.
• Benchmark-driven method engineering: the paper's ablation protocol is directly reusable for testing whether a proposed adaptive component does anything.

Industry relevance. Value-based methods that avoid policy parameterisation are attractive where stability and small hyperparameter search budgets matter. The results here show that a simpler configuration — a coarse start plus ε-greedy selection and a fixed schedule — carries most of the benefit, which lowers the engineering cost of adopting the approach. The caveat is equally relevant: SGQN's advantage over GQN is confounded with a much higher update frequency (one gradient update per environment step versus one per eight), so no published number here should be treated as a clean speed comparison.

Future Directions

1. Isolate the update-frequency confound. SGQN performs one gradient update per environment step and GQN one per eight; the authors call the larger number of updates "a candidate that we have not tested." Running GQN with SGQN's update frequency, ε schedule, network and β is the obvious missing experiment.
2. Find regimes where value-guided growth actually selects. The growth rule was inert because Q-value spreads were narrow. Whether higher spreads — different tasks, different reward scales, or more heterogeneous action dimensions — make the rule bite is untested, and the threshold was only varied to 0.95.
3. Untangle age from value in pruning. The running mean is taken since a bin was activated, so it is biased by bin age. The authors treat this as a hypothesis and say they have not isolated the effect.
4. Understood-the-baselines problem. DecQN and CQN stayed "below every other method on every task, mostly by a wide margin," which the authors ran untuned and did not investigate, leaving the comparison against those methods uninformative and, for CQN, not like-for-like because it learns from pixels.

Target Audience

Reinforcement learning researchers working on value-based methods for continuous control, action-space discretisation or adaptive representations — the paper's main value is the negative result and the ablation methodology, not the headline scores. It also suits practitioners who want a Q-learning approach to continuous control without actor-critic machinery, and students learning how to test whether a proposed mechanism is the thing actually producing results. Readers looking for state-of-the-art scores, a theoretical analysis of adaptive discretisation, or a controlled speed comparison against GQN will not find them here; the authors explicitly decline to attribute the method's advantage to any single component.

Authors’ abstract

Value-based Q-learning for continuous control can discretise the action space, but the resolution of the discretisation trades early exploration against final precision. Growing Q-Networks (GQN) grow the resolution uniformly from coarse to fine. We introduce Selective Growing Q-Networks (SGQN), which keeps decoupled Q-learning and replaces uniform growth with an adaptive action grid: bins are added next to bins with high estimated Q-value, bins with low Q-value are pruned, and actions are chosen epsilon-greedily. On eight DeepMind Control Suite (DMControl) tasks with 2 to 12 action dimensions, SGQN has the highest mean final return among the compared methods on four, is within 1.1% of the best method on three more, and is 3.4% below Bang-Bang MPO on Walker Run, using on average 7.5 of 9 available bins per dimension. We then test whether the adaptive grid explains these results, and find that it does not. Growth activates every candidate bin and behaves like uniform growth, and removing pruning changes the return by at most 1.1%. Two other choices matter: starting from a coarse grid, since starting with all 9 bins produces collapsed seeds, and epsilon-greedy selection, since an annealed softmax lowers the return by 23% to 100%. SGQN is insensitive to its growth and pruning hyperparameters. It also differs from the GQN and DecQN baselines in other respects, notably one gradient update per environment step instead of one per eight, which we did not control for, so we do not attribute its advantage to a single component. We present SGQN as a competitive value-based method and as an analysis of which ingredients of a growing action grid matter.