Selective Agent Guidance via Entropy: Learning Autonomous Policies from Imperfect VLM Teachers

AI summary

Overview

• Research area: Reinforcement learning (RL) combined with Vision-Language Models (VLMs) as teachers for interactive agents; AI / interactive decision-making.
• Technical level: Intermediate. A reader needs basic familiarity with RL concepts such as policy entropy, PPO, behavioral cloning, and advantage estimates, but the central idea (ask for help only when unsure, then learn to act alone) is intuitive.
• Scope: The paper proposes SAGE ("Selective Agent Guidance via Entropy"), a framework that queries an imperfect VLM teacher only when a lightweight RL policy is uncertain, and distills that guidance into an autonomous policy that requires no VLM at deployment.

What This Paper Is About

VLMs hold broad visual and linguistic priors that could help agents in sparse-reward environments, but using them directly as policies is expensive (a query at every step), frozen (they do not improve from interaction), and brittle (they repeat systematic mistakes). The paper studies a different setting: treat the VLM as an online, expensive, imperfect but informative teacher for a cheap RL policy, and learn a policy that acts without the VLM at evaluation time. The goal is to use the teacher selectively enough to save cost, and cautiously enough that wrong advice does not corrupt the learner.

Key Contributions

1. The SAGE framework. A dual-architecture training loop in which the learner acts when its normalized policy entropy is below a threshold ν, and otherwise queries a frozen VLM, executes the suggested action, and marks the transition with an indicator g_t. A cache C stores VLM-returned state-action pairs to reduce repeated prompting.

2. A partitioned objective for on-policy RL plus teacher distillation. PPO updates are applied only to learner-generated transitions (B_π), teacher-guided transitions (B_T) are handled by a separate behavioral cloning term (optionally advantage-weighted, "AWBC"), and the value function is trained on the full buffer B. The final objective combines L_PPO − c_H·entropy on B_π, β·L_AWBC on B_T, and c_v·L_value on B, with AWBC weights w_t = exp(Â_t/τ) clipped to 20.

3. A new benchmark and a multi-environment evaluation. The paper introduces CardMaze, a perceptual symbol-matching task (four candidate cards plus a prompt card; the correct action matches the prompt's suit, with a confounder card sharing the same number; reward only after n = 5 consecutive correct selections), and evaluates across FrozenLake 8×8, MiniGrid (LavaGap, GoToDoor, Fetch), EZPoints, CardMaze, and ALFWorld.

4. Diagnostic studies of teacher quality and failure modes. Oracle-teacher variants, a random-teacher control, a two-VLM comparison (Qwen3.5-27B vs Gemma3-27B), and a 5M-step long-horizon experiment separate failures caused by uninformative teachers from failures of the learning framework.

Main Findings

• SAGE improves over unguided PPO in several environments. Peak episodic return (mean over 3 seeds) on CardMaze: SAGE 1.000 (0.000) versus PPO 0.007 (0.006). GoToDoor: 0.147 (0.017) versus 0.131 (0.015). Fetch: 0.122 (0.025) versus 0.075 (0.034). ALFWorld (exploratory): 0.150 (0.017) versus 0.111 (0.019).

• The learned policy can exceed its VLM teacher. On CardMaze, the VLM-as-Policy baseline scores 0.000 (0.000) while SAGE reaches the optimal return of 1.000 (0.000). DAgger also performs strongly on CardMaze at 0.993 (0.012) but queries the VLM at every training step.

• Teacher queries are sharply reduced. Across the six controlled environments, SAGE prompts the VLM on only 1.2%–13.3% of training steps, concentrated in early training when entropy is high. Rates are lowest on FrozenLake, EZPoints, GoToDoor, Fetch, and LavaGap (1.2%–2.7%) and highest on CardMaze (13.3%). VLM-as-Policy, LVLM2P, and DAgger each query at 100%. At deployment, SAGE requires no VLM calls at all.

• Selective guidance is not uniformly beneficial. On LavaGap, PPO already solves the task with 0.945 (0.000) while SAGE scores 0.688 (0.212). On Fetch, the direct VLM policy (0.310, 0.044) beats SAGE (0.122, 0.025). On FrozenLake, SAGE and VLM-as-Policy obtain similar returns (0.103 vs 0.117) within seed variability while PPO stays at 0.000, but the low absolute values suggest unreliable guidance for precise grid localization. On EZPoints with Qwen3.5-27B, both VLM-as-Policy (0.175, 1.488) and SAGE (0.000, 0.000) perform poorly even though the task is solvable with oracle guidance.

• Explicit behavioral cloning is essential; advantage weighting is not. Removing BC causes performance to collapse in every environment (SAGE w/o BC: CardMaze 0.000, LavaGap 0.000, EZPoints −2.410), and even with a rule-based oracle teacher, Oracle w/o BC is near zero everywhere. In contrast, SAGE and SAGE w/o AWBC remain within seed variability across all six environments, with plain BC outperforming AWBC in some cases (FrozenLake 0.180 vs 0.103; GoToDoor 0.160 vs 0.147). The authors therefore treat AWBC as an optional refinement.

• Direct-policy VLM quality does not predict teaching usefulness. With Gemma3-27B as the teacher, SAGE reaches the optimal 10.000 (0.000) on EZPoints, whereas Gemma acting directly scores −3.400 (0.070) and Qwen-guided SAGE stays at 0.000 (0.000). On CardMaze, SAGE reaches 1.000 with either teacher despite a large gap in their direct-policy returns (Qwen 0.000 vs Gemma 0.670). On the three MiniGrid tasks, Gemma is ineffective both as a direct policy (0.000 across FrozenLake, Fetch, GoToDoor, LavaGap) and as a teacher, while Qwen provides more useful guidance. A random teacher never improves over PPO and can degrade policies that already learn from reward (LavaGap SAGE with random teacher: 0.000).

• Guidance changes which trajectories are discovered, not only how fast. In the 5M-step long-horizon study with the oracle teacher, PPO remains near zero on FrozenLake (0.003, 0.006), EZPoints (0.000, 0.000) and CardMaze (0.007, 0.006), while SAGE + Oracle reaches 0.997 (0.006), 10.000 (0.000) and 1.000 (0.000) respectively. On MiniGrid, PPO is stronger, especially on LavaGap where both reach 0.945 (0.000), but oracle guidance still improves Fetch (0.470 vs 0.137) and GoToDoor (0.283 vs 0.180).

• A perfect teacher is not always a better teacher. SAGE + Oracle does not always outperform SAGE; the authors hypothesize that the oracle's determinism may cause overfitting to specific trajectories, while the VLM's suboptimal stochasticity can act as a regularizer (CardMaze: Oracle 0.663, SAGE 1.000).

Methodology in Plain English

The setup is a standard reinforcement learning loop with an unusual helper. The environment is modeled as a Markov Decision Process with an image observation (plus text task information where available), a discrete action space, and sparse rewards.

At every training step the agent measures the entropy of its own action distribution, normalized to the range [0, 1] by dividing by log|A|. High entropy means the agent is unsure. If the normalized entropy exceeds a fixed threshold ν, the agent asks a frozen VLM for one action, executes it, and flags that transition as teacher-guided. Otherwise it acts on its own. The default threshold is 0.75 for FrozenLake, EZPoints, Fetch, GoToDoor, and LavaGap, and 0.25 for CardMaze and ALFWorld.

The resulting experience is split into two buffers. Learner-generated transitions go through PPO as usual, which keeps the on-policy assumptions intact. Teacher-guided transitions go through a behavioral cloning loss instead, because teacher actions are off-policy and would otherwise produce large importance ratios and heavy PPO clipping. Optionally, each cloned teacher action is weighted by exp(Â_t/τ) with temperature τ = 0.5 and weights clipped at 20, so that teacher advice associated with high environment return counts more. The value function is trained on the whole buffer using environment rewards, not teacher labels — important in sparse-reward tasks where teacher-guided actions may be the only source of early successful trajectories.

Evaluation happens without any VLM in the loop, on environment seeds held out from training, so the reported numbers measure what the lightweight policy internalized rather than online assistance. The six controlled environments are trained for 100k environment steps; ALFWorld uses 40k steps because of simulator and VLM inference cost; DAgger is run for 25k steps since it queries the VLM at every step. All results are means over 3 seeds. Qwen3.5-27B is the teacher unless otherwise stated; ALFWorld uses Gemma3-27B. The agent is a CNN (with an LSTM text encoder for ALFWorld), with convolution channels [16, 32, 32], gamma 0.99, GAE lambda 0.95, entropy coefficient 0.01, and learning rate 1×10⁻³.

Why This Matters

Impact on research. The paper reframes VLMs as temporary exploration priors rather than deployed policies. Its most transferable claim is negative and useful: how well a VLM performs when prompted directly does not predict how useful it is as a teacher (Gemma is worse than Qwen as a direct policy on EZPoints but yields optimal SAGE performance). That undercuts the common assumption that stronger direct policies make better supervisors, and it gives the community a diagnostic toolkit — oracle teachers, random teachers, and long-horizon runs — for separating teacher failure from learner failure.

Real-world applications

• Robotics and embodied agents: a small on-board policy could learn household or navigation tasks with occasional cloud VLM calls during training, then run entirely on-device at deployment. ALFWorld and MiniGrid Fetch/GoToDoor are the paper's proxies for this.
• Cost-sensitive deployment of AI assistants: any system that currently calls a large model at every step (for example, step-by-step planning in UI automation or game playing) could instead query selectively on 1.2%–13.3% of training steps and then act without the large model.
• Industrial control and simulation: sparse-reward settings where random exploration almost never finds the goal — the paper's FrozenLake, EZPoints, and CardMaze cases — are common in manufacturing, process control, and logistics scheduling.
• Safety-critical supervision design: the random-teacher control, where guidance actively degraded an already-solvable task (LavaGap), is a concrete warning about when not to add a foundation-model supervisor.

Industry relevance. The cost argument is directly actionable: 100% query rates versus 1.2%–13.3% during training, and zero VLM calls at inference, is a large reduction in API and latency budget. The paper also shows the framework tolerates a weak direct-policy teacher, which means teams do not necessarily need the strongest available model in the loop — a practical lever when the strongest models are the most expensive.

Future Directions

1. Better uncertainty signals than policy entropy. The authors note that entropy can conflate genuine uncertainty with action multimodality and does not estimate whether the teacher will actually be helpful in a given state. They suggest ensembles, disagreement between value estimates, or learned query policies.
2. Estimating teacher usefulness before acting or distilling. The random-teacher and LavaGap results show uninformative or misleading guidance can hurt, and SAGE is not designed to be robust in that regime. Mechanisms that score a proposed intervention before it is executed or cloned are flagged as important future work.
3. Extending beyond discrete actions. Preliminary experiments found current VLMs perform poorly when asked for precise low-level numerical actions such as torques or velocities; more realistic directions are high-level subgoals, skills, or action abstractions, or replacing the teacher with a Vision-Language-Action model trained for continuous control.
4. Scaling the embodied evaluation. ALFWorld is reported as preliminary, with a smaller training budget than the controlled environments, and the oracle used for it does not always guarantee success. Larger-scale embodied experiments, stronger teachers, and more systematic comparisons to interactive imitation-learning baselines are called for.

Target Audience

Reinforcement learning researchers working on sparse-reward exploration and sample efficiency; VLM and foundation-model researchers interested in using large models as teachers rather than policies; and applied machine learning engineers who want the reasoning and perceptual priors of a large model without paying for it at every decision step. Readers studying imitation learning, DAgger-style interactive supervision, or learning from imperfect demonstrations will find the ablation and teacher-quality sections most relevant, while practitioners with limited RL background can still follow the core idea — ask only when unsure, then learn to act alone.

Authors’ abstract

Vision-Language Models (VLMs) provide useful priors for interactive decision-making, but using them directly as policies is expensive and brittle: they must be queried at every step, do not improve from environment interaction, and can repeat systematic errors. We study how to learn a cheap autonomous policy from an online, expensive, and imperfect but informative VLM teacher. We propose SAGE (Selective Agent Guidance via Entropy), a framework that queries a VLM only when the learner is uncertain, executes the suggested action during training, and distills guidance into a lightweight Reinforcement Learning (RL) policy. Because VLM advice is not always reliable, SAGE can weight teacher-action distillation using environment-derived advantages rather than treating all suggestions as equally useful. Across sparse-reward visual reasoning and navigation tasks, SAGE learns policies that act without VLM guidance at evaluation time and improves over unguided RL in several environments, including settings where the learned policy exceeds its VLM teacher. The results show that selective guidance is most beneficial when the VLM can help the agent discover high-reward trajectories, and less useful when unguided exploration already succeeds or teacher actions do not lead to informative experience. SAGE also reduces VLM usage by prompting the teacher only on a fraction of training steps and requiring no VLM calls at deployment. Overall, our results suggest that VLMs don't need to be used as fixed policies to be useful; they can instead act as temporary, imperfect sources of guidance whose value is tested and internalized through interaction.