COVR:Collaborative Optimization of VLMs and RL Agent for Visual-Based Control

AI summary

Overview

• Research area: Visual reinforcement learning (RL), vision-language model (VLM)-assisted decision making, and continuous control. The paper sits at the intersection of computer vision, multimodal foundation models, and deep RL.
• Technical level: Advanced. The paper assumes familiarity with Soft Actor-Critic (SAC), Markov Decision Processes, experience replay, supervised fine-tuning, LoRA, entropy regularization, and Z-score/IQR statistics.
• Scope: A single paper proposing and empirically validating a bidirectional training loop in which a VLM (Qwen2.5-VL-3B) and a visual RL policy (SAC-based) improve each other, evaluated on CARLA driving scenarios and six DMControl robotic control tasks.

What This Paper Is About

Visual RL is sample-inefficient because policies must learn from high-dimensional image observations, which makes exploration slow and full of wasted interactions. Prior work tries to fix this by transferring knowledge one-way from a vision-language model into an RL policy, usually by distilling a frozen VLM or by fine-tuning the VLM and using it as the policy itself. COVR argues this is a missed opportunity: the interaction data that RL generates can also be used to improve the VLM, and the improved VLM can then give better guidance back to the RL policy — a loop rather than a one-way transfer.

Key Contributions

1. A collaborative optimization framework for VLM-assisted RL in which the VLM is fine-tuned on RL-generated trajectories and the resulting VLM supplies action priors that regularize the RL actor. The authors claim this helps even when the assisting VLM is initially weak.
2. The Exploration-Driven Dynamic Filter (EDDF) module, which selects high-quality trajectory samples for VLM fine-tuning using a threshold that adapts to the RL agent's exploration level rather than a fixed top-k cutoff.
3. The Return-Aware Adaptive Loss Weight (RALW) module, which weights each fine-tuning sample by its normalized cumulative return so that inconsistent or low-return actions under similar observations are down-weighted or zeroed out.
4. An Adaptive Progressive Fine-Tuning strategy that increases the interval between VLM updates as the policy converges, combined with LoRA fine-tuning to cut resource consumption. Experiments on CARLA and DMControl are reported to validate the approach.

Main Findings

• CARLA #HW (Highway): COVR reaches 248 ± 81 episode reward (ER) and 259 ± 85 driving distance (DD), the best in Table 1. The strongest non-VLM baseline is ResAct at 227 ± 36 ER / 236 ± 40 DD, and the strongest VLM-assisted baseline is DGC at 208 ± 13 ER / 234 ± 15 DD.
• CARLA #GP (Ghost Pedestrian): COVR reaches 235 ± 89 ER and 237 ± 89 DD, again best in Table 2. ResAct is the best non-VLM baseline at 212 ± 54 ER / 216 ± 55 DD, and DGC is the best VLM-assisted baseline at 146 ± 14 ER / 169 ± 18 DD.
• The VLM alone is weak: The VLM-based executor baseline (VBE) scores -11 ± 5 ER / 11 ± 4 DD on #HW and 9 ± 7 ER / 18 ± 7 DD on #GP, which the authors attribute to a base VLM's inability to handle continuous, dynamic environments.
• DMControl gains over multiple base algorithms: Adding COVR improves SAC, DeepMDP, and RAD on all six reported tasks. For example, RAD goes from 694 ± 28 to 872 ± 2 on Cartpole Swingup, from 734 ± 87 to 969 ± 18 on Reacher Easy, and from 364 ± 38 to 504 ± 13 on Cheetah Run.
• State-of-the-art on DMControl: COVR records 872 ± 2 (Cartpole Swingup), 969 ± 18 (Reacher Easy), 504 ± 13 (Cheetah Run), 802 ± 25 (Walker Walk), 976 ± 9 (Finger Spin), and 960 ± 23 (Ball in cup Catch). The authors also highlight that the reward standard deviation is notably lower than competing methods, which they read as better convergence stability. These values are identical to the RAD+COVR row in the baseline table, since RAD+COVR is the configuration reported as COVR.
• Ablation — EDDF matters: Removing EDDF and randomly filtering data drops performance to 144 ± 74 ER / 155 ± 77 DD. Fixed top-k filtering (top-80%, top-90%, top-95%) yields 204 ± 137 / 214 ± 141, 217 ± 95 / 229 ± 98, and 192 ± 107 / 203 ± 110 respectively, all below the full COVR.
• Ablation — design choices in EDDF: Removing Z-score standardization gives 210 ± 99 / 221 ± 102; replacing cumulative return with per-step reward gives 221 ± 105 / 231 ± 108; replacing return with the critic's Q-value gives 200 ± 118 / 212 ± 122.
• Ablation — RALW matters: Removing RALW gives 204 ± 111 / 214 ± 115, and replacing return-based weights with randomly generated weights gives 184 ± 78 / 191 ± 81.
• Ablation — VLM guidance cannot be replaced by simple data curation: Selecting top-return samples (top-80% and top-50%) and mixing in random actions to balance exploration yields 183 ± 96 / 195 ± 99 and 175 ± 58 / 185 ± 58, both well below COVR.

Methodology in Plain English

The system runs two processes side by side.

On the RL side, a standard SAC agent with a visual encoder takes RGB frames and outputs continuous actions. These actions are what actually drive the environment during training. On the VLM side, a Qwen2.5-VL-3B model looks at the same current frame plus a task prompt, describes what action to take in text, and the text is parsed into a numeric action vector. Rather than letting the VLM act, the paper adds a penalty term to the actor's loss that pulls the RL action toward the VLM's suggested action, weighted by a coefficient (λ = 2.0 in the experiments). At test time the VLM is dropped entirely and only the RL actor is used.

To make the VLM better, the framework collects trajectories into a separate buffer that stores observations, RL actions, and the cumulative discounted return of each step. Three things then happen:

First, filtering. Immediate rewards are noisy and Q-values can be biased, so the paper uses trajectory-level cumulative return as the quality signal. These returns are Z-score normalized. A threshold is computed as the median of the normalized returns plus a Sigmoid of the agent's current policy entropy times the interquartile range. Because entropy is high early in training, the threshold is low and more samples survive, including low-return but informative exploration samples. As entropy falls and the policy stabilizes, the threshold tightens and only high-return samples are kept.

Second, weighting. The selected returns are normalized to the range [-1, 1]. Samples with negative returns get zero weight, so they do not corrupt the VLM. Positive-return samples get weights proportional to their normalized return, applied to a negative log-likelihood autoregressive loss with label smoothing. The intent is to stop the VLM from being confused by the fact that RL exploration sometimes produces very different actions for visually near-identical states.

Third, scheduling. Fine-tuning the VLM on every batch would be wasteful, so updates are triggered only after a growing number of RL steps. The step size grows as ψ(c+1) = ψ(c) + ψ(c)·c with initial ψ₀ = 5000, and the filtered buffer is cleared after each update. LoRA is used so only a small set of parameters is trained.

For CARLA, the SAC backbone is augmented with a DeepMDP transition network that predicts next state and reward; in DMControl, COVR is built on RAD. The VLM is invoked once every 10 frames to limit inference latency. Training runs for 100,000 steps per benchmark, with metrics reported over 10 evaluation episodes across three seeds, each episode capped at 1000 steps. A cold-start strategy delays VLM-based policy guidance until after two rounds of fine-tuning in the #HW scenario.

Why This Matters

• Shifts the framing from distillation to collaboration. Prior VLM-assisted RL pipelines treat the VLM as a fixed source of knowledge. COVR demonstrates empirically that feeding RL experience back into the VLM, then using the improved VLM as a policy prior, outperforms both frozen-VLM distillation methods (DPL, APL, DGC, VPF) and fine-tuned-VLM-as-policy methods (VBE) on the reported benchmarks.
• Addresses a real failure mode in practice. The RALW ablation shows that naively supervised fine-tuning on RL data is harmful — random or return-ignoring weighting underperforms. The paper's contribution is not just "use RL data to train VLMs" but a specific mechanism for handling the inconsistency inherent in stochastic exploration.
• Keeps deployment cheap. The VLM is used as a training-time supervisor only; at inference time, only the RL actor runs. This directly answers a cost complaint the paper raises against methods that fine-tune a language model into the policy network.
• Real-world applications:
  • Autonomous driving and advanced driver assistance, the direct target of the CARLA Highway and Ghost Pedestrian evaluations.
  • Robotic locomotion and manipulation, represented by the DMControl tasks (Cheetah Run, Walker Walk, Reacher Easy, Finger Spin, Ball in cup Catch, Cartpole Swingup).
  • Other simulated continuous-control domains where visual observations are high-dimensional and real-world trial-and-error is expensive — the paper's introduction also names game simulation.
  • Any industrial or embodied setting where a small, deployable RL policy must be trained under a compute budget while still benefiting from foundation-model priors.
• Industry relevance: The combination of a small open VLM (3B parameters), LoRA adapters, and a test-time actor-only pipeline is designed for teams that cannot afford to serve a large multimodal model in the control loop. The training-time-only VLM usage pattern is also attractive where latency budgets are tight.

Future Directions

• Extending to harder control tasks. The paper states results for "other hard tasks" in DMControl are in the Appendix, but that appendix content is truncated in the provided text, so the full scope of the hard-task evaluation is not reported here.
• Scaling and swapping the VLM. Only Qwen2.5-VL-3B is tested. Whether the collaborative loop still pays off with larger or differently pretrained VLMs, and whether the gains shrink as the base VLM improves, is left open.
• Reducing VLM invocation cost further. The current pipeline invokes the VLM once every 10 frames; the progressive fine-tuning schedule reduces update frequency but not inference frequency, which remains a per-step bottleneck.
• Generalizing beyond two-dimensional continuous action spaces. The action interface relies on a string-to-float parsing function mapping the VLM's textual output to a continuous action, and the evaluated environments all use [steer, throttle] or DMControl's continuous actuators. Discrete or higher-dimensional action spaces are not addressed.
• Theory of the collaborative loop. The paper offers empirical evidence that the bidirectional loop helps but does not characterize when the RL-generated data becomes reliable enough for VLM fine-tuning, nor how to detect or correct a divergence between the two components.

Target Audience

• RL and embodied-AI researchers working on sample efficiency, visual representation learning, or integrating foundation models into decision-making pipelines.
• Practitioners building autonomous driving or robotic control systems who want to know whether VLM priors can be converted into deployable, low-latency policies.
• Graduate students and engineers with prior exposure to SAC and multimodal models who want a concrete template for bidirectional VLM-RL training, including the sample-selection and loss-weighting details.
• Researchers studying data curation for fine-tuning foundation models, since the EDDF and RALW modules are presented as general mechanisms for filtering and weighting noisy, return-labeled trajectories.

Authors’ abstract

Visual reinforcement learning (RL) suffers from poor sample efficiency due to high-dimensional observations in complex tasks. While existing works have shown that vision-language models (VLMs) can assist RL, they often focus on knowledge distillation from the VLM to RL, overlooking the potential of RL-generated interaction data to enhance the VLM. To address this, we propose COVR, a collaborative optimization framework that enables the mutual enhancement of the VLM and RL policies. Specifically, COVR fine-tunes the VLM with RL-generated data to enhance the semantic reasoning ability consistent with the target task, and uses the enhanced VLM to further guide policy learning via action priors. To improve fine-tuning efficiency, we introduce two key modules: (1) an Exploration-Driven Dynamic Filter module that preserves valuable exploration samples using adaptive thresholds based on the degree of exploration, and (2) a Return-Aware Adaptive Loss Weight module that improves the stability of training by quantifying the inconsistency of sampling actions via return signals of RL. We further design a progressive fine-tuning strategy to reduce resource consumption. Extensive experiments show that COVR achieves strong performance across various challenging visual control tasks.