Data-Centric Recursive Self-Improvement for Navigation via Policy-Synchronized Language-Action Dual Bootstrapping

AI summary

Overview

Research area: Computer Vision / Embodied Vision-and-Language Navigation (VLN), specifically continuous-environment navigation (VLN-CE) with vision-language-action models.

Technical level: Advanced. The paper assumes familiarity with VLN benchmarks (R2R-CE, RxR-CE), success-rate/SPL metrics, imitation-style behavior cloning over action chunks, and vision-language model fine-tuning.

Scope in one sentence: RSI-Nav is a single Qwen3.5-9B-based model that alternately writes route instructions for unlabeled navigation trajectories and then executes them to grade its own instructions, updating after every small shard of data so that generated supervision keeps pace with the improving policy.

What This Paper Is About

Training a navigation agent to follow language instructions requires large amounts of paired instruction-trajectory data, and human annotation is too costly. Prior work synthesizes instructions with a speaker model or data flywheel, but usually keeps the navigator and the instruction writer as separate models, and annotates a whole corpus with one frozen model before retraining — so the synthetic supervision drifts out of date as the navigator improves. The paper's goal is to close that gap by making one shared model generate instructions, execute them to check whether they actually lead to the right place, and immediately retrain on the survivors before moving to the next batch of unlabeled trajectories.

Key Contributions

1. A unified policy for three tasks. RSI-Nav uses one model, built on Qwen3.5-9B, for instruction following (vision-language-to-action), instruction generation (vision-action-to-language), and spatial understanding, distinguished only by task-specific prompts rather than separate generators, verifiers, or output heads.

2. Policy-synchronized recursive self-improvement. Unlabeled trajectories are split into non-overlapping shards; the current model generates an instruction per trajectory, re-executes it from the original start state, keeps the pair only if the goal is reached within 3 m and SPL clears a threshold, and updates immediately before the next shard — keeping supervision aligned with the evolving model rather than a fixed annotator.

3. Execution-verified data at scale. The process yields about 2.5M instruction-following samples and 1.1M instruction-generation samples, produced and filtered by the model's own rollouts.

4. The paper states this is the first navigation framework to combine unified language-action dual learning with policy-synchronized recursive self-improvement, supported by ablations on update granularity, dual-task training, action conditioning, and spatial supervision.

Main Findings

• Navigation results: RSI-Nav reaches 68.6% SR and 58.6% SPL on R2R-CE Val-Unseen (NE 3.70, OSR 78.0), and 72.4% SR, 56.5% SPL, and 70.6 nDTW on RxR-CE Val-Unseen (NE 3.53). These are marked in bold in the paper's main table as the best among the compared methods — for example, DualVLN reports 64.3% SR / 58.5% SPL on R2R-CE and AwareVLN 65.4% SR / 55.1% SPL.

• Large gain over supervised initialization: the cold-start checkpoint scores 52.9% SR / 45.5% SPL on R2R-CE and 54.7% SR / 48.8% SPL / 62.4 nDTW on RxR-CE, so recursive evolution adds roughly 15.7 points of SR and 13.1 points of SPL on R2R-CE and 17.7 points of SR on RxR-CE.

• Route fidelity, not just goal reaching: the RxR-CE nDTW rise from 62.4 to 70.6 is interpreted as the model better following the instructed route rather than reaching the goal by an alternative path.

• Spatial understanding is preserved and slightly improved: VSI-Bench 64.6 to 64.9, ScanQA 93.2 to 96.4, SQA3D 65.1 to 65.4, MindCube 91.8 to 93.5 — all described by the authors as modest gains over cold start.

• Generated instructions are more useful than their NLP scores suggest: although SRDF scores higher on most text-similarity metrics (BLEU-1 75.3 vs 71.0, BLEU-4 31.1 vs 28.2, METEOR 25.0 vs 23.2, ROUGE 51.4 vs 48.5, SPICE 26.2 vs 20.1; RSI-Nav leads on CIDEr, 50.6 vs 49.2), training identical navigation models on RSI-Nav's instructions gives 41.3% SR / 35.1% SPL versus 36.8% SR / 30.4% SPL for SRDF's.

• Dual learning helps navigation almost free: training on navigation plus vision-action-to-language data raises SR from 44.7% to 48.8% and SPL from 38.6% to 41.0% over navigation-only training, while instruction-generation metrics stay comparable (BLEU-1 71.7 to 71.0, CIDEr 49.8 to 50.6 when both directions are trained together).

• Actions matter in the generation direction: with the same visual input, adding action conditioning (VA2L) beats images-only (V2L), lifting SR from 47.0% to 48.8% and SPL from 38.8% to 41.0%.

• Spatial supervision compounds the gains: adding spatial-understanding data on top of VA2L moves NE from 5.69 to 5.54, OSR from 59.9 to 66.4, SR from 48.8% to 52.9%, and SPL from 41.0% to 45.5.

• Update granularity has a sweet spot: on a fixed 64K-trajectory subset with 64 GPUs, updating after 256 candidates per GPU performs best (SR 57.2, SPL 48.7); 64 candidates is too frequent (SR 51.9, SPL 45.9) and 1,024 candidates too infrequent (SR 55.1, SPL 46.3), which is why the full run uses 256 per GPU — a global shard of 16,384 trajectories.

• Action wording matters: single-word actions ("forward/left/right/stop") beat arrow symbols and two-word phrasing, with SR 48.72 vs 43.28 vs 43.83 and SPL 42.67 vs 37.80 vs 39.22 on R2R val-unseen.

• Real-world demonstration: the model was deployed on a DEEP Robotics M20 robot with a remote NVIDIA GeForce RTX 4090, and the paper reports qualitative success on obstacle avoidance, narrow-space traversal, multi-step instruction following, open-vocabulary landmark grounding, room-to-room and outdoor navigation — presented as qualitative results, not scored benchmarks.

Methodology in Plain English

Start with a strong vision-language model and fine-tune it, in one epoch on 64 GPUs with a learning rate of 5×10⁻⁶ and batch size 128, on a mix of existing navigation data (SRDF, EnvDrop, ScaleVLN, R2R, RxR), reversed instruction-generation pairs, and spatial question-answering data (VSI-590K, 546K LLaVA-3D samples, MindCube). This "cold start" gives one model that can both drive and describe.

Then turn it loose on a big pool of trajectories that have no instructions. Chop the pool into shards. For each shard:

1. Write. The model looks at 32 sampled frames plus the full action sequence of a trajectory and writes a natural-language route instruction for it.
2. Drive. The model is placed back at the original start state and told to follow that instruction, using 8 sampled frames as visual context and predicting actions in chunks of four, choosing among FORWARD, LEFT, RIGHT, STOP.
3. Grade. If the run ends within 3 m of the trajectory's true endpoint and its SPL clears a threshold, the instruction is accepted as a faithful description.
4. Train twice on each accepted pair. The instruction becomes the language input for a navigation example over the original trajectory, and the trajectory becomes the input for an instruction-generation example producing that instruction. The original trajectory is kept rather than the executed one, so execution errors don't accumulate and the work stays focused on improving the language supervision.
5. Refresh. Spatial-understanding data from the cold start is replayed alongside these samples, the model is updated, and the new model handles the next shard.

Because step 5 happens after every shard rather than after the whole corpus, the annotator grading the data is always roughly the model being trained — the "policy-synchronized" property the paper names.

Why This Matters

Research impact. The paper reframes synthetic instruction data as a moving target rather than a one-off offline asset, and shows that a single set of weights can hold both halves of the language-action loop without task-specific heads. It also argues, with evidence, that text-similarity metrics like BLEU and SPICE are poor proxies for whether generated instructions are useful for training — a methodological point that applies well beyond navigation.

Real-world applications:

• Service and delivery robots that must learn from their own logged routes in buildings where nobody has written instructions.
• Warehouse and hospital logistics, where map changes outpace manual annotation of wayfinding language.
• Assistive navigation for visually impaired users, where spoken route descriptions and route execution must agree with each other.
• Home robots that need to turn a demonstrated walk-through into a verbalizable, reusable skill.

Industry relevance. The pipeline converts cheap unlabeled robot experience into training signal with only execution-based filtering as the check, which is attractive where annotation budgets are the bottleneck. The demonstrated M20 deployment and the finding that wording choices (single-word actions) and update frequency materially change results are practical engineering lessons for teams building embodied agents.

Future Directions

• Breaking the dependence on simulation and an offline pool. The authors list this as a limitation: the method still relies on simulated environments and a pre-collected trajectory pool, so online or in-the-wild continual improvement is untested.
• Reducing the cost of execution-based filtering. Every candidate instruction requires a full rollout in the environment, which the paper calls considerable computational cost; cheaper verification is an open problem.
• Establishing systematic real-world robustness. Only qualitative real-world results are given, so generalization to diverse environments, unseen domains, and substantially longer horizons remains open.
• Pushing the spatial-understanding gains further. Spatial scores improved only modestly (for instance VSI-Bench 64.6 to 64.9), leaving room to ask how much spatial supervision the loop can absorb before navigation gains saturate.

Target Audience

Researchers and engineers working on embodied AI, vision-language-action models, and vision-and-language navigation who are already comfortable with VLN-CE benchmarks and imitation-based policy training. It is most useful to those focused on data engines and self-improving training loops — people deciding how to generate, filter, and schedule synthetic supervision — rather than readers looking for a new navigation architecture. The ablation on update granularity and the instruction-usefulness comparison make it directly actionable for practitioners building their own data flywheels.

Authors’ abstract

Vision-language navigation requires agents to follow natural-language instructions to reach target locations, making high-quality language supervision essential. Since human annotation is costly, prior work has explored scalable data synthesis methods. However, existing methods typically either use separate models for navigation and instruction generation or couple the two tasks through task-specific components; most also annotate the full trajectory corpus with a fixed model before updating, which can cause generated supervision to lag behind the evolving policy. Therefore, we introduce RSI-Nav, a data-centric recursive self-improvement framework for navigation via policy-synchronized language-action dual bootstrapping, which unifies vision-language-to-action instruction following and vision-action-to-language instruction generation within a shared model. Specifically, after supervised fine-tuning establishes fundamental capabilities, RSI-Nav progressively processes unlabeled trajectories in small shards. The current model generates instructions for these trajectories, executes them to evaluate goal completion and path efficiency, retains qualified samples, and immediately updates itself before processing the next shard. This shard-wise update keeps newly generated supervision aligned with the evolving policy. We also incorporate spatial understanding data, since stronger spatial reasoning benefits navigation. Extensive experiments show that RSI-Nav reaches 68.6% SR on R2R-CE and 72.4% SR on RxR-CE, while providing stronger instruction-generation utility and modestly improved spatial understanding after recursive evolution.