From Failures to Learnable Worlds: Policy-Conditioned Environment Generation for Navigation Agents

AI summary

Overview

• Research area: Robotics / embodied AI — procedural environment generation for training and adapting embodied navigation agents (ObjectNav), combining LLM-guided failure diagnosis with 3D scene construction.
• Technical level: Intermediate. The paper assumes familiarity with reinforcement learning, imitation learning, and embodied navigation benchmarks, but the core idea (mine a policy's failure modes, then generate environments that target them) is explained without heavy math.
• Scope (1 sentence): The paper introduces Wedge, a framework that diagnoses a pretrained navigation policy's specific weakness from its own successful and failed rollouts, then generates ProcTHOR training environments that target that weakness, and evaluates the resulting adaptation on three agents (DD-PPO, Uni-NaVid, Gemini-3.8-flash) in ProcTHOR and HM3D.

What This Paper Is About

Most environment-generation pipelines for embodied agents optimize generic properties such as realism, diversity, or task coverage, which means the same generated environment may be redundant for one policy and too hard for another. This paper argues that training environments should be generated relative to the current policy, by contrasting its successful and failed episodes to find the environmental conditions that systematically predict failure. The goal is to construct environments that sit in a policy's specific weak regime while still being solvable, and to show that adapting on them improves performance both in the generation simulator and beyond it.

Key Contributions

1. A policy-conditioned environment generation framework (Wedge) that derives its generation targets from a pretrained policy's own success/failure patterns within the same task, rather than from a predefined set of tasks or skills.
2. An LLM-guided weakness discovery and validation loop: an LLM proposes executable weakness features from contrastive successful/failed rollouts, and each candidate is retained only if it passes sequential checks for coverage, non-redundancy, predictiveness, and incremental value, with failed candidates triggering new episode sampling and re-proposal.
3. Weakness-conditioned environment construction with rollout calibration: validated weaknesses are mapped to controllable simulator factors (target re-placement, new start states, new layouts), candidates are kept only if their feature value falls in the diagnosed weak regime, and the frozen policy is rolled out five times per candidate with only episodes succeeding 2–3 times retained.
4. An empirical study across three heterogeneous agents and two simulators, showing distinct per-policy weakness profiles, gains over matched random and hard-example baselines on ProcTHOR, transfer to HM3D for Uni-NaVid and Gemini, and iterative re-diagnosis across adaptation rounds.

Main Findings

• Different policies fail for different reasons on the same task. The ProcTHOR-trained DD-PPO policy's top weakness is target_visibility (ΔSR = 28.5), with doorway_visibility also identified as complementary; Uni-NaVid's top weakness is blind_path_ratio (ΔSR = 37.3) followed by room_hops (ΔSR = 34.8); Gemini's is also blind_path_ratio (ΔSR = 37.6).
• Weakness-conditioned adaptation beats matched baselines on ProcTHOR. DD-PPO: pretrained 82.1 SR / 58.2 SPL, random 82.3 / 58.8, hard-mined 82.4 / 58.1, Wedge round 1 85.4 / 59.8, round 2 85.6 / 59.8. Uni-NaVid: pretrained 51.6 / 35.3, random self-rollout 46.6 / 34.1, hard self-rollout 53.1 / 36.9, Wedge 54.5 / 37.3 (round 1) and 55.7 / 38.5 (round 2). Gemini: zero-shot 41.0 / 31.8, random example 40.7 / 32.6, hard example 41.8 / 32.7, Wedge 43.7 / 33.4 (round 1) and 44.3 / 33.7 (round 2).
• Gains transfer to HM3D after ProcTHOR-only adaptation for two agents. Uni-NaVid: pretrained 69.2 SR / 28.5 SPL, random self-rollout 67.1 / 26.7, hard self-rollout 69.0 / 26.2, Wedge 71.4 / 28.4 (round 1) and 73.4 / 28.6 (round 2). Gemini: zero-shot 57.9 / 27.0, random example 58.0 / 27.9, hard example 60.9 / 28.7, Wedge 63.5 / 29.1 (round 1) and 64.2 / 30.2 (round 2). DD-PPO's HM3D numbers are not reported (shown as "–").
• Hard-example mining can improve in-domain while failing to transfer. Hard self-rollout raised Uni-NaVid's ProcTHOR SR to 53.1 but left HM3D SR at 69.0 versus 69.2 for the pretrained model.
• Rollout calibration matters as much as weakness matching. For DD-PPO, the calibrated subset is only 16% of weakness-matched environments (668 of 4,215) yet scores higher (85.4 vs. 84.4 SR), both above pretrained 82.1. For Uni-NaVid, increasing the share of hardest episodes from 12.1% to 25.1% lowered ProcTHOR SR from 49.0 to 47.2 and HM3D SR from 70.0 to 69.2.
• Generation adds value beyond mining existing episodes. For Uni-NaVid on ProcTHOR: random self-rollout 46.6 / 34.1, weakness-mined existing episodes only 49.2 / 34.9, weakness-generated only 51.3 / 35.4, mined plus generated 54.5 / 37.3.
• Gains concentrate in the targeted weak regime. DD-PPO improves most on small, hard-to-see targets (Bowl, Laptop, AlarmClock) while several larger objects change little; for Gemini, long-blind-path episodes improve by +7.0 SR with little or no gain on shorter paths.
• Dominant weaknesses can persist or shift across rounds. DD-PPO keeps target_visibility and effectively saturates after round 1; Uni-NaVid keeps blind_path_ratio while still improving from 54.5 to 55.7 SR; Gemini shifts from blind_path_ratio to target_room_remoteness.
• Discovery is cheap relative to search it saves. Diagnosis runs on already-collected rollouts on CPU with about ten LLM calls per agent; to obtain the same number of hard target placements, unguided hard-example mining evaluated roughly twice as many candidates (8.4k vs. 4.2k), corresponding to 42k vs. 21k policy rollouts.

Methodology in Plain English

The framework runs a loop with four stages.

1. Diagnose. Run the frozen policy on a held-out set of ObjectNav episodes and record rollouts and outcomes. Each discovery round samples 15 successful and 15 failed episodes and describes each one using a shared vocabulary of measurable primitives grouped into scene (rooms, doors, start-room type), target (category, target-room area, visibility), path (shortest-path length, room transitions, doors crossed, distance before the target is first visible), and agent (coverage, revisitation, whether the target room or target was reached). An LLM (Gemini-2.5-Pro) is shown the contrasting pairs and asked to propose candidate weakness features, each with a rationale and an executable Python function that computes it — for example blind_path_ratio, the ratio of first-visible path length to shortest-path length.

2. Validate. Each candidate is run over the full diagnostic set and checked in order: coverage (computable on at least 95% of episodes and not constant), non-redundancy (Spearman correlation below 0.85 with already-retained features), predictiveness (success-rate gap between bottom and top terciles of at least 10 percentage points, p < 0.05 on a two-proportion z-test, and a gap of at least 5 points on a held-out 30% split), and incremental value (at least 0.01 gain in five-fold cross-validated AUC for predicting success). A candidate that fails any check is discarded, new contrastive episodes are sampled, and the LLM proposes again. Survivors are called "mined weaknesses."

3. Generate. Take the weakness with the largest success-rate gap and identify the weak regime — the feature-value ranges where the policy's success rate drops below its overall diagnostic success rate. Modify the episode factors the feature depends on using three reusable operators: re-place the target, sample a new start state, generate a new layout. Keep only candidates whose feature value falls in the weak regime, and generate extra layouts when existing scenes yield too few. Then roll out the frozen policy five times per candidate and keep episodes with 2 or 3 successes, filtering out both near-solved and near-impossible cases.

4. Adapt. Preserve each agent's original learning mechanism so the method and baselines differ only in how training episodes are chosen. RL policies (DD-PPO) mix selected episodes with the original training distribution and continue training for 10M environment steps; imitation-based policies (Uni-NaVid) are fine-tuned with LoRA on successful self-rollouts; closed-weight models (Gemini) receive a successful trajectory as an in-context worked example. After adaptation, the same diagnosis can be re-run to find the next weakness.

Why This Matters

Impact on research: The paper reframes policy failures as a specification for data generation rather than only a diagnostic signal, and argues against policy-agnostic objectives like realism, diversity, or difficulty. It provides a validated pipeline (LLM proposal plus statistical filtering, not LLM judgment alone) and shows that matching environments to a policy's learning range can matter more than simply making them harder — a concrete counterpoint to hard-example mining, which improved in-domain ProcTHOR performance for Uni-NaVid but did not transfer to HM3D.

Real-world applications:

• Home robots and assistive navigation: targeting the specific conditions where a deployed robot fails, such as targets that are hard to see or routes requiring long stretches without line of sight, instead of retraining on generic scenes.
• Warehouse and logistics robots: generating layouts and start configurations that exercise a particular fleet policy's blind-path or room-transition weaknesses.
• Autonomous inspection and search-and-rescue in buildings: constructing simulated buildings around the failure regimes observed in field deployments, where collecting real failure data is expensive.
• Simulation-to-reality pipelines: using a diagnostic loop that re-runs after each deployment to decide which environments to synthesize next, including for closed-weight models adapted through prompting rather than retraining.

Industry relevance: The method reduces the search cost of finding useful training environments — roughly half the candidate evaluations and rollouts compared with unguided hard-example mining in the reported comparison — and it applies without parameter access to proprietary models via in-context examples, at a reported cost of about half a GPU-day plus around $110 in API cost for one Gemini adaptation round.

Future Directions

• Multi-weakness generation. The framework deliberately targets one weakness per adaptation round to isolate which factor drives gains; whether combining several weaknesses yields better or confounded adaptation is left open.
• Breaking saturation. DD-PPO's dominant weakness stayed the same and its SR and SPL effectively saturated after round 1, so how to escape a persistent weakness that targeted generation no longer improves is unresolved.
• Beyond object navigation. Diagnosis and generation are studied only in 3D indoor ObjectNav in ProcTHOR, so extending the mined-primitive vocabulary and operators to other embodied tasks or to non-procedural simulators remains to be tested.
• Richer adaptation formats for closed-weight models. The paper mentions deriving three navigation rules and a target-specific search prior from successful practice episodes as an alternative adaptation format for Gemini; how such formats compare with the single worked example is a natural next question.

Target Audience

Researchers and practitioners in embodied AI and robot learning who train or adapt navigation policies in simulation; engineers building procedural environment-generation or curriculum pipelines; and applied scientists using LLMs as tools for diagnosing model behavior. Readers interested in failure-driven learning, sim-to-real and cross-simulator transfer, or in adapting closed-weight foundation-model agents without weight updates will find the setup directly relevant.

Authors’ abstract

Simulated environments enable scalable training data generation for embodied agents, but more data does not necessarily provide what a particular agent needs to learn. Existing environment-generation pipelines largely optimize generic properties such as realism, diversity, or task coverage, while different pretrained policies can fail for different environmental reasons. We therefore introduce WEDGE, a framework for weakness-guided navigation learning in which the failure modes of a pretrained agent guide the generation of training environments at each adaptation round. Given a pretrained policy, WEDGE uses an LLM to propose executable weakness features from contrasting successful and failed rollouts, automatically validates them on the diagnostic set, and selects the strongest validated weakness based on its success-rate gap. The validated weaknesses are then mapped to the environmental factors that control them. WEDGE varies these factors to construct environments in the difficulty regimes where the policy struggles, producing targeted training experience for adaptation. Across three navigation agents with different adaptation mechanisms, we find that policies exhibit distinct weakness profiles and benefit from different training environments. Weakness-conditioned adaptation consistently outperforms matched random and hard-example alternatives on ProcTHOR, while the resulting gains for Uni-NaVid and Gemini also extend to HM3D after ProcTHOR adaptation. Iterative re-diagnosis further shows that dominant weaknesses can persist or shift as the policy improves, providing updated guidance for subsequent environment construction. Together, these results show that policy failures are not only diagnostic signals, but actionable guidance for constructing training environments that target a policy's current weaknesses.