Execution-Aligned Progressive Noise for Consistent Asynchronous Replanning in Generative Robot Policies

AI summary

Overview

Research area: Robotics — generative robot policies, action chunking, and asynchronous real-time replanning.

Technical level: Advanced. The paper assumes familiarity with Flow Matching, first-order autoregressive processes, covariance structure of stochastic sources, and action-chunk execution protocols.

Scope: The paper introduces Execution-Aligned Progressive Noise (EAPN), a method for structuring the stochastic source of a generative robot policy so that consecutive replanning steps stay correlated with actual robot execution progress, and evaluates it on D3IL, Kinetix, LIBERO, and two real-world manipulation tasks.

What This Paper Is About

Generative robot policies generate an entire future action sequence ("action chunk") at once, but inference is slower than control, so the robot keeps moving while the next chunk is being generated. Because each new chunk normally starts from freshly sampled, independent Gaussian noise, two consecutive chunks can each be individually plausible yet belong to different behavioral modes, causing the robot's motion to silently drift or switch strategy across chunk boundaries. The paper argues that the problem is not only in action space but in the stochastic state from which each generation begins, and proposes keeping that state temporally linked across replanning events.

Key Contributions

1. Execution-aligned latent correlation. Using Flow inversion, the authors show that neighboring action chunks in a standard policy exhibit correlated initial noise whose correlation peak shifts with the actual execution displacement (rather than sitting at a fixed token index) and decays as the temporal separation between windows grows.

2. Joint inter- and intra-chunk stochastic modeling. EAPN recursively propagates a shared noise trajectory across consecutive inferences while also introducing temporal correlation along action time within each chunk, reducing stochastic drift and unnecessary mode switching caused by independent reinitialization.

3. History-conditioned noise representation. The aligned previous source is converted into an explicit conditioning signal injected into the policy, alongside a validity mask that marks positions with no historical counterpart (episode starts or invalid history), so the model does not assume historical context is always available.

4. Evaluation across simulation and real robots. Experiments on D3IL, Kinetix, LIBERO, and two real-world tasks (Object Storage and bimanual Cloth Folding) demonstrate improved task performance, multimodal consistency, cross-chunk continuity, and robustness to inference delays.

Main Findings

• Latent correlation exists even in a policy trained with independent noise. On π0.5 with token-wise independent Gaussian sources, the same-episode 5-NN hit rate in the raw inverted-noise space is 100%; mean cosine similarity is 0.774 within the same episode versus 0.043 across different episodes and 0.000 for an independent Gaussian reference. As the observation interval grows from 5 to 45 frames, within-episode similarity falls continuously from 0.855 to 0.378 (16 episodes, 95% bootstrap confidence intervals).

• The correlation peak follows execution displacement, not token index. For actual execution displacements D = 1, 5, 9, the correlation peak in the inverted latent follows j − h = D rather than a fixed same-index correspondence.

• D3IL: better success and fewer mode switches. On the D3IL Avoiding task with DDPM-ACT, EAPN raises success from 73.13% (351/480) to 78.96% (379/480), a gain of 5.83 percentage points, while reducing Mode Switch Rate from 3.969% to 3.167% and Switch Gap from 0.001913 to 0.001672 — without collapsing the policy to a single behavior.

• Kinetix: 88.59% average success. EAPN achieves an average success rate of 88.59% across the 12 Kinetix tasks under the asynchronous protocol with prediction horizon H = 8 and committed/inference-prefix lengths D ∈ {0, 1, 2, 3, 4}. Performance degrades more gradually than baselines as either the execution horizon K or the delay D increases.

• Ablation: inter-chunk noise drives the gain. On Kinetix mjc_swimmer, starting from ttRTC at 46.60 success, adding Inter-Chunk Noise raises success to 76.89 and reduces Switch Gap (1.58 to 1.55) and Tail Gap (1.58 to 1.51). Adding Intra-Chunk Noise reaches 78.85 with the lowest gaps (1.54, 1.50), and full EAPN reaches the best success of 79.26 with comparable continuity.

• LIBERO low delay: consistently high success. With π0.5 as the base policy and D ∈ {1, 2, 3, 4}, EAPN maintains 96.4%–97.0% average success and exceeds VLASH by 4.9 points at D = 4. At D = 4, EAPN averages 96.6% across Spatial, Object, Goal, and Long suites, versus 91.7% for VLASH and 92.2% for ttRTC.

• LIBERO high delay: advantage grows with staleness. With D ∈ {5, 10, 15}, EAPN reaches 83.0% and 75.0% average success at D = 10 and D = 15, improving over ttRTC by 2.7 and 2.0 points respectively. At D = 5, ttRTC averages 89.8% versus EAPN's 88.3%.

• Real-world: 90.0% on Object Storage. EAPN achieves 90.0% (18/20) success, improving over Naive Async (65.0%, 13/20) and VLASH (85.0%, 17/20) by 25.0 and 5.0 percentage points; RTC and ttRTC both reach 80.0% (16/20).

• Real-world: 96.7% on bimanual Cloth Folding. EAPN achieves 96.7% (29/30) success, compared with 93.3% (28/30) for VLASH, 63.3% (19/30) for Naive Async, 63.3% (19/30) for ttRTC, and 60.0% (18/30) for RTC. In a representative sequence, the baseline requires 56 s while EAPN completes the task in 33 s.

• No response lag or motion instability. Velocity and acceleration profiles over 0–5 s show EAPN comparable to ttRTC while avoiding some larger fluctuations seen in RTC and VLASH, indicating the correlated stochastic states do not prevent prompt reaction to garment deformation and the latest observation.

Methodology in Plain English

The core idea. Instead of giving every new action chunk a fresh, unrelated random starting point, the method gives it a random starting point that is deliberately related to the one used for the previous chunk — and related in the way the robot actually moved in between.

Two levels of correlation.

• Across chunks: a single shared noise trajectory persists across policy calls. Because the robot keeps executing while the next chunk is being generated, the shared trajectory is re-indexed by the actual execution displacement (how many actions have been consumed since the last inference), so the strongest dependency lands where the physical windows actually overlap. If E_k actions execute before the next inference starts and d_k more are consumed during inference, the displacement is Δ_k = E_k + d_k.
• Within a chunk: a second, private noise process runs along action time inside the chunk, so noise tokens within one chunk are also temporally related rather than independently sampled.

The math in brief. Both processes are first-order autoregressive (AR(1)) processes with Gaussian noise, scaled so that each source token still has a standard Gaussian marginal. A mixing weight γ balances the shared history against window-specific innovation, and a decay parameter ρ controls how quickly correlation fades with temporal distance. The resulting covariance between consecutive chunks is γ ρ^{|Δ_k + h − j|} I, whose maximum sits at the execution-aligned position rather than at a fixed index.

Making the policy aware of its own history. Correlated noise alone does not tell the policy which part of the previous noise corresponds to the current window, so the previous source is shifted by the same displacement and passed through a conditioner that also receives the displacement, the current inference delay/committed-prefix length, and a validity mask. This conditioning representation is injected into the velocity field at every flow step. Crucially, because the full joint covariance of the H × D_a source changes (only the single-token marginal is preserved), the policy is trained to adapt to this structured source rather than used as a training-free sampler.

Training and deployment. During training, delay and displacement are sampled from the evaluation support, and paired sources with matching execution geometry are generated from the shared process. With probability p_h the entire history is dropped and the current source is resampled independently with the history and mask zeroed, so the model does not assume history is always available. Committed prefix positions are written directly into the state and treated as fully noised, and the Flow Matching loss is applied only to the generated suffix. At deployment, an episode reset clears the buffer and disables history; thereafter the shared trajectory advances by the number of actions actually consumed and the aligned history conditions every flow step while the first D_k actions stay fixed. The base velocity-field objective and the original action-chunk execution logic are unchanged.

Evaluation design. The authors first run an inversion analysis on π0.5 to establish the empirical premise. Then they test multimodal consistency on D3IL Avoiding (which has interpretable behavioral modes), robustness to dynamic inference delay on Kinetix (a Jax2D-based 2D physics-control benchmark), delay robustness at both low and high delay regimes on LIBERO using π0.5, and finally two real-robot tasks — Object Storage (216 training episodes, 20 independent 60 s trials per method) and bimanual Cloth Folding (3,056 training episodes, three garments, 10 rollouts each, 30 trials per method, 3 min execution limit).

Why This Matters

Impact on research. Most prior work on asynchronous replanning constrains the actions or the executed prefix — RTC and ttRTC use a committed action prefix, VLASH and FutureRTC estimate the execution-time context, Legato/SEAM/Soft RTC use learned dynamics, velocity guidance, or soft action priors, and BID/Self-Guided Action Diffusion/ChunkFlow/FocalPolicy/POTR improve continuation through candidate selection, guidance, or overlap constraints. EAPN identifies a distinct design axis: the stochastic state from which each generation begins. It shows that this axis can be modeled explicitly and complements action-space alignment, opening a new line of work on source priors for generative robot control (related in spirit to PYoCo's correlated noise prior for video, but with an execution-aligned dependency instead of a fixed correlation structure).

Real-world applications.

• Warehouse and logistics picking: repeated grasp-and-place cycles of objects with varied appearance, size, and geometry, where per-cycle mode switching would cause dropped or misplaced items.
• Deformable-object handling: garment folding and similar bimanual tasks where nonrigid deformation and long temporal dependence make cross-chunk inconsistency especially costly.
• High-latency VLA and world-action model deployment: settings where inference is slow relative to control and old chunks must keep executing for a long time.
• Household and service robotics: long-horizon manipulation where motion direction, end-effector state, contact relations, or manipulation semantics can accumulate errors across successive chunks.

Industry relevance. The method is designed to be dropped onto existing flow-based policies: the base velocity-field objective and the original action-chunk execution logic are left unchanged, and only the joint temporal source structure plus a conditioning representation are modified. For companies running high-latency vision-language-action models on real hardware, that means robustness to inference delay can potentially be improved without replacing the policy architecture or retraining the whole stack from scratch. The reported low-delay LIBERO numbers (96.4%–97.0% average) suggest little cost on favorable hardware, while the high-delay numbers (75.0% at D = 15) suggest real benefit when the hardware is the bottleneck.

Future Directions

1. Adaptive or learned correlation parameters. The method exposes ρ (correlation decay) and γ (shared-history versus window-innovation balance) as fixed scalars. Whether these should be tuned per task, per delay regime, or predicted online from observed execution remains open.

2. Extending beyond AR(1) structure. The paper notes that the structured source changes the full joint covariance of the H × D_a source while preserving only the single-token Gaussian marginal. Whether richer or learned source distributions produce further gains is untested.

3. Delay regimes beyond the tested ranges. LIBERO is evaluated with D ∈ {1, 2, 3, 4} and D ∈ {5, 10, 15}, and Kinetix with D ∈ {0, 1, 2, 3, 4} at horizon H = 8. Behavior at much larger or highly variable delays — and the interaction between D_k (committed prefix) and Δ_k (stochastic-state advancement), which the paper states need not coincide — is not characterized.

4. Broader and larger-scale real-robot validation. Real-world results cover two tasks with 20 and 30 trials per method. Scaling to more objects, garments, embodiment types, and longer horizons would test whether the reported consistency advantages hold generally.

Target Audience

Robotics and embodied-AI researchers working on generative policies, diffusion/flow-matching action generation, and asynchronous real-time control; engineers deploying high-latency vision-language-action or world-action models on real hardware who need cross-chunk continuity under inference delay; and graduate students with a background in generative modeling and autoregressive processes who want a concrete example of treating the sampling source — not just the sampled output — as a design lever in robot learning.

Authors’ abstract

Continuous asynchronous replanning is essential for real-time generative robot policies, but independent stochastic initialization can cause mode switching and inconsistent continuation across action chunks. We propose Execution-Aligned Progressive Noise (EAPN), which introduces structured stochasticity at both inter-chunk and intra-chunk levels. Across replanning steps, EAPN propagates a shared noise trajectory and aligns it with the actual execution displacement, establishing execution-aligned inter-chunk correlation. Within each action chunk, it models temporal correlation along action time. The aligned stochastic history is further combined with committed action context to condition subsequent generation, allowing new chunks to continue from execution-consistent generative states rather than restart from independent noise. We evaluate EAPN on D3IL, Kinetix, LIBERO, and real-world manipulation tasks. EAPN improves multimodal behavior consistency on D3IL and achieves an average success rate of 88.59% on Kinetix. On LIBERO, it remains robust and maintains strong task performance even under long inference delays. Real-robot experiments further achieve 90.0% success on Object Storage and 96.7% on bimanual Cloth Folding, demonstrating reliable continuous execution under asynchronous replanning.