Learning Native Continuation for Action Chunking Flow Policies

AI summary

Overview

• Research area: Robotics — real-world robot manipulation with Vision-Language-Action (VLA) policies, specifically action-chunked flow-matching generative policies and the problem of trajectory continuity at chunk boundaries.
• Technical level: Advanced. The paper assumes familiarity with flow matching, diffusion-style denoising, classifier-free/inpainting guidance, and asynchronous chunked robot execution.
• Scope: The paper proposes Legato, a training-time method that makes chunk-to-chunk continuation a native, learned property of a flow-based VLA policy rather than an inference-time correction, and validates it on five real-world dual-arm manipulation tasks against Real-Time Chunking (RTC) and training-time RTC.

What This Paper Is About

Action chunking lets VLA policies predict a sequence of future actions per inference call, which amortizes inference cost and enables real-time control, but the transitions between consecutive chunks are often discontinuous due to inference delay and the inherent multimodality of flow-based policies. Real-Time Chunking (RTC) reduces these boundary artifacts by constraining newly generated chunks to previously generated actions, but this mechanism lives outside the policy, so the policy still exhibits spurious multimodal switching, hesitation, and trajectories that are not intrinsically smooth. Legato's goal is to internalize continuation into the policy's own learned denoising dynamics during training, so the same model both generates actions and maintains continuity as a native property.

Key Contributions

1. Legato, a training-time continuation framework. It applies per-step, schedule-shaped guidance across denoising steps while maintaining strict training-inference consistency, achieved by reshaping the flow dynamics of action-chunked policies rather than adding an external clamp at inference.

2. Randomized schedule conditioning. The policy is trained over a family of guidance schedules (randomizing the inference delay d and ramp length r) and explicitly conditioned on the schedule, so a single model can adapt to varying inference latencies and provide flexible control over trajectory smoothness without retraining.

3. Extensive real-robot validation. Experiments across five manipulation tasks (stack bowls, pour things, pick and place, fold towel, open drawer) show Legato consistently outperforms RTC and training-time RTC, producing smoother trajectories and shorter task completion time.

Main Findings

• Legato outperforms RTC on all five tasks (Table I). Across bowls, pour, pick-and-place, drawer, and towel tasks, Legato achieves higher task scores, shorter completion time, lower NSPARC, and lower chunk-overlap RMSE than RTC. For example, on the pour task, completion time drops from 95.07 ± 2.86 s (RTC) to 75.73 ± 1.51 s (Legato), and overlap RMSE falls from 7.64 ± 0.70 to 5.14 ± 0.17 (×10³); on towel, completion time falls from 25.93 ± 0.98 s to 20.00 ± 0.78 s.

• Headline aggregate result. The abstract reports approximately 10% improvements in both trajectory smoothness and task completion time relative to RTC.

• NLDLJ improvements are not consistently statistically significant. The paper explains that NLDLJ integrates squared jerk over the entire trajectory and is therefore dominated by motion segments outside the chunk overlap regions; critically, NLDLJ does not degrade under Legato.

• Task success is not traded away for smoothness. Legato also exceeds RTC's task completion scores, so efficiency and smoothness gains come with more reliable manipulation performance.

• One-shot guidance is insufficient for continuation. In an experiment (Section III-B, evaluated on the pour task), training a policy that clamps the prefix only at initialization causes the overlap region to drift away from the reference actions as denoising proceeds (Fig. 4). This motivates per-step guidance as a requirement rather than a design preference.

• Legato beats training-time RTC (Table II). On the pour task with Legato configured as d=8, s=30, r=22: score 9.72 ± 0.13 vs 9.46 ± 0.16, completion time 75.73 ± 1.51 s vs 81.73 ± 1.12 s, NSPARC 1.65 ± 0.08 vs 2.46 ± 0.14, and NLDLJ 39.50 ± 0.13 vs 39.95 ± 0.13. The paper notes that setting ramp length to zero makes Legato's schedule resemble training-time RTC's hard overlap constraint in shape, but the two differ in that training-time RTC leaves the flow dynamics unchanged while Legato reshapes them.

• Robustness to inference delay (Table III, pour task). With fixed stride s and varying d, Legato beats RTC at every configuration: at (10,30,20), NSPARC 1.68 ± 0.09 vs 2.03 ± 0.08 and overlap RMSE 7.00 ± 0.50 vs 9.23 ± 0.75; at (8,30,22), 1.50 ± 0.07 vs 2.10 ± 0.09 and 5.94 ± 0.38 vs 7.00 ± 0.52; at (6,30,24), 1.38 ± 0.05 vs 2.03 ± 0.08 and 5.44 ± 0.31 vs 9.23 ± 0.75. Within Legato, reducing the delay shrinks the overlap region while lengthening the ramp, improving continuity and smoothness.

• Schedule and stride control a trade-off (Fig. 7). Legato outperforms RTC on almost all metrics across schedule configurations; the only exception noted is overlap RMSE in the d=s=r=8 setting, which the paper says is discussed in the appendix. Reducing execution stride s improves chunk-to-chunk consistency (lower overlap RMSE) but can accumulate high-frequency content and degrade whole-trajectory smoothness. Shortening the ramp while keeping s small improves NSPARC (frequency-domain smoothness) but weakens overlap consistency. Because r+s+d=H, changing s forces compensating changes in r.

• Schedule conditioning helps (Table IV). The provided text reports the (10,30,20) configuration: without the condition row, NSPARC is 1.64 ± 0.07 and overlap RMSE is 7.88 ± 0.70; with the condition row, NSPARC is 1.68 ± 0.09 and overlap RMSE is 7.00 ± 0.5. The table is truncated in the supplied content, so remaining schedule configurations are not reported here.

• Spurious multimodal switching is visibly suppressed. In a representative bowl-stacking rollout (Fig. 6), RTC alternates between competing grasp goals and execution arms over successive chunks, producing hesitant corrections, while Legato preserves a consistent grasp goal and arm choice.

• Controlled comparison setup. Both RTC and Legato are initialized from the same π0.5 pretrained checkpoint, trained on identical task datasets, and optimized with the same training hyperparameters and number of training steps. The exact number of training steps, dataset sizes, and number of rollouts per task are not reported in the supplied content.

Methodology in Plain English

Starting point. The policy generates fixed-length action chunks using flow matching: training interpolates between Gaussian noise and a ground-truth action chunk and teaches a network to predict the velocity that transports samples along that path; at inference, generation starts from noise and integrates the learned field over multiple denoising steps.

The diagnosis. RTC improves the boundary by constraining, at inference time, the overlapping portion of a new chunk to actions from the previous chunk. Because that constraint is external and applied only at test time, the resulting behavior does not match what the policy was trained to do. The authors also show that applying the constraint only once at initialization is not enough: the overlap drifts away from the reference as denoising continues.

The fix. Legato changes what the model sees and what it is trained to predict:

1. It defines a horizon-wise continuation vector ω over the chunk, with full guidance near the chunk's beginning and a gradual ramp-down toward the end. Using ω, it builds an action-noise mixture as the effective starting point, so training examples already contain partial action information rather than pure noise.
2. At inference, guidance is applied before every denoising step, not once. The authors derive the exact recurrence that repeated guidance induces and take its continuous-time limit to obtain an ordinary differential equation.
3. To keep training and inference consistent despite that repeated guidance, they solve for the velocity field the network must predict so the guided dynamics coincide with the standard flow-matching target. This gives a closed-form training target: the standard flow-matching direction with a schedule-dependent magnitude reweighting. Large ω suppresses the velocity in that dimension, making the guided region intrinsically less mutable and discouraging mode switching; because ω decreases along the horizon, continuation relaxes smoothly into free generation.

Handling real-world variability. The schedule is fully determined by two scalars: the inference delay d (the fully guided prefix) and ramp length r, with executed length s satisfying r+s+d=H. During training, these are randomized, and the noisy action is augmented with the schedule appended along the feature dimension so the policy is explicitly conditioned on it. At deployment, adapting to a new latency or a different desired smoothness requires only changing ω, not retraining.

Evaluation. Five real-world dual-arm manipulation tasks are used, with a fixed 120 s cutoff. Metrics are task-specific completion score, total completion time, and three smoothness measures computed on model output commands rather than robot states to decouple model behavior from low-level controllers: NSPARC (frequency-domain smoothness of the velocity profile over the whole trajectory), NLDLJ (integrated squared jerk over the whole trajectory), and chunk-overlap RMSE (local continuity at chunk connections).

Why This Matters

Impact on research. The paper reframes chunk continuation from a test-time patch into a property of the learned generative dynamics. It shows formally that repeated guidance induces a specific ODE and derives the velocity target that makes training and inference coincide, which is a reusable recipe for any flow- or diffusion-based policy that must satisfy constraints during iterative denoising. It also provides a direct empirical comparison against both inference-time RTC and concurrent training-time RTC, clarifying what part of the benefit comes from the constraint shape versus from reshaping the dynamics.

Real-world applications (as grounded in the evaluated tasks):

• Bimanual household manipulation such as stacking bowls, folding towels, and opening drawers, where hesitation and mid-task direction reversals are visible failures.
• Pouring and other contact-rich transfer tasks, where a smooth speed profile matters for not spilling or overshooting.
• Pick-and-place and packing tasks where multiple visually similar objects create many plausible grasp targets and the policy must commit to one consistently.
• Deployment on heterogeneous hardware, where inference latency varies and the schedule-conditioned model can adapt without retraining.

Industry relevance. Action-chunked VLA policies are the dominant deployment pattern for learned manipulation, and inference latency is a hard constraint on real robots. A method that shortens task completion time by roughly 10% while improving smoothness and not sacrificing success directly affects throughput in logistics, manufacturing, and service robotics. The ability to retune smoothness and latency response at inference by changing a schedule, rather than retraining, is also operationally valuable when models or hardware change.

Future Directions

• Closing the gaps in the reported ablations. The paper notes that the overlap-RMSE exception at d=s=r=8 is discussed only in the appendix, and the truncated schedule-conditioning table leaves most configurations unreported; broader schedule sweeps would clarify when schedule conditioning pays off most.
• Extending beyond the five evaluated tasks and one policy family. Legato is compared against RTC and training-time RTC under a shared π0.5 checkpoint; generalization across other VLA architectures, action dimensions, and task distributions beyond the evaluated manipulation set remains open.
• Scaling and long-horizon behavior. Whether native continuation remains stable over long multi-stage tasks, and how schedule randomization should be bounded for very different latency regimes, is not established by the current experiments.
• Theory-to-practice for the derived ODE. The paper derives an exact continuous-time system for repeated guidance and matches discretization between training and inference; how sensitive results are to the number of denoising steps N and to different integrators is not quantified in the supplied content.

Target Audience

Researchers and engineers working on learned robot manipulation, particularly those deploying action-chunked diffusion or flow-matching VLA policies in real time. It is most useful to readers already comfortable with flow matching and denoising-based generative policies who need to solve boundary discontinuity, latency-induced hesitation, or multimodal switching in deployed systems. It is also relevant to practitioners comparing training-time versus inference-time approaches to constrained generation, and to students studying how to align a generative model's training objective with the dynamics its inference procedure actually induces.

Authors’ abstract

Action chunking enables Vision Language Action (VLA) models to run in real time, but naive chunked execution often exhibits discontinuities at chunk boundaries. Real-Time Chunking (RTC) alleviates this issue but is external to the policy, leading to spurious multimodal switching and trajectories that are not intrinsically smooth. We propose Legato, a training-time continuation method for action-chunked flow-based VLA policies. Specifically, Legato initializes denoising from a schedule-shaped mixture of known actions and noise, exposing the model to partial action information. Moreover, Legato reshapes the learned flow dynamics to ensure that the denoising process remains consistent between training and inference under per-step guidance. Legato further uses randomized schedule condition during training to support varying inference delays and achieve controllable smoothness. Empirically, Legato produces smoother trajectories and reduces spurious multimodal switching during execution, leading to less hesitation and shorter task completion time. Extensive real-world experiments show that Legato consistently outperforms RTC across five manipulation tasks, achieving approximately 10% improvements in both trajectory smoothness and task completion time.