PALM: Progress-Aware Policy Learning via Affordance Reasoning for Long-Horizon Robotic Manipulation

AI summary

Overview

• Research area: Robotics — vision-language-action (VLA) models for long-horizon, language-conditioned robotic manipulation.
• Technical level: Intermediate. The paper assumes familiarity with VLA policies, diffusion-based action generation, imitation learning, and affordance representations, but its core ideas are explained in accessible terms.
• Scope: The paper introduces PALM, an end-to-end framework that forecasts structured future affordances and jointly predicts robot actions plus a continuous subtask progress signal, evaluated on the LIBERO and CALVIN simulation benchmarks and on three real-world long-horizon generalization settings.

Authors are affiliated with the University of Pennsylvania, the University of Illinois Urbana-Champaign, Nanyang Technological University, the University of Oxford, and the Massachusetts Institute of Technology. The paper is listed under arXiv:2601.07060v2 [cs.RO], dated 04 Apr 2026, with a project page at https://plan-lab.github.io/palm.

What This Paper Is About

Current VLA models map visual observations and language instructions directly to robot actions, and they work reasonably well on short tasks but break down on long, multi-step ones. The paper argues that two things are missing: internal representations that say which object to interact with, where to contact it, where to place it, and how it should move next; and a persistent sense of how far along the current subtask the robot is. PALM adds both — a set of future-facing affordance predictions and a within-subtask progress value — so that visually similar states are no longer ambiguous and execution stays coherent across an entire long sequence.

Key Contributions

1. A unified VLA framework (PALM) that couples structured affordance reasoning with progress-aware action generation in a single closed perception–action–progress loop for long-horizon, contact-rich manipulation.

2. A fine-grained affordance predictor that acts as an intermediate reasoning step, producing structured, task-relevant latent representations at a future time offset through four specialized query types: <Global>, <Local>, <Spatial>, and <Dynamic>.

3. A progress-aware inverse-dynamics module that couples action generation with continuous subtask progress estimation, predicting a scalar in [0, 1] alongside each action so that the policy has an online notion of "where we are" without needing a separate planner or hierarchical controller.

4. Comprehensive evaluation across simulation and real-world settings, reporting a 91.8% success rate on LIBERO-LONG, a 12.5% improvement on CALVIN ABC→D, and a 2× improvement over real-world baselines across three long-horizon generalization settings.

Main Findings

• CALVIN ABC→D: PALM reaches a 96.9% success rate on the first subtask and 82.0% on five consecutive subtasks. On the CALVIN ABC→D benchmark the paper reports a 12.5% improvement (described in the abstract as an average-length improvement; PALM's average length is 4.48 versus Seer's 3.98 and π₀'s 3.92). At the 5-task horizon, this is a +17.7% absolute improvement over the strongest prior baseline, Seer at 64.3%.

• Progress prediction matters on CALVIN: Removing the progress-aware policy (PALM ✗ progress) drops performance across horizons, with average length falling from 4.48 to 4.02.

• LIBERO: PALM achieves a 94.5% ± 1.0% average success rate across all four suites (Spatial 95.2% ± 1.2%, Object 96.7% ± 0.7%, Goal 94.3% ± 1.6%, Long 91.8% ± 0.8%). The largest gain is LIBERO-LONG, where PALM's 91.8% outperforms the strongest baseline (CoT-VLA at 69.0%) by 22.8%.

• Ablating PALM's modules (CALVIN ABC→D, average length): Removing affordance foresight causes the largest drop during fine-tuning (4.48 → 3.58); removing progress prediction causes the largest drop during pre-training (4.48 → 3.73); removing inverse dynamics prediction reduces performance in both stages (4.17 pre-training, 3.92 fine-tuning).

• Ablating affordance types: Adding Global affordance alone improves both CALVIN and LIBERO-LONG. Adding Local affordance gives further gains on CALVIN but a slight dip on LIBERO-LONG, attributed to viewpoint-induced geometric bias in fine-grained edge features. Adding Spatial affordance restores and further improves performance on both. Adding Dynamic affordance on top of the full set yields the best performance.

• Ablating training data (CALVIN ABC→D average length / LIBERO-LONG success rate): Removing in-the-wild data gives 3.90 / 73.5%; removing long-horizon video data gives 3.73 / 84.5%; removing human-annotated data gives 3.58 / 76.5%; removing simulation pre-training data gives 3.96 / 81.0%; the full model gives 4.48 / 91.8%.

• Real-world long-horizon task (6 consecutive subtasks, one high-level instruction, 20 rollouts per task): PALM's average length is 3.05 under random localization (versus OpenVLA 0.95 and Octo 0.65), 3.80 under visual distraction (versus OpenVLA 1.60 and Octo 0.95), and 3.55 under unseen lighting (versus OpenVLA 1.25 and Octo 1.05). On the sixth subtask under random localization PALM succeeds 0.30 of the time, while both baselines reach 0.00.

• Reporting note: The paper's text states it compares against Seer on LIBERO as well, but Seer does not appear in the LIBERO results table.

Methodology in Plain English

PALM takes three synchronized inputs — a language instruction, image observations, and the robot's proprioceptive state.

Encoding. The instruction goes through a CLIP text encoder. Images are encoded with a Masked Autoencoder (MAE-pretrained ViT-B) over two viewpoints (a static eye-on-base camera and an eye-on-hand camera), then compressed by a Perceiver Resampler to keep the token count manageable over long sequences. Robot state (a six-dimensional end-effector pose plus a binary gripper value turned into a two-dimensional one-hot vector) passes through a lightweight MLP. Everything is concatenated into one multimodal token sequence.

Backbone and queries. A GPT-2-style transformer with causal attention processes the sequence. On top of it sit two learnable query sets. The first is the affordance queries, split into four subqueries that are deliberately kept disentangled by attending only to shared context tokens.

The four affordance predictions. Each subquery forecasts a future state at time t+n:

• Global — which object the instruction refers to and roughly where it is. Supervision comes from Grounding DINO to resolve the referent and SAM to segment it into a binary mask, trained with focal loss plus soft Dice loss.
• Local — fine-grained contact geometry, such as edges and part boundaries, predicted as a dense contact-likelihood distribution. Following GLOVER++, annotated contact points become Gaussian heatmaps, trained with focal loss plus a KL term on ℓ₁-normalized maps.
• Spatial — a small set of candidate placement points, so the model proposes regions rather than memorizing one coordinate. Targets come from SpatialVLM for spatial semantics and RoboPoint for executable 2D coordinates, trained with a set-matching objective pairing each target point with its closest predicted candidate.
• Dynamic — which pixels belong to the gripper or movable objects and how they will move. Supervision uses an N×N grid of query points placed on a frame at t−δ, tracked forward with CoTracker; trajectories whose cumulative displacement exceeds a threshold are rasterized into a dynamic region mask at t+n. Trained as a masked reconstruction loss with a latent-variable model (posterior, decoder likelihood, and a KL term with weight β).

Progress-aware control. The second query set (action–progress) pools control-relevant context and conditions on the predicted affordance latent. A denoising diffusion transformer (DiT) then decodes a horizon-n sequence of both actions and scalar progress values in [0, 1]. Classical inverse dynamics predicts one action between two observations; PALM extends this to an n-step action–progress sequence conditioned on a single-step affordance latent. Training follows a standard diffusion objective with Gaussian noise and a noise predictor ϵ_θ.

Why progress helps. At each step the model infers the active subtask stage from the affordance latent and derives a stage embedding, then predicts a scalar quantifying within-stage completion. This disambiguates visually similar observations that call for different actions depending on stage, encourages monotonic, stage-consistent latent evolution, and smooths transitions between sub-policies.

Training pipeline. Pre-training mixes DROID and BridgeData V2 for large-scale in-the-wild robot demonstrations with EPIC-KITCHENS and RoboCerebra for fine-grained sub-steps and time-segment annotations that supervise semantic progress estimation. Fine-tuning uses 942 trajectories selected from robot data and annotated with affordance data and continuous progress labels through a semi-automated method.

Evaluation setup. Simulation uses LIBERO (four suites — Spatial, Object, Goal, Long — each with 10 tasks and 50 demonstrations) and CALVIN (34 tasks across four environments, with the challenging ABC→D setting where the model trains on ABC and is evaluated on the unseen D environment). CALVIN results report the average success rate of the top three checkpoints over 1,000 rollouts per task; LIBERO reports average success rate and standard error over 3 seeds with 500 episodes each. Real-world experiments use a UFACTORY xArm6 with a Gripper G2 and two RealSense D455 cameras configured as eye-on-hand and eye-on-base, fine-tuned on 200 demonstrations collected on the robot with RGB images, robot states, and actions. Baselines are OpenVLA and Octo, each allowed a maximum of three execution attempts per rollout, trained for an equal number of iterations and evaluated with the final checkpoint.

Why This Matters

Impact on research. The paper reframes the long-horizon failure problem as a representation problem rather than a scaling problem. Instead of predicting dense future images (expensive) or adding an external planner (brittle), it predicts a compact structured set of future affordances and a scalar progress signal, both of which are decoded jointly with actions by the same diffusion head. The ablation results also isolate where each ingredient pays off: affordance foresight matters most after fine-tuning on robot data, while progress prediction matters most when large-scale long-horizon video is available during pre-training. That is a useful signal for how the field should allocate data and supervision.

Real-world applications:

• Household and service robotics for multi-step chores such as clearing a cluttered table, where a robot must sequence several pick-and-place stages under one instruction.
• Warehouse and logistics pick-and-place, where objects, lighting, and layouts change and the robot must not repeat or skip steps.
• Industrial assembly and kitting involving contact-rich sequences of subtasks with spatially under-specified instructions (for example, "put it there").
• Kitchen and food-prep automation, motivated directly by the paper's use of EPIC-KITCHENS data for learning progress in contact-rich multi-step scenarios.

Industry relevance. The results suggest that modest affordance and progress supervision can substantially extend the horizon over which a generalist policy remains reliable, which is a prerequisite for deploying VLA-style models in commercial settings where failures mid-task are costly. The 2× improvement over real-world baselines with limited fine-tuning data (200 demonstrations) is directly relevant to companies that cannot collect millions of in-domain trajectories.

Future Directions

• Closing the annotation bottleneck. Fine-tuning relies on 942 trajectories annotated with affordance and continuous progress labels via a semi-automated method, and removing human-annotated data caused the largest LIBERO-LONG degradation (91.8% → 76.5%). Reducing or automating this labeling is a clear next step.
• Resolving the Local affordance trade-off. Local affordance improved CALVIN but slightly hurt LIBERO-LONG, which the authors attribute to viewpoint-induced geometric bias. Understanding and fixing this would make the affordance set more uniformly beneficial across embodiments and camera setups.
• **Extending beyond six-sub

Authors’ abstract

Recent advancements in vision-language-action (VLA) models have shown promise in robotic manipulation, yet they continue to struggle with long-horizon, multi-step tasks. Existing methods lack internal reasoning mechanisms that can identify task-relevant interaction cues or track progress within a subtask, leading to critical execution errors such as repeated actions, missed steps, and premature termination. To address these challenges, we introduce PALM, a VLA framework that structures policy learning around interaction-centric affordance reasoning and subtask progress cues. PALM distills complementary affordance representations that capture object relevance, contact geometry, spatial placements, and motion dynamics, and serve as task-relevant anchors for visuomotor control. To further stabilize long-horizon execution, PALM predicts continuous within-subtask progress, enabling seamless subtask transitions. Across extensive simulation and real-world experiments, PALM consistently outperforms baselines, achieving a 91.8% success rate on LIBERO-LONG, a 12.5% improvement in average length on CALVIN ABC->D, and a 2x improvement over real-world baselines across three long-horizon generalization settings.