RynnValue: Scaling Robotic Value Foundation Models with Temporal Distance

AI summary

Overview

Research area: Robotics — general-purpose reward/value modeling for robot manipulation, and its use as a dense reward interface for reinforcement learning.

Technical level: Advanced. The paper assumes familiarity with value functions, cost-to-go, potential-based reward shaping, distributional regression, and large multimodal transformer training.

Scope: The paper introduces RynnValue, an open-source, language-conditioned value foundation model for robotic manipulation that replaces preference- and progress-based supervision with timestamp-derived temporal distance (directed cost-to-go to a language-specified goal), trained on over 7,000 hours of heterogeneous embodied data and evaluated both as a trajectory-ranking model and as a reward signal for real-world policy learning.

What This Paper Is About

Scaling reinforcement learning for generalist robot policies is limited less by policy capacity than by reward supervision: hand-designed rewards do not generalize, sparse success signals give weak guidance over long tasks, and existing general-purpose reward models depend on task-internal anchors (preferences, reference demonstrations, local comparisons) that do not transfer across embodiments and data sources. The common fallback, normalized [0,1] progress, is an intra-trajectory coordinate rather than a goal-conditioned cost-to-go, making it poorly aligned with the standard notion of value in control.

The paper reframes the problem from a reward model scoring trajectory-level anchors to a value foundation model predicting goal-conditioned cost-to-go as a single reusable interface. Its scaling target is temporal distance — the directed remaining time from an observation to a relabeled task-completion cutoff — which can be read directly from timestamps, requiring only instructions, timestamps, and relabeled cutoffs rather than preference pairs or progress annotations.

Key Contributions

1. Reframes robotic reward modeling around temporal distance. The authors identify normalized progress as a supervision bottleneck and propose learning a value foundation model whose scaling target is temporal distance (cost-to-go): a preference-free, goal-conditioned value target that unifies heterogeneous data under one interface.

2. A label-cheap data recipe at scale. Labels are derived directly from timestamps, aided by subtask segmentation and cutoff relabeling, over a corpus exceeding 7,000 hours and roughly 3M instruction-conditioned clips spanning diverse embodiments, viewpoints, and task families (from 1.67M original episodes).

3. Shortcut-suppression designs plus dual distributional heads. Temporal-order shuffling and value-isolation attention keep temporal-value predictions grounded in visual evidence rather than spurious correlations from sampling intervals, sequence position, or cross-query value extrapolation; two distributional heads produce absolute and relative temporal values.

4. Preference-free state-of-the-art ranking and a working reward interface. RynnValue surpasses preference-supervised state of the art on out-of-distribution trajectory ranking without any preference annotation, and via potential-based shaping improves both online and offline real-world policy learning.

Main Findings

• Ranking performance: RynnValue-8B attains an average Kendall's τ_a of 0.675 on RBM-EVAL-OOD, beating the fully preference-supervised Robometer (RBM-1M) at 0.655. Even the smaller RynnValue-4B reaches 0.670, and its per-dataset scores are 0.542 (USC Franka), 0.488 (USC Koch), 0.917 (USC Trossen), 0.667 (USC xArm), 0.473 (MIT Franka), 0.933 (UTD SO101).
• Large margin over progress-only supervision: The progress-only Robometer counterpart scores only 0.292 on average, which RynnValue more than doubles.
• Best preference-free method on every dataset: RynnValue variants attain the best results on all six datasets among methods trained without explicit trajectory-level preference supervision, improving the strongest preference-free prior average of 0.502 (RoboReward-4B) to 0.670 at 4B and 0.675 at 8B.
• Per-dataset bests: RynnValue-8B ranks first overall on USC Franka (0.667), USC Koch (0.544), and USC Trossen (1.000); RynnValue-4B attains the top score on UTD SO101 (0.933). RynnValue-8B scores 0.500 on USC xArm and 0.503 on MIT Franka.
• Scale matters little: Performance is already strong at 4B and improves only marginally at 8B, suggesting gains come from the temporal-distance formulation and its training and architectural designs rather than sheer model scale.
• Instruction grounding: In instruction-trajectory confusion-matrix analysis, RynnValue produces the clearest diagonal structure with the highest normalized diagonal margin of 0.79, versus 0.67 for the strongest baseline — indicating it distinguishes matched from mismatched instruction-trajectory pairs rather than tracking generic visual progress.
• Ablations (average Kendall's τ_a on RBM-EVAL-OOD): Full model 0.675; w/o shuffle 0.189 (largest degradation); uniform sampling instead of random temporal sampling 0.379; w/o value-isolation attention 0.482; w/o natural-language supervision 0.537; w/o relative temporal-distance supervision 0.627.
• Downstream policy gains: Converted into dense rewards via potential-based shaping, RynnValue raises real-world policy success from 52.5% to 72.5% online and from 63.8% to 82.5% offline over the strongest baseline.
• Zero-shot generalization: The model generalizes zero-shot to unseen tasks, embodiments, and viewpoints.

Methodology in Plain English

The supervision target. Instead of asking annotators which of two trajectories is better, or rescaling every task onto a [0,1] progress curve, the model learns how much time remains until the task is done. For each observation before a relabeled completion cutoff, the label is the remaining time to that cutoff; observations at or after the cutoff get zero. A second, relative signal records the signed time displacement between two consecutively presented observations: positive means forward progress, negative means temporal regression. Both come straight from timestamps.

Data preparation. The authors pool real-world, simulated, and egocentric robot data from ten sources (AgiBot, EgoDex, Galaxea Open-World, InternData-A1, Open X-Embodiment, RDT, RoboCOIN, RoboMIND, RoboTwin, Soft-FOLD), covering single-arm platforms, dual-arm mobile manipulators, bimanual tabletop robots, dexterous-hand systems, simulated embodiments, and first-person human demonstrations. Long demonstrations are split using native temporal annotations where available, otherwise kept whole; each segment gets a completion cutoff (segment endpoint by default, with dataset-specific ratio- or duration-based trimming where needed) so that post-completion motion does not corrupt labels. Qwen3-VL-27B generates segment-level behavior descriptions, which supervise the natural-language video-analysis output but do not affect temporal targets.

Architecture. RynnValue builds on RynnBrain and is further pretrained on large-scale robot data. The input is an interleaved multimodal sequence: metadata, instruction, then for each of K=8 sampled observations an absolute-value query group followed (from the second observation on) by a relative-value query group, then a verification prompt. Each query group contains N=8 repeated query tokens; their hidden states are concatenated rather than averaged, yielding richer temporal representations. Two distributional heads (BroNet residual MLPs, hidden width 4096, depth 8, ReLU) read out bin logits over 256 symlog-spaced bins spanning [0,512] seconds for absolute and [-256,256] seconds for relative distances, trained with two-hot targets and decoded via an inverse symlog transform. The LM head handles natural-language analysis and verification only.

Shortcut suppression. Two ideas prevent the model from gaming the format. Random temporal sampling picks observations at irregular timestamps so fixed intervals cannot be exploited; temporal-order shuffling scrambles chronological order (half the training sequences unsorted; the rest a forward-biased walk with rewind probability 0.3), so sequence position is no longer a proxy for progress. Value-isolation attention lets queries within one group see each other and the language-visual context, but blocks queries from different observations from seeing one another — and blocks context tokens from attending to query tokens — so values cannot be extrapolated from other value predictions.

Training. Three cross-entropy objectives are combined: absolute temporal distance, relative temporal displacement, and causal language modeling (weighted by λ_lang = 2). Instruction-mismatch augmentation replaces the instruction with one from a different trajectory for 10% of samples, supervising "Match: No" and "Success: No" and masking the absolute loss (the relative loss is retained, since it is instruction-independent). Optimization uses AdamW with learning rate 1×10⁻⁶, β₁ = 0.9, β₂ = 0.95, weight decay 0.1, ε = 10⁻⁸, a constant schedule without warm-up, gradient-norm clipping at 100, bfloat16, FSDP hybrid sharding, and per-device batch size 2.

Inference and reward. At inference, sampling augmentations are disabled and observations are ordered chronologically, though the value-isolation mask stays active. The predicted temporal distance is turned into a potential by sign reversal (Φ_t = −v_t), so states before completion have negative potential and the potential approaches zero at the goal — preserving the temporal scale rather than normalizing to [0,1]. Trajectories are scored by this potential at the final queried observation, which requires no cross-dataset calibration since Kendall's τ_a depends only on score ordering.

Why This Matters

Impact on research. The paper shows that a preference-free, timestamp-derived target can match and even surpass fully preference-supervised training for general-purpose robotic value models. If temporal distance transfers as well as these results suggest, it removes a major labeling bottleneck: no preference pairs, no normalized progress scales, and no per-dataset calibration — just instructions, timestamps, and completion cutoffs. The ablation pattern also gives the field a concrete account of which failure modes matter, with order shuffling and value isolation each worth large gains, and it suggests the gains are not primarily a function of model scale.

Real-world applications.

• Online reinforcement learning for manipulation: using RynnValue as a dense reward during physical robot training, where the paper reports success rising from 52.5% to 72.5%.
• Offline policy improvement: scoring and reweighting existing demonstration data, where the paper reports success rising from 63.8% to 82.5%.
• Failure and regression detection: because the model tracks signed relative temporal displacement, it can flag non-monotonic events rather than producing a smooth monotone progress curve.
• Interpretable task verification: the language branch emits a video description plus instruction-match and success judgments, supporting automated QA of robot episodes without separate classification heads.

Industry relevance. The work is released openly (project page, GitHub, Hugging Face and ModelScope collections) from DAMO Academy, Alibaba Group and Hupan Lab. A reward model that ingests heterogeneous corpora without new annotation pipelines is directly useful for anyone assembling robot learning data at scale, and the finding that a 4B variant is nearly as good as the 8B variant matters for deployment cost.

Future Directions

1. Does temporal distance hold outside manipulation? The corpus is manipulation-centric (single-arm, dual-arm, mobile, dexterous, simulated, egocentric); whether the same target transfers to navigation, locomotion, or whole-body tasks is untested here.
2. How far does the data recipe scale? The paper trains on 7,000 hours and ~3M clips and reports only marginal improvement from 4B to 8B. The scaling behavior of temporal-distance supervision with further data and larger backbones is not established.
3. Cutoff estimation remains a weak link. Completion cutoffs are assigned by segment endpoints or dataset-specific ratio/duration trimming. Learning or verifying cutoffs automatically — rather than trimming heuristically — could matter for tasks whose endpoint does not correspond to first semantic completion.
4. Robustness under language-visual inconsistency and embodiment shift. Instruction-mismatch augmentation covers 10% of training samples, and evaluation includes out-of-distribution embodiments, but the limits of reward reliability under adversarial or subtly wrong instructions are left open.

Target Audience

Robotics and embodied-AI researchers working on reward modeling, value functions, and reinforcement learning for manipulation; robot learning engineers who need dense reward signals for real hardware; multimodal and foundation-model researchers interested in distributional regression, shortcut suppression, and attention masking; and industry teams building robot data pipelines who want a supervision target that avoids preference annotation and per-dataset progress normalization.

Authors’ abstract

General-purpose reward models are increasingly the bottleneck for scaling robot learning, yet the recipe for learning value-related capabilities from large-scale heterogeneous corpora remains underexplored. Existing approaches tie supervision to task-internal anchors such as preferences or normalized progress, none of which transfer cleanly across embodiments and data sources. We introduce RynnValue, an open-source value foundation model for robotic manipulation that replaces these anchors with temporal distance, the directed cost-to-go from an observation to the language-specified goal. Because temporal-distance labels can be derived directly from timestamps, RynnValue scales to over 7,000 hours and roughly 3M instruction-conditioned clips without preference or progress annotations. To make temporal-value learning reliable at scale, we combine random temporal sampling, temporal-order shuffling, and value-isolation attention, suppressing shortcuts that would leave predictions insensitive to failures and regressions. Trained without preference labels, RynnValue attains an average Kendall's tau_a of 0.675 on RBM-EVAL-OOD, surpassing the fully preference-supervised state of the art (0.655) and more than doubling a progress-only counterpart (0.292), while generalizing zero-shot to unseen tasks, embodiments, and viewpoints. Converted into dense rewards via potential-based shaping, it raises real-world policy success from 52.5% to 72.5% online and from 63.8% to 82.5% offline. These results establish temporal distance as a scalable supervision target and practical reward interface for generalist robot policies.