RoboJev: An Open Vision-Language Judge for Robot Success Detection

AI summary

Overview

Research area: Robotics — robot learning, policy evaluation, and vision-language models (VLMs) used as automatic success detectors for manipulation episodes.

Technical level: Intermediate. Readers should be comfortable with fine-tuning (LoRA), classification metrics (AUROC, balanced accuracy, calibration error), and the idea of a vision-language backbone scoring answer tokens.

Scope: The paper introduces RoboJev-27B, an open-weight, task-conditioned vision-language judge that predicts whether a robot manipulation attempt succeeded, and reports its accuracy, transfer behaviour, calibration, and serving cost across three evaluation settings.

What This Paper Is About

Many robot manipulation videos carry no machine-readable label saying whether the task was actually completed, and simple proxies such as trajectory length or motion smoothness do not reliably indicate success. The authors fine-tune an existing vision-language model into a detector that takes a task instruction, ten frames of an attempt, and optional final frames from auxiliary cameras, then outputs a calibrated probability of success in a single forward pass. The goal is a reusable, inspectable judge that can be dropped into review queues, offline evaluation, or terminal-reward pipelines without building a separate classifier per task or environment.

Key Contributions

1. An open-weight success judge. RoboJev-27B is released under Apache-2.0 as a rank-32 LoRA adaptation of JEV-27B-VL (language model adapted, vision encoder frozen), with the merged checkpoint, calibration file, and inference code.
2. A complete training and inference recipe. The paper documents the data mixture (52,598 episode records), the loss (class-weighted cross-entropy plus a source-specific KL anchor to the base model on research sources), the two-token calibrated readout with temperature T = 0.9, and adapter-scale selection on held-out validation sources.
3. A transparent evaluation artifact. 20,628 recorded predictions organized into 29 model/protocol entries, including frontier API baselines and historical variants, with per-episode records for 862 of the 927 labelled episodes (the other 65 are retained as ambiguous community clips).
4. An operational cost and latency measurement. Median 0.257 seconds per episode for serial inference on a single H100, estimated at roughly $0.19 per 1,000 success predictions at $2.69 per GPU-hour, excluding media processing and serving overhead.

Main Findings

• Strong in-domain discrimination. On 268 held-out SO-101 episodes, RoboJev-27B reaches 95.2% balanced accuracy and 0.993 AUROC (ECE 0.032). The same-input base model, JEV-27B-VL with ten frames, reaches 88.1% balanced accuracy and 0.979 AUROC (ECE 0.114) — a paired balanced-accuracy difference of 7.08 percentage points (95% interval [3.04, 11.45]).
• Adaptation does not transfer. On 459 episodes from unseen labs and robots, RoboJev-27B gets 78.2% balanced accuracy and 0.890 AUROC, slightly below the unadapted base model's 79.4% and 0.900 on the identical inputs (AUROC difference −0.0092, 95% interval [−0.0218, 0.0026]). The authors present this as a meaningful counterexample rather than a win.
• Frontier API models are close in the recorded comparison. GPT-6 Astra attains 93.3% balanced accuracy on the full SO-101 split using four frames, giving a paired difference of +1.91 percentage points for RoboJev (95% interval [−1.56, 5.21]), which is not resolved as superior. On the clearly labelled community subset, RoboJev gets 93.3% accuracy (AUROC 0.984), Claude Opus 5.5 gets 94.1%, and Astra and Sol each get 92.6% — differences of only one or two episodes.
• Held-out tasks are the weakest in-domain case. On the 80 episodes from the two SO-101 tasks never seen in robot-specific training (cable_unclip and ring_insert, 40 successes and 40 failures), RoboJev reaches 92.5% balanced accuracy (AUROC 0.986) versus 91.3% (0.978) for the same-input base, while four-frame Astra and Sol each reach 96.3% (AUROC 0.987 and 0.998).
• Anchoring helps preserve transfer. The released anchored model (DA) has higher unseen-domain AUROC than the unanchored variant (DB), 0.890 versus 0.847, though each is a single run rather than an average over seeds.
• Temporal context matters. Restricting the released model to only the final frame drops unseen-domain AUROC to 0.776, and the previous four-frame release scored 0.987 on SO-101 and 0.872 on the unseen domain versus 0.993 and 0.890 here (training data and inputs also changed, so this is not controlled).
• Calibration does not survive domain shift. ECE rises from 0.032 on SO-101 to 0.148 on unseen domains (Brier scores 0.033 and 0.162); on the clear community subset ECE is 0.027 and Brier score is 0.052. The unseen-domain reliability plot shows the model often underestimates the true success probability.
• Ambiguity is preserved, not hidden. 65 of the 927 evaluation episodes (32.5% of the community set) have no definite label and are excluded from binary metrics but retained in the artifact.

Methodology in Plain English

The authors start from an existing vision-language model and teach it one narrow habit: look at a short visual record of a robot attempt and answer "true" or "false" to the question of whether the instruction was completed. Input is deliberately fixed — the task text, ten evenly spaced frames from the primary camera, and the last frame from any other available cameras — so the same judge can be pointed at any task or environment without retraining.

Only the language side of the model is trained, using a small low-rank adapter (rank 32, alpha 64, dropout 0.05, about 233 million trainable parameters); the vision encoder stays frozen. Training runs for one epoch at a learning rate of 5×10⁻⁵ with AdamW, a global batch of 14, and seed 7, taking about 5.5 hours on 7 H100 GPUs. The loss combines a class-balanced cross-entropy term — with per-source inverse-frequency weights clamped to [0.25, 4], and 0.5 for synthetic negatives — with a KL penalty pulling predictions back toward the original model, applied only to the research sources (ViFailback and DROID) and not to community data or ArmnetBench.

Training data is a mixture: 28,170 episodes from RBM-1M sources, 1,388 from ArmnetBench, 10,216 teacher-labelled community episodes, 3,718 community demonstrations, 4,962 from ViFailback, and 4,144 successful DROID episodes. Community labels come from a consensus pipeline that keeps high-confidence GPT-6 Astra and GPT-6.1 Sol annotations on four primary-camera frames, uses Claude Opus 5.5 as a veto when available, averages the remaining teacher probabilities, and requires the mean to be at least 0.9 or at most 0.1. Training examples from all 106 owners appearing in the human-labelled community check are discarded to prevent leakage. Augmentation (truncated episodes, mismatched instructions, resizing, JPEG recompression, colour perturbation) brings the epoch to 102,172 examples.

At inference, the model reads the logits for the "true" and "false" tokens, adds fitted per-token biases, divides by a temperature, and applies a softmax — producing a probability in one forward pass. The temperature and biases are fit on validation data by minimizing negative log-likelihood, and the adapter strength is chosen from {0, 0.25, 0.5, 0.75, 1} by a rule that must stay within 0.005 AUROC of the base model on each checked validation source.

Why This Matters

Automatic success labelling turns unlabeled robot video into usable evaluation data. For the research community, the paper's most valuable move may be its honesty: it reports a domain where fine-tuning actively loses to the base model, publishes per-episode predictions so others can inspect failures, and keeps ambiguous clips in the artifact instead of forcing them into a binary metric.

Real-world applications:

• Curating and auditing robot datasets. Filtering or flagging episodes in large public collections — the paper sketches a catalogue of 68,555 episodes from 7,641 public datasets (501.17 hours), estimated at roughly 4.9 GPU-hours to score end to end.
• Offline policy evaluation. Scoring rollouts from competing policies without hand-labelling, so review queues and leaderboards can be built from video alone.
• Learned terminal rewards. The release includes a LeRobot plugin that emits a thresholded terminal reward and a grader that resolves video offsets from dataset metadata, though the effect of these rewards on policy learning is not yet measured.
• Post-hoc debugging of failures. The overlay scores prefixes of an attempt, letting a human watch the score evolve as evidence accumulates and spot episodes where the judge is misled.

Industry relevance: an open-weight checkpoint that fits in about 55 GB of bf16 memory and runs on an 80 GB GPU gives teams a self-hostable alternative to per-query API calls, with a stated compute cost of roughly $0.19 per 1,000 predictions. The paper also makes clear what such a judge is not: it is not a safety monitor, and its probabilities should not be taken at face value on a new domain without local labelled validation.

Future Directions

• Making the frontier comparison like-for-like. The API baselines receive only four primary-camera frames while RoboJev receives ten plus auxiliary views, and the authors list matching the input budget as their main planned follow-up.
• Closing the transfer gap. The anchored model still trails the base model on unseen domains, so the question of which adaptation choices preserve generalisation — anchoring weight, adapter scale, data mixture — remains open.
• Expanding human-labelled domain coverage. Reported results span 8 testing domains, and wrist-camera views and long-horizon tasks such as stirring and folding remain difficult; more human-labelled domains are proposed.
• Using the judge as a training signal. The terminal-reward integration is implemented but its impact on downstream policy learning is unmeasured, and closed-loop use would require choosing an operating threshold that guards against exploiting reward errors or premature termination.

Target Audience

Researchers and engineers working on robot learning infrastructure — dataset curation, policy evaluation, and learned reward models — who need automated success labels and are weighing a self-hosted open-weight judge against API-based frontier models. It is also useful for practitioners evaluating VLMs as decision modules, because the paper reports calibration drift, token-level readouts, and inference economics rather than only headline accuracy. Readers wanting a first look at robot success detection can follow the main results; the appendices on protocol variants, data provenance, and licensing are aimed at those who intend to reproduce or build on the artifact.

Authors’ abstract

Many robot manipulation videos do not indicate whether the task was successfully completed. RoboJev-27B is an open-weight vision-language model that takes a task instruction, 10 frames of a manipulation attempt, and (if available) final frames from auxiliary cameras and predicts whether the robot accomplished the task. RoboJev-27B is a variant of JEV-27B-VL, trained with a rank-32 LoRA adapter on the language model only (keeping the vision encoder frozen) using 52,598 episode records. It also uses a source-specific KL penalty that applies only to research data, anchoring predictions closer to the original model on those sources. It uses a calibrated two-token readout that outputs success/failure probabilities in a single forward pass. On 268 held-out episodes of the same robot model (SO-101), RoboJev-27B attains 95.2% balanced accuracy and 0.993 AUROC. It also generalises to episodes collected by unseen labs/robots (AUROC = 0.890 over 459 episodes) as well as to clearly labelled community-collected episodes (accuracy = 93.3% over 135 episodes), where the base model attains an AUROC = 0.900 on the unseen-lab/robot subset with the same inputs. It takes 0.26 seconds per episode (median, serial) to infer on a single H100, which is about \$0.19 per 1,000 success predictions at \$2.69 per GPU-hour (not including media processing and serving costs). RoboJev-27B is released together with its inference code, along with a transparent evaluation: 20,628 predictions organized into 29 model/protocol entries are made available to facilitate browsing and to enable reproducible benchmarking of success detection. We also compare against strong API-based frontier VLMs, which take in fewer input frames (4 images from the primary camera only); note that this comparison is not strictly like-for-like because RoboJev-27B is given more frames at inference.