Measuring Language Transfer in Robot Policies: Adding Greek to a Cosmos3 Vision-Language-Action Policy

AI summary

Overview

Research area: Robotics — specifically vision-language-action (VLA) policy learning, cross-lingual transfer, and the methodology of evaluating whether a robot policy actually follows instructions in a non-English language.

Technical level: Advanced. The paper assumes familiarity with imitation learning, vision-language-action stacks, diffusion/autoregressive world models, tokenization, and paired statistical testing.

Scope in one sentence: The authors machine-translate an existing English robot demonstration corpus into Greek, change nothing in the Cosmos3 architecture, and then spend most of the paper showing that the five evaluation instruments a practitioner would reach for first all report success where subsequent controls show there is none.

What This Paper Is About

Robot foundation models are trained, benchmarked, and instructed almost entirely in English, and no robot demonstration corpus exists for most other languages. The authors ask what the cheapest possible localization path buys: take an open Cosmos3 vision-language-action stack, machine-rephrase its English instructions and captions into Greek under a mandated glossary with no human-authored Greek anywhere, and measure what actually transfers. Their central claim is that producing the Greek data took hours and worked, while determining whether it worked was the real bottleneck — so the paper is framed as a study of controls and retractions rather than as a system report.

Key Contributions

1. The text tower is the bottleneck. Supervised fine-tuning cannot induce Greek grounding in a video world model whose text tower was pretrained effectively on English. Across a dose ladder from a frozen tower up to full learning-rate tower training at triple duration, Greek-conditioned generation remained structureless noise while English improved markedly. Swapping to a base model with a multilingual 8B tower and applying the same stock recipe yields coherent Greek-conditioned scenes, judged from a subsequent 6,836-clip run (70/30 Greek/English) built on 1,222 captions plus LIBERO renders.

2. Grounding transfers by level, not uniformly. Judged on 51 held-out scenes per language, the localized world model produces coherent in-domain scenes for 64% of Greek prompts (50 of 51 returned an evaluable judgement), where the base model produced none judged coherent, but matches the specific prompt content in 0% of cases and reaches a partial match on 30%. English reaches 100% coherence and 90% good content match.

3. Negative transfer from the world model to the policy. Warm-starting the policy from the Greek-adapted world model degrades both languages: English 79.2% versus 96.4% when trained from the base model, and Greek 13.6% versus 48.6%. The authors offer erosion of action-relevant representations by video-generation fine-tuning as one explanation, and note a domain shift (BridgeData V2 real WidowX versus simulated Franka in LIBERO) as a second that their design does not exclude.

4. A cross-lingual attribution control with a null guaranteed by construction. Rather than deleting or paraphrasing an instruction, the authors issue instructions in a language the policy is independently proven not to follow: an otherwise identical policy trained without target-language demonstrations scores at chance. This complements perturbation-based controls at the cost of training a second policy, and removes the question of how much a paraphrase ought to matter.

Main Findings

• Five standard instruments reported success where there was none. A colour-histogram coherence metric doubled twice on generations that were pure noise; a standard single-goal benchmark scored 84.6% under Greek instructions and 82.6% under deliberately wrong ones; a ten-goal suite credited a policy with Greek instruction-following that the ninety-task suite shows to be marginal at best, under three points over its own control on every seed; training loss ranked six policies within 1.4% of each other while their Greek ability spans a factor of 7.2; and single-run comparisons between recipes are uninterpretable because target-language success moves 31.6 points on the random seed while English moves 1.0.

• Target-language demonstrations are necessary but not sufficient. A multilingual tower transfers nothing to the action pathway without them: English-only training leaves Greek at its own wrong-instruction floor across three seeds (9.0/9.2/9.2%). Target-language demonstrations alone are little better: on ninety tasks, across three seeds, the Greek-only policy's margin over its own wrong-instruction floor never exceeds 2.7 points, while the bilingual policy's never falls below 6.7.

• Two intuitive interventions make things worse. Warm-starting from a language-adapted world model (one run) and unfreezing the text tower (three seeds: 18.0/13.4/29.8% Greek, versus bilingual at 48.6/80.2/55.2%) both degrade performance. The tower-unfreezing result is consistent with prior work arguing that action-training gradients degrade the backbone's semantic knowledge.

• A bilingual policy reaches 48.6% Greek success on ten goals and 27.4% (three-seed mean) on ninety tasks, from machine-produced Greek alone, with no architecture change. This is roughly half the episodes English reaches on ten goals and two fifths on ninety tasks.

• Ten-goal results are bimodal and largely a property of the run. Per goal, five goals score 0.72–0.98 and five score 0.00–0.24 with nothing in between, while English is at least 0.88 on all ten. The split reproduces in an independent smaller run at the same seed but does not survive reseeding: three seeds give five, nine and six of ten goals above 50%, and no goal fails in all three.

• A near-minimal pair shows the failure is discrimination, not vocabulary. "Put the bowl on top of the cabinet" and "put the wine bottle on top of the cabinet" differ in Greek only in the object noun phrase and score 0.94 and 0.00 respectively, with both content words independently grounded elsewhere.

• The wrong-instruction residue is command-dependent. Reassigning every distractor moves the residue entirely: the single goal carrying all of it drops to zero and five different goals become non-zero. A policy falling back on one dominant behaviour would produce the same cell under both assignments; it does not. The authors could not identify the feature that governs which wrong command elicits which behaviour, and a head-noun account is not supported (of six goals whose reassigned distractor shares a noun stem, only three produce any success, and two of the five non-zero cells share no stem).

• Aggregate success rates conceal mechanism even after passing the control. The English-only policy is a measured per-goal null with Greek success of 0.34 on one goal, 0.32 on another, and near zero elsewhere. Read against that null, the bilingual policy's largest genuine gains are +0.84 on two goals, and on three goals it sits at or below a policy with no Greek at all. The defensible claim is "partial, non-compositional referent-level grounding", not 48.6% Greek competence.

• Much of the headline number is translator idiolect. An independent Greek translation costs 23.3 points on average (drops of 30.8, 24.8 and 14.4 across three seeds, from 48.6/80.2/55.2% to 17.8/55.4/40.8%). The three seeds lose 63%, 31% and 26% of their trained-phrasing level, a mean of 40%. The degradation is not monotone in distance from the training translator: one run scores 42.6% on a mildly reworded set and 78.6% on a structurally distant one, a 36-point inversion.

• Generic brittleness to rewording does not explain the Greek drop. Rewording in English costs 2.4 to 6.6 points (mean 3.9) while rewording in Greek costs 14.4 to 30.8 (mean 23.3). A second English control matched on perturbation size — substituting everyday synonyms in seven of ten commands, a harder noun perturbation than the Greek set's five of ten — still costs only a mean of 4.7 points, with a single seed losing 9.4. English absorbs the harder perturbation at 4.7 points where Greek loses 23.3 under a milder one, roughly five times more

Authors’ abstract

Robot foundation models are trained and evaluated predominantly in English, and robot demonstration corpora do not exist for most languages. We study the addition of Greek to an open vision-language-action stack using only machine-rephrased instructions and no architecture changes. The main challenge is measurement rather than translation. Several plausible instruments produce false conclusions: a color-histogram metric rewards noise, a single-goal benchmark scores 84.6% under correct Greek and 82.6% under deliberately wrong instructions, training loss fails to predict Greek success, and single-run comparisons are dominated by seed variation. On a discriminative ninety-task suite with three seeds per arm, a multilingual text tower without Greek demonstrations remains at its wrong-instruction floor, while Greek-only training exceeds its control by at most 2.7 points. Bilingual training yields a consistent 6.7-7.1 point margin over its control and reaches about two fifths of English performance. The policy also overfits the translator's phrasing; training on seven phrasings per task approximately halves this penalty. Warm-starting from a language-adapted world model and unfreezing the text tower both degrade performance. The results support two practical requirements for low-resource robot-policy localization: build a guaranteed null before trusting a metric, and replicate low-resource-language results across seeds.