Limited Linguistic Diversity in Embodied AI Datasets

AI summary

Overview

Research area: Natural Language Processing and robotics — specifically the linguistic properties of instruction data used to train Vision-Language-Action (VLA) models.

Technical level: Intermediate. The paper is a dataset audit rather than a modeling contribution; it combines standard NLP diversity metrics with robotics-domain analysis, so readers need some familiarity with embeddings, part-of-speech patterns, and VLA training pipelines.

Scope: A systematic linguistic audit of the instruction text in widely used VLA datasets (RT-1, BRIDGE, TacoPlay, Language Table, and LIBERO), benchmarked against language-focused robotics datasets (ALFRED, SCOUT) and general instruction-tuning corpora (OASST2, Alpaca, LLaVA-Instruct) across lexical, semantic, and structural dimensions.

What This Paper Is About

VLA models like OpenVLA, RT-X, and pi-0.5 use natural language as a core input, and their training corpora (notably the Open X-Embodiment collection) are documented mainly in terms of objects, scenes, and embodiments rather than the language itself. The authors ask a simple but under-examined question: what kinds of instructions do these datasets actually contain, and how much linguistic variety do they offer? Their goal is to produce descriptive documentation of the available language signal so that future dataset reporting, selection, and augmentation can be more principled.

Key Contributions

1. A multi-dimensional diversity evaluation for embodied AI instructions. The authors construct an audit spanning three analyses — (A.1) duplication and lexical diversity, (A.2) semantic diversity, and (A.3) structural diversity — following the form-versus-content distinction of Tevet and Berant (2021) and drawing on metrics such as BLEU, ROUGE-L, Jaccard similarity, Levenshtein distance, compression ratio, BERTScore, PCA intrinsic dimensionality, POS patterns, constituency tree kernels, and grammatical-phenomena annotation.

2. A comparative characterization of ten datasets across three regimes. They quantify instruction language in five VLA datasets (four from OXE plus LIBERO), two language-focused robotics datasets, and three instruction-tuning datasets, using the latter two groups as interpretive reference points rather than as definitions of "ideal" data.

3. Domain-specific semantic probes for manipulation and navigation. Beyond embedding-based metrics, they analyze verb–direct-object co-occurrence in manipulation datasets and adverbial usage (direction, location, manner) in navigation data to surface interpretable, robotics-relevant patterns.

4. Concrete implications for augmentation and data collection. The findings are translated into three proposed strategies: targeted augmentation, cross-domain transfer, and annotation guidance.

Main Findings

• Extreme duplication in VLA instructions. Fewer than 2% of language instructions in the VLA datasets analyzed contain unique wording. RT-1 has 3.7M+ sentences but only 577 unique (0.02%); TacoPlay has 214K sentences and 403 unique (0.2%); Language Table has 7.0M+ sentences and 127,370 unique (1.81%); LIBERO has 6.5K sentences and 112 unique (1.72%); BRIDGE has 864K+ sentences and 11,693 unique (1.4%). This is largely because the same command is paired with multiple action trajectories across trials.

• Non-VLA robotics datasets are more diverse. The lowest uniqueness outside the VLA datasets is SCOUT at 39.4% (8,795 unique out of 23K+), and ALFRED at 79.9% (126,005 out of 162K+). SCOUT's short utterances — averaging fewer than five words — are offered as a likely reason for its higher repetition relative to other non-VLA datasets.

• Instruction-tuning corpora score high on uniqueness. Alpaca reaches 99.81% unique (52,996 of 53K+), OASST2 reaches 93.33% (39,301 of 42K+), and LLaVA-Instruct reaches 71.45% (261,892 of 366K+).

• Compression ratio separates the groups most sharply. CR values are lowest (most diverse) for OASST2 (2.75) and Alpaca (3.20), mid-range for LLaVA-Instruct (4.41), SCOUT (4.85), and ALFRED (5.91), and dramatically higher for VLA data: Language Table 56.64, BRIDGE 64.90, RT-1 118.20, LIBERO 134.86, and TacoPlay 158.86.

• Lexical breadth is very narrow. RT-1's entire unique vocabulary is only 49 words. BRIDGE has the largest unique word count among VLA datasets at 1,189, exceeding Language Table's 928 despite Language Table having roughly ten times more unique commands. TacoPlay has 74 unique words and LIBERO 79. Only four words appear in all datasets: move, close, open, and pick.

• Semantic coverage is compressed. Using USE embeddings and measuring the minimum number of PCA components needed to explain 95% of cumulative variance, VLA datasets require the fewest components: RT-1 (33), LIBERO (34), TacoPlay (42), Language Table (86), BRIDGE (125). Reference points are ALFRED (159), SCOUT (148), LLaVA-Instruct (184), Alpaca (231), and OASST2 (254). All four tested encoders (USE 512D, SBERT 768D, CLIP 512D, SONAR 1024D) show similar results. The authors note this measure correlates strongly with the number of unique unigrams but only weakly with the number of unique commands.

• Verb–object pairings are lopsided. In the manually annotated RT-1 data, "banana" is frequently "picked" but never "moved," and the rare verb "knock" appears mostly with can-shaped objects even though it would apply to items like an upright sponge. Across datasets, most objects co-occur with fewer than ten verbs, indicating limited task diversity; ALFRED and Language Table show more balanced distributions. The authors argue some constraints are artificial — for example, TacoPlay's stacked blocks could support richer interactions such as "observe" or "tip."

• Navigation language skews directional. In SCOUT, adverbial usage is heavily weighted toward directionals such as "north," "forward," and "south"; the authors suggest more attention to manner adverbs like "fast" or "slow" could help.

• Structural forms are templated and logically thin. Most frequent POS patterns typically begin with a verb followed by object descriptions plus an adposition and a second object; in RT-1 the top pattern covers 11% of instances and in TacoPlay 24%. Multi-step instructions are the most prevalent structural phenomenon and dominate LIBERO, whereas RT-1 and SCOUT favor shorter atomic actions. Negation and conditionals are rare — the abstract states that across all studied datasets fewer than 1% of commands contain negation, while Section 5 describes negation and conditional constructions as occurring in fewer than 2% of cases. Cyclical or loop-like structures are similarly underrepresented, with only SCOUT and ALFRED showing a modest signal. Command lengths cluster between 3 and 15 words.

• Qualitative dataset quirks. An informal review identified cultural terms in BRIDGE ("put the kadai on the stove," "grab the brinjal from the drawer"), unsafe actions in ALFRED ("store a knife in a microwave"), and commonsense violations in both ALFRED ("Put an egg in a pan in the fridge") and BRIDGE ("take sushi out of the pan").

Methodology in Plain English

The authors picked a representative subset of datasets rather than trying to cover everything, because OXE alone contains over 40 datasets with language annotations. Their selection balanced how heavily each dataset is used in the literature, whether it is large enough for stable estimates, and whether it represents a distinct linguistic regime — templated versus natural, open-ended. They deliberately added datasets outside the usual VLA training mix so the numbers would have reference points.

For the measurements, they treat each instruction sentence as the unit of analysis and run three families of checks. The first is straightforward counting and text-similarity: how many sentences there are, how many are unique, how many distinct words appear, and how compressible the text is (compression ratio, ROUGE-L, BLEU, Jaccard similarity, Levenshtein distance). The second family converts instructions into vector embeddings using four different encoders and measures how much spread there is — both pairwise similarity via BERTScore and dataset-level spread via how many principal components are needed to explain 95% of the variance. They also hand-annotate verb–object pairings in RT-1 to see which actions are used with which objects. The third family looks at grammar: they extract part-of-speech patterns, compare constituency parse trees, and flag logical constructions like negation, conditionals, cycles, and multi-step sequences.

Pairwise scores are computed by sampling 1,000 commands per dataset and repeating the sampling three times for robustness. For the grammatical-phenomena counts, datasets with fewer than 600 unique sentences were annotated manually, while larger ones used an automated pipeline; standard error bars come from manually reviewing 500 randomly sampled commands per dataset.

Why This Matters

Impact on research. The paper reframes language as an under-documented axis of VLA dataset quality. Generalization failures reported elsewhere — sensitivity to paraphrases, degradation with distractor objects — become easier to contextualize when the training data contains only a handful of unique phrasings. The audit gives dataset creators a vocabulary and a metric suite for reporting language coverage alongside object and scene coverage, and it points to structural gaps (negation, conditionals, loops) that current benchmarks simply do not exercise.

Real-world applications.

• Household and service robots that must handle instructions like "give me the apple that is not rotten" or "if you have picked up the apple, wash it" — constructions that appear in under 2% (and by the abstract's count, under 1% for negation) of the audited data.
• Warehouse and logistics automation, where multi-step and conditional task logic ("if the shelf is full, place it on the cart") is routine but absent from training language.
• Safety-critical deployment, where understanding negation and exceptions matters for avoiding misuse of tools or unsafe object combinations of the kind the authors found in ALFRED and BRIDGE.
• Dataset and benchmark design, where the proposed augmentation and annotation-guidance strategies could be applied when building or refreshing new corpora.

Industry relevance. Organizations training generalist robot policies pay heavily for data collection, and this analysis suggests that adding unique phrasings may be cheaper than adding episodes — since RT-1's 3.7M+ sentences collapse to 577 unique commands, extra episodes of the same instruction add little language signal. The 49-word vocabulary of RT-1 also implies that a language backbone fine-tuned on such data has very little linguistic material to bind to actions, which is directly relevant to teams choosing datasets or building LLM-based augmentation pipelines.

Future Directions

• Targeted augmentation. Use the structural analysis — tree-kernel similarity or POS pattern histograms — to guide LLM paraphrase generation via phrase permutation or inversion, and use LLM-guided synonym replacement to break the lexical bottlenecks identified in Analysis 1.
• Better data collection interfaces. Prompt human operators in real time to rephrase instructions, diversify verb–object pairings during collection (motivated by RT-1's skewed co-occurrence patterns), and incorporate adverbial phrases to counteract clustered POS patterns.
• Cross-domain transfer. Selectively import linguistically richer corpora — procedural text, situated dialogue — to complement VLA data, and study whether interactive collection methods such as SCOUT's Wizard-of-Oz approach can yield comparable lexical and structural gains.
• Open questions the authors flag. Whether these linguistic gaps causally affect VLA performance is not established; the paper is descriptive. The authors also note their study covers only text, so cross-modal alignment between commands and trajectories or visual scenes is unexamined, and the findings may not generalize to all embodied instruction-following datasets.

Target Audience

Robotics and VLA researchers who select or train on OXE-derived corpora; dataset creators and benchmark designers who need a reporting standard for instruction language; NLP researchers interested in how instruction-tuning data properties differ from embodied data; and practitioners deciding how to allocate budget between collecting new episodes and diversifying the language attached to existing ones. Readers focused on narrow, constrained task domains may find the patterns less applicable, as the authors themselves acknowledge.

Authors’ abstract

Language plays a critical role in Vision-Language-Action (VLA) models, yet the linguistic characteristics of the datasets used to train and evaluate these systems remain poorly documented. In this work, we present a systematic dataset audit of several widely used VLA corpora, aiming to characterize what kinds of instructions these datasets actually contain and how much linguistic variety they provide. We quantify instruction language along complementary dimensions--including lexical variety, duplication and overlap, semantic similarity, and syntactic complexity. Our analysis shows that many datasets rely on highly repetitive, template-like commands with limited structural variation, yielding a narrow distribution of instruction forms. We position these findings as descriptive documentation of the language signal available in current VLA training and evaluation data, intended to support more detailed dataset reporting, more principled dataset selection, and targeted curation or augmentation strategies that broaden language coverage.