J-ORA: A Framework and Multimodal Dataset for Japanese Object Identification, Reference, Action Prediction in Robot Perception

AI summary

Overview

• Research area: Robotics and multimodal perception, specifically vision-language models (VLMs) applied to egocentric human-robot dialogue in Japanese.
• Technical level: Advanced (assumes familiarity with VLM fine-tuning, coreference/reference resolution, and benchmark evaluation).
• One-sentence scope: The paper introduces J-ORA, a Japanese multimodal dataset of object-attribute annotations built on top of the J-CRe3 human-robot dialogue corpus, and uses it to evaluate proprietary and open-source VLMs on three robot perception tasks: object identification, reference resolution, and next-action prediction.

What This Paper Is About

Robots operating in homes need to recognize objects, understand what a person is referring to in conversation, and decide what to do next. Existing datasets and VLMs are trained mostly on object labels or captions, which the authors argue is not enough to generalize to dynamic scenes where objects change state, appear, and disappear. The core goal is to build a rich, attribute-annotated Japanese dataset for real egocentric robot-view dialogue and to measure how much that attribute information actually helps VLMs perform the three perception tasks.

Key Contributions

1. An attribute annotation schema for indoor robot assistance. The authors define a template of object attributes (category, color, shape, size, material, surface texture, position, state, functionality, brand, interactivity, and proximity to the person) and apply it to a real Japanese dialogue dataset. They state that no large-scale dataset of this kind of attribute information exists for Japanese.
2. Construction of J-ORA. A multimodal dataset providing object attribute annotations—1,817 attribute annotations across 160 unique object classes—within Japanese human-robot dialogue scenarios derived from J-CRe3.
3. A measurement of how object attributes affect VLM performance. The authors evaluate proprietary models (GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro), English-developed open models (LlaVa-v1.6-Vicuna-13B, LLaVA-v1.6-Mistral-7B, Qwen2-VL 7B Instruct), and Japanese open models (EvoVLM-JP-v1-7B, Japanese-stable-vlm 7B, bilingual-gpt-neox-4b-minigpt4) on their three tasks.
4. Joint optimization of the three tasks. They implement fine-tuning of VLMs with a combined objective, L_joint = λ_obj · L_obj + λ_ref · L_ref + λ_act · L_act, to realize an end-to-end multimodal perception model for robots.

Main Findings

• Object attributes improve performance. Incorporating detailed object attributes substantially improves multimodal perception compared to running without object attributes, as shown in the fine-tuning comparisons between Table III (without attributes) and Table IV (with attributes).
• A large proprietary-versus-open gap remains. Under zero-shot settings, GPT-4o achieves the best average score of 90.9%, followed by Claude 3.5 Sonnet at 88.6% and Gemini 1.5 Pro at 85.8%. The best open model, Qwen2-VL 7B Instruct, reaches 61.1%—a difference of 29.8% versus GPT-4o. The open-source averages were 43.8% (LlaVa-v1.6-Vicuna-13B), 29.1% (LLaVA-v1.6-Mistral-7B), and 48.1% (EvoVLM-JP-v1-7B).
• Some Japanese VLMs failed entirely. Japanese StableVLM 7B and Bilingual-gpt-neox-4b-minigpt4 only generated unintelligible responses, and Bilingual-gpt-neox-4b-minigpt4 did not improve even after fine-tuning, so it was excluded from the results.
• Fine-tuning helps almost every open model, and attributes help more. With attributes (Table IV), Qwen2-VL 7B Instruct reached 71.2% average accuracy (Δ +10.1 vs. baseline), LlaVa-v1.6-Vicuna-13B 56.2% (Δ +12.4), LLaVA-v1.6-Mistral-7B 41.3% (Δ +12.2), EvoVLM-JP-v1-7B 56.6% (Δ +8.5), and Japanese-stable-vlm 7B 36.3% (Δ +36.3, the largest gain). Without attributes (Table III), averages were 54.6, 31.6, 56.3, 53.9, and 19.6 respectively.
• Affordance does not track perception accuracy. In the zero-shot affordance evaluation (Table V), Claude 3.5 Sonnet had the highest composite affordance score at 68.3%, despite GPT-4o leading the perception tasks; GPT-4o scored 51.4%, Gemini 1.5 Pro 46.7%, LlaVa-v1.6-Vicuna-13B 61.3%, LLaVA-v1.6-Mistral-7B 18.2%, Qwen2-VL 7B Instruct 24.9%, and EvoVLM-JP-v1-7B 34.2%. VLMs mostly predicted unsophisticated actions (e.g., putting and removing books on a shelf) and omitted more complex ones.
• English dialogue text raises results for most models. When dialogues were translated into English (Table VII, omitting the Japanese-specific VLMs because they have limited English ability), averages were 89.4% for Claude 3.5 Sonnet (Δ +0.8), 88.8% for GPT-4o (Δ −2.1), 87.1% for Gemini 1.5 Pro (Δ +1.3), 56.5% for LlaVa-v1.6-Vicuna-13B (Δ +12.7), 36.5% for LLaVA-v1.6-Mistral-7B (Δ +7.4), and 66.1% for Qwen2-VL 7B Instruct (Δ +5.0).
• GPT-4o as an evaluator aligns with humans. Against two human annotators reviewing 100 responses from LlaVa-v1.6-Vicuna-13B, the intraclass correlation coefficient between the human annotators and the GPT-4o evaluator was 85%.
• Error patterns differ by model class. Proprietary models occasionally mis-extracted non-physical words as objects (Gemini 1.5 Pro identified "side," "work," "confirmation," "all-nighter," and "sleep" in one case) or gave imprecise location descriptions. Open models more often repeated the instruction instead of answering, produced unintelligible or irrelevant output, or emitted random numbers without context.

Methodology in Plain English

The researchers started from J-CRe3, an existing Japanese dataset of egocentric video and dialogue recordings in which two people role-play a master and an assistant robot in three laboratory environments (kitchen, dining room, living room). J-CRe3 originally began with 180 dialogue scenarios collected from 101 crowdworkers, with 10 to 16 utterances each; after filtering, 142 scenarios remained. The J-ORA authors extended those 142 scenarios (11,024 seconds, i.e., 3 hours 3 minutes 44 seconds of video) by writing rich object descriptions for each dialogue text-and-image pair. They prompted GPT-4o through an API with the attribute template and the images to generate descriptions in JSON and in Japanese, requiring at least 10 objects per image and attention to what the person is doing and where objects sit relative to them. Human reviewers then checked every dialogue scenario and corrected inaccurate descriptions in 7 of the 142 scenarios, yielding a 95% accuracy rate. Annotation explicitly tracked four kinds of scene change: objects appearing, disappearing, and re-appearing; object state changes (e.g., a bag closed, fully open, partially open, closed again); changes in human pose and gaze; and objects placed inside other objects.

For evaluation, the authors framed the robot's job as three linked tasks under a shared VLM that is fine-tuned on all three at once. They tested models zero-shot with in-context learning prompts, then fine-tuned every open-source model on a re-formatted Japanese Visual Genome VQA dataset containing 99K images and 793K QA pairs in Japanese, using the entire data as the training set and J-ORA as the test set, across three seeds for ten epochs on two NVIDIA A100 GPUs. Accuracy was produced by using GPT-4o's API as an LLM-judge, which receives the image, dialogue texts, task prompts, model responses, and evaluation criteria, and scores each task from one to ten; these scores were converted into percentages. A separate affordance analysis scored how well models reason about what actions each major object (bag, book, technical book, document, shelf, table) affords.

Why This Matters

The paper argues that object labels and captions alone are insufficient for robots that must operate in dynamic, cluttered homes, and it provides both a concrete annotation schema and an empirical measure of how much attribute detail buys in practice. It also documents the size of the current gap between proprietary and open VLMs in a non-English setting, which matters for anyone building robots that must run locally.

Real-world applications:

• Home assistant robots. The dataset is built on dialogue scenes where a user asks a helper to place or fetch items in a living space, matching the domestic assistance scenario the framework targets.
• Object manipulation and grounding. Reference resolution maps spoken mentions to specific image regions, which is a prerequisite for a robot to reach for or hand over the right object.
• Action planning in changing scenes. Tracking object state changes (a bag opening, documents being inserted) supports predicting the correct next action rather than reacting to a static snapshot.
• Affordance-aware interaction. Understanding what an object can be used for helps robots go beyond simple pick-and-place toward contextually appropriate behavior.

Industry relevance: The work offers a Japanese-language benchmark for the robotics and VLM communities, highlights which open models are usable for on-device perception, and shows that affordance reasoning is a separate, weaker capability from perception accuracy—useful for teams deciding where to invest fine-tuning effort.

Future Directions

• Dataset size. The authors explicitly name dataset size as future work, noting that model tuning is needed to use attribute information appropriately and that their framework should be able to construct larger datasets covering more situations.
• Closing the proprietary-versus-open gap. The 29.8% zero-shot gap between GPT-4o and the best open model (Qwen2-VL 7B Instruct) indicates that end-to-end multimodal perception in Japanese dialogue remains an open challenge.
• Improving object affordance understanding. Since high perception accuracy did not translate into high affordance scores, and models produced mostly unsophisticated action predictions, expanding how VLMs reason about the range of actions an object supports is an identified need.
• Richer attribute coverage and complexity. Because a fixed functionality attribute cannot enumerate every possible use of an object, the authors suggest VLMs must extrapolate from context—raising the question of how to annotate and evaluate more complex, less literal object functions.

Target Audience

Robotics and multimodal perception researchers, particularly those working on human-robot dialogue, egocentric vision, and Japanese-language NLP. It is also relevant to VLM practitioners evaluating model trade-offs between proprietary APIs and open models, and to engineers designing indoor service robots that need grounded reference resolution and action prediction. Readers without background in VLM fine-tuning or benchmark construction will find the dataset schema accessible but the evaluation details dense.

Authors’ abstract

We introduce J-ORA, a novel multimodal dataset that bridges the gap in robot perception by providing detailed object attribute annotations within Japanese human-robot dialogue scenarios. J-ORA is designed to support three critical perception tasks, object identification, reference resolution, and next-action prediction, by leveraging a comprehensive template of attributes (e.g., category, color, shape, size, material, and spatial relations). Extensive evaluations with both proprietary and open-source Vision Language Models (VLMs) reveal that incorporating detailed object attributes substantially improves multimodal perception performance compared to without object attributes. Despite the improvement, we find that there still exists a gap between proprietary and open-source VLMs. In addition, our analysis of object affordances demonstrates varying abilities in understanding object functionality and contextual relationships across different VLMs. These findings underscore the importance of rich, context-sensitive attribute annotations in advancing robot perception in dynamic environments. See project page at https://jatuhurrra.github.io/J-ORA/.