Holi-World: Building Holistic World Intelligence from Human-Ego Experience

AI summary

Overview

• Research area: Computer Vision — specifically egocentric (first-person) video understanding, 4D/scene-level world representation, automated data construction, and vision-language model training.
• Technical level: Advanced. The paper describes a multi-stage system spanning data capture hardware, perception pipelines, automated annotation, model training, and benchmark construction. Readers will get the most from it with background in 3D scene understanding, tracking, and multimodal model training.
• Scope (one sentence): The paper introduces an integrated ecosystem — capture, representation, automated data construction, model learning, and evaluation — designed to convert continuously recorded human egocentric experience into large-scale learning resources for models that understand how the physical world is structured and how it changes.

What This Paper Is About

Human first-person recordings contain a great deal of information about how scenes are laid out and how objects change as people interact with them, but that information is locked inside raw video and is expensive to label by hand. The authors want to turn this growing supply of recordings into usable training material that ties together scene structure, object meaning, and interaction history across different viewpoints and across time. Their goal is a full pipeline that automatically builds such resources and then uses them to train and independently evaluate a model with stronger spatial and temporal understanding.

Key Contributions

1. An integrated ecosystem, HOLI-WORLD, that links five stages that are usually handled separately: experience capture, world representation, automated data construction, model learning, and independent evaluation.
2. HOLIWORLD-Foundry, an automatic construction pipeline that combines complementary static and dynamic perception to produce "world records" — representations that connect persistent scene structure and evolving object states with interaction histories and supporting observations.
3. HOLIWORLD-Data, a grounded learning resource built from roughly 100 self-collected scenes, including approximately 80K question-answer pairs used to train the vision-language model HOLIWORLD-VL-4B.
4. HOLIWORLD-Bench, an evaluation set built from independent human annotations of held-out recordings, which supports assessment of both the quality of the automatically produced annotations and the spatial and temporal understanding learned by the model.

Main Findings

• Strong perception and tracking results: Experiments on ScanNet, ScanNet++, Aria Digital Twin, and HOLIWORLD-Bench show strong performance in static perception and dynamic tracking. The abstract does not report specific scores or comparisons against named baselines.
• Data improves reasoning: Training HOLIWORLD-VL-4B on HOLIWORLD-Data substantially improves its spatial and temporal reasoning, measured on HOLIWORLD-Bench and on EgoDynamic4D. Numeric gains are not stated in the abstract.
• Annotation quality is itself evaluated: Because HOLIWORLD-Bench is built from independent human annotations of held-out recordings, it serves as a check on how good the automatically constructed labels are — not only on what the model learned.
• Scale is achievable without heavy manual labeling: The pipeline is presented as a way to build large grounded resources (about 100 scenes, about 80K QA pairs) from raw egocentric recordings while reducing reliance on costly human annotation.

Methodology in Plain English

The authors built a system rather than a single model. First, they collect first-person recordings of people moving through and interacting with scenes. A construction stage then runs two kinds of perception side by side: static perception, which works out the stable layout and structure of a scene, and dynamic perception, which follows how objects move and change over time. These are merged into "world records" that keep the scene structure, the changing object states, and the history of interactions together, along with the observations that support each claim. From these records, the team derives question-answer pairs and other grounded learning material in an automated way, and uses them to train a 4-billion-parameter vision-language model. To check the result fairly, they set aside some recordings, had humans annotate them independently, and used that as a separate benchmark for both annotation quality and the model's spatial and temporal reasoning.

Why This Matters

Impact on research. The paper argues that egocentric experience can be a scalable substitute for hand-labeled 3D and 4D data, and it offers a shared pipeline — data construction plus a held-out benchmark — that others could reuse or extend. By evaluating the automatically generated annotations against independent human labels, it also addresses a common weak point in auto-labeled datasets: whether the labels are trustworthy.

Real-world applications (as motivated by the work):

• Assistive and wearable systems that need to understand a user's surroundings and what the user is doing from a first-person view.
• Robotics and embodied agents that must reason about persistent scene layout and how objects change after interaction.
• Augmented and virtual reality, where scene structure and object state must be tracked and predicted over time.
• Large-scale training-data production for spatial and temporal video-language models, lowering annotation cost.

Industry relevance. Any organization sitting on large volumes of first-person video — wearable devices, AR/VR platforms, robotics, or dataset vendors — could in principle apply this capture-to-training pipeline instead of commissioning manual 3D annotation.

Future Directions

• Scaling beyond the current collection: The dataset covers roughly 100 self-collected scenes; whether the pipeline holds up on much larger and more diverse recording streams is left open.
• Broadening evaluation: The abstract reports improvements on two benchmarks (HOLIWORLD-Bench and EgoDynamic4D); more independent, third-party benchmarks would test how general the learned spatial and temporal reasoning is.
• Closing the annotation-quality gap: Because HOLIWORLD-Bench exists partly to measure automated annotation quality, an obvious next step is iterating on the Foundry pipeline to reduce whatever gap remains against human labels.
• Extending the representation: The world records couple scene structure with interaction history; extending this to longer time spans, more complex multi-object interactions, or multi-agent settings is a natural continuation.

Target Audience

Researchers and engineers working on egocentric video, 3D/4D scene understanding, embodied AI, and multimodal model training. It is also relevant to practitioners who need to build large annotated datasets from raw sensor recordings without proportional manual labeling effort, and to those evaluating spatial and temporal reasoning in vision-language models. The paper is less suited to beginners, since the abstract presumes familiarity with scene reconstruction, tracking, and vision-language model training.

Authors’ abstract

Human egocentric experience offers a rich foundation for learning how the world is organized and how it changes through interaction. Realizing this potential at scale requires turning growing volumes of recordings into learning resources that connect scene structure, object semantics, and interaction histories across viewpoints and time, while reducing dependence on costly manual annotation. We introduce HOLI-WORLD, an integrated ecosystem connecting experience capture, world representation, automated data construction, model learning, and independent evaluation. To sup- port learning at scale, HOLIWORLD-Foundry combines complementary static and dynamic perception to automatically construct world records that connect persistent scene structure and evolving object states with interaction histories and supporting observations. HOLIWORLD-Data organizes these records from approximately 100 self-collected scenes into grounded learning resources, including approximately 80K QA pairs used to train HOLIWORLD-VL-4B. Independent human annotations of heldout recordings form HOLIWORLD-Bench, supporting evaluation of both annotation quality and learned spatial and temporal understanding. Experiments on ScanNet, ScanNet++, Aria Digital Twin, and HOLIWORLD-Bench demonstrate strong performance in static perception and dynamic tracking, while training on HOLIWORLD-Data substantially improves HOLIWORLD-VL-4B's spatial and temporal reasoning on HOLI- WORLD-Bench and EgoDynamic4D. HOLIWORLD provides a practical foundation for expanding learning resources from the continuous flow of human experience and advancing machine understanding of the dynamic world.