VERAGMIL: Virtual Environment for Scooping Granular Foods with Imitation Learning Models

AI summary

Overview

Research area: Assistive robotics (Robot-Assisted Feeding), imitation learning, virtual reality teleoperation, and high-fidelity physics simulation of granular materials.

Technical level: Intermediate. The paper assumes familiarity with imitation learning terminology (behavioral cloning, recurrent policies, batch-constrained Q-learning), robot control stacks (inverse kinematics, joint-space PD control), and simulation frameworks (Omniverse, IsaacSim/IsaacLab, PhysX).

Scope in one sentence: The paper introduces and evaluates VERAGMIL, a VR-based simulation framework for collecting human demonstrations and training imitation learning policies that let a robot scoop and transport granular foods such as rice, buckwheat, barley, and pasta.

What This Paper Is About

Robot-assisted feeding systems handle rigid and semi-solid foods reasonably well, but granular foods like rice, peas, and pasta are difficult because they behave partly like solids and partly like fluids and respond nonlinearly to external forces. Manually designing controllers for such materials is hard, and collecting large numbers of high-quality human demonstrations in the physical world is labor-intensive and costly.

VERAGMIL addresses this by pairing a GPU-accelerated, physically detailed granular-food simulator with an intuitive VR interface (natively supporting the Oculus Quest 2) for recording demonstrations, then using those demonstrations to train three imitation learning models and compare them against a human-expert baseline.

Key Contributions

1. A VR-based simulation framework for granular food handling in RAF. The framework combines a high-fidelity physics simulation environment with a VR interface designed to capture precise human demonstrations. The complete system is publicly released at https://github.com/AmanuelErgogo/VERAGMIL.git.

2. An empirical evaluation of three imitation learning models (BC, BC-RNN, BCQ) trained on the acquired human demonstrations for granular scooping and transport tasks, benchmarked against a human baseline and assessed on generalization to unseen food items.

3. A study of how the demonstration-acquisition device affects learned policy quality, comparing VR-based demonstrations against 3D space mouse demonstrations and showing that VR-trained policies achieve near-human performance and better generalization.

4. An integrated, reusable environment built as an Omniverse extension on top of the IsaacLab framework, bundling robots, end-effectors, sensors, and a large set of granular foods modeled with individually specified physical parameters.

Main Findings

• BCQ was the strongest model overall. It consistently outperformed BC and BC-RNN across success rate, spillage, and generalization. In the VR interface (VRI) scenario, BCQ achieved an average success rate of over 88% on food items seen during training and more than 68% on unseen items, approaching the human success rate of 95%.

• BC and BC-RNN degraded more on unseen foods. BC-RNN performed well on seen items with an average success rate of 84.5%, but dropped by roughly 30.95% on average when applied to unseen items. BC averaged 76.25% on seen items and reached only 40% success on unseen items in the 3D mouse (3DM) scenario.

• VR demonstrations beat 3D space mouse demonstrations. VR-based training produced higher success rates and lower spillage across all models. BCQ showed a 15% improvement in success rate and a 10% reduction in spillage when trained on VR data versus 3D mouse data. BC, previously the weakest in 3DM, gained 16% success with VR training.

• BCQ reduced spillage but did not match humans. BCQ demonstrated a 20% reduction in spillage across all materials in the VRI scenario, yet spillage remained nearly three times that of human demonstrations (8 pieces for BCQ-VRI versus 3 for humans). The advantage was most visible with larger items such as pasta.

• 3D mouse demonstrations were faster but less precise. Task completion times were generally faster for all models in the 3DM scenario, but this came with a noticeable decline in precision and increased spillage, with BC showing the largest increase (45 pieces spilled versus 20 in VRI).

• Completion times were stable for BCQ. In the VRI scenario, BCQ completed tasks in 15 seconds on average, slightly slower than BC's 20 seconds but with considerably less spillage. Human performance in the same scenario was 14 seconds.

• Aggregate success rates across all materials: BC 64% (VRI) and 47% (3DM); BC-RNN 73% and 58%; BCQ 80% and 68%; humans 95% and 89%.

Methodology in Plain English

The researchers built a simulation environment on top of the IsaacLab framework as an Omniverse extension, using the NVIDIA PhysX 5 engine for GPU-accelerated real-time simulation. Each granular food is modeled as a set of rigid bodies using Position Based Dynamics, with per-item physical parameters (shape, size range, friction coefficient, restitution, and density) stored in Universal Scene Description format, and with Signed Distance Field mesh colliders matching each item's geometry to produce accurate contact with the spoon.

The robot used in the experiments is a Ufactory xArm7 equipped with a standard spoon as the end effector (the framework also supports a Unitree G1 humanoid and customizable end effectors). Control flows from the VR interface at 60 Hz as a 6-DoF command, which is converted into 7-DoF joint commands through task-space control with inverse kinematics, then executed by a joint-space PD controller running at 1000 Hz.

A human demonstrator wears an Oculus Quest 2. The headset's vision system tracks the 6-DoF poses of the headset and both controllers; these poses are transformed into the simulation's global frame, filtered to remove high-frequency jerks, and sent to the simulation at 100 Hz. The interface streams RGB-D frames from the simulation back to the operator at 30 Hz so they can see what they are doing. Operators can select relative or absolute control commands, adjust command frequency, and change scaling.

Demonstrations were recorded for scooping and transporting with two input devices: the VR interface and a 3D space mouse. Each demonstration captured the robot's 7-DoF joint positions, end-effector pose, desired post-scoop pose, RGB-D images from a fixed overhead camera, and the end effector's relative movements. Each demonstration lasted 14-19 seconds (840-1,140 time steps at 60 Hz). The team collected 30 demonstrations per food item (120 per dataset) from a single human demonstrator, totaling 28-38 minutes of demonstration time and 42-76 minutes of overall collection time including resets.

Training foods were rice, buckwheat, barley, and pasta. Peas, beans, and flakes were held out to test generalization. Three imitation learning methods were trained: Behavioral Cloning (BC), which minimizes the squared error between predicted and demonstrated actions; BC-RNN, which adds a recurrent structure modeling temporal dependencies; and BCQ, which adds a Q-function that estimates future reward and selects actions maximizing that Q-value. All experiments ran on a computer with 16 GB RAM, an Intel Core i7-14700K CPU, and an RTX 4060 GPU.

Evaluation used four metrics: success rate (percentage of tasks completed with less than 5% of total volume spilled), spillage (volume of food lost, approximated as the number of lost pieces multiplied by each piece's size), generalization (performance on unseen foods), and task completion time (from the initial pre-scoop pose to the post-scoop pose).

Why This Matters

Impact on research. The paper pushes robot-assisted feeding beyond rigid foods into granular materials, a regime where conventional fixed-trajectory and model-based adaptive controllers struggle because granular media are hard to model. It also provides a direct comparison of demonstration-acquisition hardware (VR headset versus 3D space mouse) as a factor in imitation learning quality, and it shows the whole stack can run on modest consumer hardware (16 GB RAM, i7-14700K, RTX 4060), lowering the barrier to entry for RAF research. The public code release makes results reproducible and extensible.

Real-world applications:

• Assistive feeding robots that help people with motor impairments, disabilities, or age-related limitations eat foods like rice, beans, and pasta independently, reducing the time caregivers spend on feeding.
• Broader granular-material manipulation in food service, such as portioning, plating, or packaging loose foods in commercial kitchens.
• Agricultural and industrial bulk handling, where scooping and transporting particulate material without spillage is a recurring task.
• Training data generation for manipulation policies, where a VR interface can replace slow and costly physical-world demonstration collection.

Industry relevance. The framework targets commercial assistive-care robotics and food automation, both of which are constrained by the scarcity of high-quality demonstration data. Because it is built on NVIDIA's Omniverse/IsaacLab/PhysX stack, it aligns with widely used simulation infrastructure, and the demonstration that VR beats a 3D space mouse gives equipment and interface designers a concrete signal about where to invest for teleoperation and data collection.

Future Directions

• Sim-to-real transfer and physical validation. The authors state they are actively working to extend validation to the real environment and to adopt techniques that reduce the gap between simulation and reality. No real-robot experiments are reported in this paper.

• Broader model comparison and demonstration-free learning. The authors plan to extend validation to additional imitation learning methods and to adopt demonstration-free reinforcement learning techniques to see which approach is more effective for RAF.

• A formal user study of the VR interface. The authors plan an extended user study including objective usability and workload evaluation to more clearly confirm the effectiveness of the proposed VR interface.

• More realistic and more varied food simulation. Planned improvements include more granular foods and modeling their interaction with liquids of different densities and viscosities, in order to represent soups or mixed-composition foods.

• Closed-loop fine-grained control. The paper notes that although spillage is significantly reduced, fine-grained control over small portions of granular materials still requires improvement.

Target Audience

This paper is most useful to robotics researchers working on assistive feeding, manipulation of deformable or granular materials, and imitation learning; to engineers building teleoperation or demonstration-collection pipelines and choosing between VR and desktop input devices; to simulation developers working with IsaacSim, IsaacLab, Omniverse, and PhysX who need a granular-material use case; and to practitioners in assistive-care or food-automation industries evaluating whether simulation-based training is a viable path toward deployable feeding systems. Readers with no background in imitation learning or robot control will find the model-comparison details harder to follow, though the high-level results and the VR-versus-mouse comparison remain accessible.

Authors’ abstract

Robot-Assisted Feeding (RAF) systems are essential for assisting individuals with disabilities or motor impairments in eating tasks. Manipulating granular food items, such as rice and beans, poses significant challenges due to their dynamic physical properties. Learning from human demonstrations offers a promising solution, but acquiring high-quality demonstrations is complex. To address this, we present VERAGMIL, a framework that combines a high-fidelity simulator with an intuitive Virtual Reality (VR) interface for recording demonstrations and supporting different imitation learning methods. VERAGMIL provides a realistic environment for training RAF systems to handle granular materials, including robots, sensors, and various food items with distinct physical characteristics. We evaluate VERAGMIL by training three imitation learning models, BC, BC-RNN, and BCQ, on granular scooping and transporting tasks using both VR interface and 3D space mouse demonstrations, comparing them with a human-expert baseline. The models are assessed on success rate, spillage, generalization to unseen food items, and task completion time. Results show that VR-based demonstrations significantly outperform 3D space mouse data, with BCQ achieving the best overall performance, particularly in reducing spillage and approaching human performance. These findings underscore the effectiveness of our framework for training RAF systems in granular material handling. The code for our framework is publicly available at: https://github.com/AmanuelErgogo/VERAGMIL.git.