HumanSkill-4D: Training-Free Language Querying of Animatable Gaussian Avatars in Space and Time

AI summary

Overview

Research area: Computer vision — 3D/4D language grounding, Gaussian splatting, human avatar reconstruction, and multimodal large language model (MLLM) agents.

Technical level: Advanced. The paper assumes familiarity with 3D Gaussian Splatting, SMPL body models, CLIP-style feature matching, and MLLM prompting.

Scope: The paper defines a new task ("avatar referring segmentation") in which a language query selects both a region of an animatable Gaussian avatar (Where) and, optionally, the time intervals in which a described pose or motion occurs (When), and presents a training-free framework that answers such queries by giving a frozen MLLM per-avatar memories and shared geometric operators.

What This Paper Is About

Gaussian avatars can be animated with a driving pose sequence, but their Gaussians carry no part or garment labels and their frames carry no motion labels, so a user cannot simply ask for "the denim garment" or "the interval where both arms lift until horizontal." The paper's goal is to make such queries answerable without any per-avatar training, despite two obstacles the authors identify: region ambiguity (adjacent, overlapping parts and garments that are often referred to indirectly, by relation or function) and motion ambiguity (fine-grained motions that look alike in RGB frames). The proposed system, HumanSkill-4D, returns a traceable region mask plus matching time intervals in minutes of setup per avatar.

Key Contributions

1. A new task and framework. The authors introduce avatar referring segmentation (3D queries select a region; 4D queries also select intervals) and HumanSkill-4D, a training-free framework that hands the structure of Gaussian avatars to a frozen MLLM as an "agent skill," so every decision traces back to chosen categories and operators rather than to opaque feature similarity.

2. A semantic memory for region ambiguity. Labels from a frozen 2D human parser (Sapiens2-1B) are accumulated across views and poses into per-Gaussian category scores over 16 body-part and garment categories, combined with category names and MLLM-generated appearance descriptions. This memory also supports plain part segmentation as a by-product.

3. A motion memory and operator registry for motion ambiguity. Driving joint trajectories are stored and queried with 46 geometric templates (20 state, 26 change) across seven measurement types, adapted from PoseScript and MotionScript, plus seven temporal relations. The MLLM composes these into executable programs, and the registry can be extended by composing or adding entries.

4. Three human-annotated benchmarks. ZJU-Refer-3D (built on ZJU-MoCap, with dense part annotations and 20 referring queries), HuMMan-Refer-4D (built on HuMMan, converting its fine-grained body-part motion descriptions into 20 queries), and ZJU-Refer-4D (20 queries on ZJU-MoCap). HumanSkill-4D is reported as the most accurate on all three.

Main Findings

• Best part segmentation on every avatar. On ZJU-Refer-3D over 16 categories and nine avatars (313, 315, 377, 386, 387, 390, 392, 393, 394), the method reaches overall OA 94.2, mAcc 79.5, and mIoU 75.0, versus 4D-DRESS (78.6 / 35.7 / 31.2), OpenHuman3D (66.6 / 51.8 / 41.2), OpenHuman4D (77.6 / 52.8 / 40.0), and Find3D (14.9 / 31.5 / 12.7). It has the highest mIoU on all nine avatars.

• Large margin on indirect references. In 3D referring segmentation, HumanSkill-4D scores 82.4 mIoU overall (Part 78.4, Appearance 89.9, Relation/Function 55.1). The strongest baseline overall is LangSplat at 56.2, and even LangSplat with the authors' Sapiens2 masks reaches only 61.9; ReferSplat scores 37.4 (35.3 with the parser), OpenHuman3D 44.2, Dr. Splat 13.6 (39.9 with the parser), and Find3D 32.7. The largest gap is on relation/function queries, where the best baseline result is ReferSplat's 12.7 with the parser.

• Better joint region-and-time selection. In 4D referring segmentation averaged over queries from ZJU-Refer-4D and HuMMan-Refer-4D, the method obtains vIoU 45.0, tIoU 62.2, precision 73.4, and recall 86.0. Baselines: 4-LEGS (1.9 / 13.6 / 17.6 / 35.7), 4D LangSplat (4.9 / 17.7 / 22.6 / 45.3), and OpenHuman4D (5.4 / 22.1 / 22.1 / 100.0). OpenHuman4D keeps predictions active throughout the clip, which yields 100% temporal recall but lowers tIoU and vIoU.

• MLLM-based category selection beats text-embedding retrieval. Replacing the MLLM with CLIP, E5, or JINA-v5 top-three retrieval over the same category descriptions drops 3D mIoU to 34.9, 41.4, and 63.1, and 4D vIoU to 13.1, 13.3, and 19.4, respectively, against 82.4 / 45.0 for Qwen3.8-27B.

• MLLM choice matters more in 3D than 4D. Qwen3.8-27B and Qwen3.6-27B both reach 82.4 mIoU (44.3 and 45.0 vIoU); Gemma 4-31B reaches 45.0 vIoU but only 55.3 mIoU; Mistral 3.2-24B reaches 58.6 mIoU and 38.8 vIoU.

• Stable to the score threshold. With τ from 0.2 to 0.5 the results vary only slightly (mIoU 82.2–82.4, vIoU 44.6–45.0); at τ = 0.1 they dip to 80.7 and 43.2, which the authors attribute to weakly supported Gaussians leaking across boundaries.

• Minutes of setup, small memories. Time to the first query is about 4 minutes in 3D and about 8 minutes in 4D — the shortest among compared methods — with about 5 MB of retained memory in 3D and about 10 MB in 4D, described as nearly 10× smaller than the most compact 4D baseline (4D LangSplat at about 94 MB). Comparable baselines take roughly 32 to 369 minutes of setup, with OpenHuman3D, OpenHuman4D, and 4-LEGS retaining hundreds to thousands of megabytes.

Methodology in Plain English

Offline, per avatar. The system starts from an avatar reconstructed with 3DGS-Avatar, whose Gaussians live in a canonical space and keep their identity across poses. It renders that avatar from a set of camera-frame pairs and runs a frozen 2D human parser on each rendering. Each pixel's label is weighted by how much each Gaussian contributes to that pixel (its opacity times the transmittance of everything in front of it), and these weighted votes are summed into a score per (Gaussian, category). The score is essentially the share of a Gaussian's visible contribution that carried a given label, so no training is involved. A threshold τ = 0.4 decides which categories a Gaussian belongs to. The frozen MLLM also looks at four camera views near a T-pose and writes a short appearance description per category (color, texture, graphics, coverage), so queries like "the denim garment" can match.

Offline, shared across avatars. A motion memory stores SMPL joint positions, root orientation, and root translation for all frames, expressed in a fixed coordinate frame defined by the initial root position, gravity, and initial facing direction. An operator registry holds 46 templates organized by seven measurement types (joint angle, relative position, displacement, root rotation, segment tilt, distance, height above the lowest joint) plus seven temporal relations. Each template is a rule — a measurement plus a condition — with a text card describing it in words; applying a template to a specific body part produces an executable operator. State templates select frames where a measurement falls in a valid range; change templates first find candidate intervals where a measurement moves in the requested direction, then keep those exceeding a minimum total change and a minimum length of L_min = 4 frames.

Online. The MLLM receives a text-only prompt with the query, the category names and descriptions, and a few in-context examples — and, for 4D queries, the registry's text cards. It returns target categories and, for 4D, a small program that applies templates to body parts and links the resulting operators with temporal relations (co-occur, follow in order, alternate, or a state holding at the end of, throughout, or during a movement). The category scores turn the chosen categories into a spatial mask; a deterministic executor runs the program on the joint trajectories to produce a temporal mask; multiplying them gives the spatiotemporal selection. Because the MLLM picks categories and operators rather than pixels or frames, a user can see why a selection was made.

Why This Matters

Impact on research. The work reframes language grounding for avatars away from per-scene CLIP feature fields toward structured, reusable knowledge (memories) plus shared procedures (operators), and it argues that RGB evidence is the wrong substrate for fine-grained motion, since motions can look alike in pixels while differing in dynamics. It also supplies three annotated benchmarks for a task that previously had no evaluation, and shows a training-free method beating per-avatar-optimized baselines on both accuracy and setup cost.

Real-world applications (as motivated or implied by the paper):

• AR/VR and sports analysis, where users inspect a specific body part or garment during a specific motion.
• Appearance editing and clothing transfer, which need the exact region to modify identified by language rather than by manual painting.
• Motion inspection and coaching, where a query isolates the intervals in which a described pose condition holds.
• Local avatar editing pipelines that want traceable, auditable selections rather than opaque feature similarity.

Industry relevance. A training-free system needing minutes and roughly 5–10 MB per avatar is far cheaper to deploy than per-avatar feature optimization that can take hours and, in 4D, gigabytes of storage. The skill formulation — frozen MLLM plus swappable memory and an extensible operator registry — maps cleanly onto product settings where new garments, new avatars, or new motion queries must be added without retraining.

Future Directions

• Extending the operator vocabulary. The authors present extensibility as a design goal: new pose and motion conditions should be addable by composing existing templates and relations or registering new ones, which invites work on how large and how automatically discoverable that registry can become.

• Broadening beyond the fixed 16 categories. The category set is fixed by a mapping of the parser's labels, merging upper and lower arms, upper/lower legs and feet, and lips; finer or open-vocabulary category sets are an open question.

• Robustness of the frozen MLLM as the interpreter. The MLLM ablation shows a 3D mIoU spread from 55.3 to 82.4 across models, so how much of the pipeline can rely on smaller or weaker models — the paper defers smaller MLLMs to its supplement — remains unresolved.

• Limitations proper. The main text defers limitations and future directions to Supplementary Sec. E and does not state them in the content available here; what they are is not reported in this portion of the paper.

Target Audience

Researchers and practitioners in 3D/4D computer vision working on Gaussian splatting, language grounding, and animatable human avatars; MLLM-agent researchers interested in skill-based, tool-composing pipelines over structured geometry; and graphics/AR-VR engineers who need language-driven selection and editing of avatar regions and motions without per-asset training. Readers should be comfortable with Gaussian splatting rendering equations, SMPL joint conventions, and standard segmentation metrics (mIoU, OA, mAcc, vIoU, tIoU).

Authors’ abstract

We introduce avatar referring segmentation, which selects regions of animatable Gaussian avatars from language queries. A query can also specify a pose or motion to select matching intervals. This lets users interact with avatars for local editing and motion inspection. However, making these selections with existing language Gaussian splatting methods poses two challenges: adjacent parts and garments are hard to separate and may be referred to indirectly (region ambiguity), while fine-grained motions can look alike in RGB (motion ambiguity). We address these challenges with HumanSkill-4D, a training-free framework that shares a multimodal large language model (MLLM) and geometric operators across avatars. For each avatar, a semantic memory links body-part and garment categories to persistent Gaussian regions and appearance descriptions. The MLLM uses category names and descriptions to select categories by appearance, relation, or function. To find matching intervals, it composes operators into programs that check query conditions on driving joint trajectories stored in a motion memory. These category choices and operator outputs make selections traceable, while composing or adding operators supports new motion conditions. To evaluate this task, we introduce three human-annotated benchmarks and show that HumanSkill-4D outperforms the baselines in part segmentation and 3D and 4D referring segmentation, with compact memories and only minutes of setup per avatar.