InfiniHuman: Infinite 3D Human Creation with Precise Control

AI summary

Overview

Research area: Computer vision and graphics — controllable 3D human avatar generation, large-scale synthetic dataset construction, and diffusion-based generative modeling.

Technical level: Advanced. The paper assumes familiarity with diffusion models, score distillation sampling (SDS), Gaussian splatting, SMPL parametric body models, flow matching, and multi-view attention mechanisms.

Scope in one sentence: The paper introduces a fully automatic pipeline that distills existing foundation models into a 111K-identity annotated 3D human dataset, plus two generative models that synthesize controllable 3D avatars from text, body shape, and clothing-image inputs.

What This Paper Is About

Building realistic, controllable 3D human avatars is hard because photorealistic human scans are expensive to capture and annotate, which limits both the scale and the demographic diversity of training data. Existing methods either reconstruct from images or generate from text via slow optimization, and none allow direct control over clothing, body shape, and text simultaneously. The authors ask whether foundation models can be repurposed to synthesize theoretically unlimited, richly annotated 3D human data at minimal cost, and then use that data to train fast, precisely controllable avatar generators.

Key Contributions

1. InfiniHuman, a fully automatic framework that distills vision-language, image synthesis, pose estimation, and diffusion models to generate richly annotated human identities that the authors report are indistinguishable from real scan renderings.
2. InfiniHumanData, described as the first large-scale multi-modal human dataset with 111K diverse identities, each annotated with multi-granularity text descriptions, multi-view RGB images (full body and head), clothing asset images, and SMPL body shape parameters.
3. InfiniHumanGen, a generative framework with two models: Gen-Schnell for fast end-to-end 3D Gaussian splatting generation from text, body shape, and clothing images, and Gen-HRes for high-resolution, photorealistic textured meshes with fine-grained text control.
4. Instruct-Virtual-TryOff, a component that reverses the virtual try-on process to extract clean garment images from full-body photos, enabling clothing control from real photographs without paired image-scan training data.

Main Findings

• Benchmark performance: On a user study over 32 prompts with 42 participants, Gen-HRes was preferred for appearance quality 92.39% of the time and for text alignment 89.56% of the time, versus 2.54% and 3.48% for HumanNorm, the strongest SDS-based baseline in the table. Gen-Schnell scored 77.14% and 78.10%, versus 20.83% and 20.36% for MVDream.
• Image-quality metrics: Gen-HRes achieved the lowest FID (82.28) and a CLIP Score of 30.43, against 101.84 FID / 28.30 CLIP for HumanNorm and 156.52 FID / 28.69 CLIP for AvatarVerse.
• Speed: Gen-Schnell produces 3D Gaussian splats in about 12 seconds (12.9s in the comparison table). Gen-HRes generates high-fidelity avatars in approximately 4 minutes — the paper states this is at least 8 times less computational time than high-resolution baselines such as TADA (213m), DreamAvatar (384m), HumanNorm (117m), HumanGaussian (40m), and AvatarVerse (44m).
• Dataset realism: In a paired user study mixing real scan renderings with InfiniHumanData samples, scan renderings received 746 votes and InfiniHumanData received 765 votes, a small difference the authors interpret as near-parity in visual realism.
• Dataset scale versus prior work (Table 1): InfiniHumanData has 111K identities, compared to IDOL (100K), MVHumanNet (4500), THuman2.1 (2500), 2K2K (2050), HuMMan (1000), HUMBI (772), DNA-Rendering (500), Sizer (97), CustomHuman (80), ZJU-MoCap (10), and ActorsHQ (8). It is the only entry in the table marked as providing multi-text (fine-grained text) and cloth assets.
• Controllability: Gen-HRes follows SMPL body shape and pose, transfers clothing from garment images while preserving identity when the initial Gaussian noise is fixed, and supports fine-grained text editing of high-level attributes (ethnicity, age, gender) and accessories (glasses, garments, stockings, scarves, sunglasses).
• Cost of data generation: Automatic cloth labeling via GPT-4o costs around $0.03 per subject, in contrast to commercial human scans that the paper says often cost around 100 USD per identity.
• Applications demonstrated: extracting garments from real photographs for try-on, re-animating avatars with SMPL motion data via barycentric interpolation of skinning weights, and 3D-printing watertight meshes as physically stable figurines.
• Negative finding on Gen-Schnell detail: Because the pretrained MVDream operates at 256×256, Gen-Schnell cannot generate faithful details such as faces, which is the stated motivation for Gen-HRes.

Methodology in Plain English

The authors avoid collecting real scans by reusing models that already exist. Starting from a small set of existing human scan datasets, they use a captioning protocol (from Trellis) to describe scans, then ask GPT-4o to generate variations of those captions, and finally summarize each caption into ten levels of granularity from 40 words down to 5 words.

To turn those descriptions into images suitable for 3D reconstruction, they fine-tune the text-to-image model FLUX with a LoRA adapter on orthographic, uniformly lit scan renderings, so outputs look like scans instead of strongly perspectival photos. Clothing control comes from reversing try-on: they fine-tune OminiControl on Virtual-TryOn dataset pairs (garment image, try-on image, text prompt) so the model can extract a clean garment from a dressed person. They generate four candidate garments per subject and let GPT-4o pick the best based on color, texture, length, and details like zippers and pockets.

Body shape and pose come from regressing SMPL parameters with NLF at a field of view of 0.1, then refining pose by minimizing the reprojection error between orthographically projected SMPL joints and OpenPose 2D joints, with tuned per-joint weights.

For multi-view generation, they train a diffusion model on orthographic projections. Because orthographic views share horizontal epipoles, rows in one view correspond to rows in other views, allowing efficient row-wise attention. They condition on an input image plus rendered SMPL normal maps encoded by a PSHuman VAE, and generate four body views and four head views, using separate attention within body and head views plus dense pixel-level cross-attention between corresponding body and head views.

The generative stage has two branches. Gen-Schnell couples a 2D multi-view diffusion model (built on MVDream) with a Gaussian splatting decoder; at each sampling step, 2D predictions are replaced by 3D splat renderings so the result stays consistent across views. Gen-HRes instead fine-tunes OminiControl2 as a multi-image-to-image translator that outputs high-resolution full-body images conditioned on text, clothing image, and SMPL; surface normals are then computed with Sapiens2B and SMPL-driven volumetric carving via PSHuman produces the final textured mesh.

Training details: the orthographic multi-view diffusion is built on SD2.1-unclip and trained on 8 H100 GPUs with an effective batch size of 128 for 2 days, using orthographic uniform-lighting renderings of 6000 high-quality human scans from Twindom, CustomHuman, and THuman2.1. Gen-Schnell was trained on 8 A100 GPUs with effective batch size 256 for about 2 days; Gen-HRes on 2 H100 GPUs with effective batch size 32 for 2 days.

Why This Matters

The work argues that relying on captured scans is what has held back controllable 3D human generation, and that foundation models can be distilled into a scalable substitute. By releasing the dataset, models, and generation pipeline, it lowers the barrier to entry for anyone who needs diverse, photorealistic 3D humans without scan budgets. It also sets a comparison point for the field: prior 3D human datasets cap out below InfiniHumanData's 111K identities and none listed provide fine-grained text and clothing annotations.

Real-world applications the paper points to:

• Digital fashion and virtual try-on: extracting garments from ordinary photographs and generating avatars wearing them while preserving identity.
• Gaming and AR/VR: fast avatar creation with user-specified appearance, shape, and clothing.
• Social telepresence: personalized, re-animatable avatars driven by SMPL motion data.
• Physical fabrication: watertight meshes that can be 3D-printed as stand-alone figurines.

Industry relevance centers on cost and speed: roughly $0.03 per subject for annotation, 12 seconds for the fast model, and 4 minutes for the high-resolution model compare favorably against the hours (40 to 384 minutes) required by the optimization-based baselines cited in Table 2.

Future Directions

• Higher-resolution end-to-end generation: The authors could not train a higher-resolution Gen-Schnell due to limited training resources, so they release the 768×768 InfiniHumanData and suggest future work train a high-resolution text-based 3D-GS model that is both fast and high-quality.
• Broader identity coverage: GPT-4o refuses to identify unmatched samples for privacy reasons, which blocks including famous names in InfiniHumanData; the authors propose using a different vision-language model.
• Removing mesh artifacts: Gen-HRes relies on multi-view mesh carving, which produces texture artifacts in self-occluded parts of the avatar; a data-driven mesh reconstruction approach from multi-view images is suggested as a fix.
• Closing the speed-detail trade-off: Gen-HRes remains slower than Gen-Schnell, and Gen-Schnell cannot render faithful facial detail, leaving an open question of how to get both properties in one model.

Target Audience

Researchers and practitioners in 3D generative modeling, digital humans, and computer graphics who need controllable avatar synthesis or large annotated human datasets. It is also relevant to applied teams in digital fashion, virtual try-on, gaming, and AR/VR evaluating whether synthetic data can replace expensive scanning. Given the density of diffusion, SMPL, and multi-view attention concepts, readers without a background in generative 3D vision will find the method sections difficult.

Authors’ abstract

Generating realistic and controllable 3D human avatars is a long-standing challenge, particularly when covering broad attribute ranges such as ethnicity, age, clothing styles, and detailed body shapes. Capturing and annotating large-scale human datasets for training generative models is prohibitively expensive and limited in scale and diversity. The central question we address in this paper is: Can existing foundation models be distilled to generate theoretically unbounded, richly annotated 3D human data? We introduce InfiniHuman, a framework that synergistically distills these models to produce richly annotated human data at minimal cost and with theoretically unlimited scalability. We propose InfiniHumanData, a fully automatic pipeline that leverages vision-language and image generation models to create a large-scale multi-modal dataset. User study shows our automatically generated identities are undistinguishable from scan renderings. InfiniHumanData contains 111K identities spanning unprecedented diversity. Each identity is annotated with multi-granularity text descriptions, multi-view RGB images, detailed clothing images, and SMPL body-shape parameters. Building on this dataset, we propose InfiniHumanGen, a diffusion-based generative pipeline conditioned on text, body shape, and clothing assets. InfiniHumanGen enables fast, realistic, and precisely controllable avatar generation. Extensive experiments demonstrate significant improvements over state-of-the-art methods in visual quality, generation speed, and controllability. Our approach enables high-quality avatar generation with fine-grained control at effectively unbounded scale through a practical and affordable solution. We will publicly release the automatic data generation pipeline, the comprehensive InfiniHumanData dataset, and the InfiniHumanGen models at https://yuxuan-xue.com/infini-human.