COSMI: COmpositional Synthesis of Multi-object Interactions

AI summary

Overview

Research area: Computer vision and 3D human motion generation — specifically, human-object interaction (HOI) synthesis, dataset construction by composition, and text-conditioned diffusion models.

Technical level: Advanced. The paper assumes familiarity with SMPL-X body models, diffusion transformers, classifier-free guidance, and standard text-to-motion evaluation metrics.

Scope: The paper introduces COSMI, a pipeline that composes multi-object interaction data from existing single-object motion captures, plus a diffusion transformer trained on that data to generate text-conditioned interactions with one to five objects.

What This Paper Is About

Generative models of human-object interaction can only produce what the data contains, and almost all captured datasets record a person interacting with one object at a time, because capturing several objects at once is combinatorially expensive. The authors observe that interactions are local — a cup is held by a hand, a chair supports the pelvis, a ball is kicked by a foot — so a contact-consistent clip of a single interaction already contains the limb-and-object pair needed for a multi-object activity. Their goal is to compose such single-object clips into a large, plausible multi-object dataset, and then train a single generator that handles a variable number of objects and grounds each object to the body part that moves it.

Key Contributions

1. Composition pipeline and the COSMI dataset. Multi-object interactions are composed from single-object captures, with left and right hands balanced by mirroring. The dataset holds 222k sequences and 275 hours with up to five objects, and is extensible by adding a rule rather than a recording session.

2. The COSMI method. A text-conditioned diffusion transformer for a variable number of objects, using weight-shared object slots, joint-relative prediction, a part-assignment head, auxiliary losses, and contact-consistent guidance at sampling time.

3. A generalization benchmark. Built on the COSMI dataset, it tests models on an unseen object (the black chair) and on five unseen interaction combinations, such as walking with an apple.

4. Demonstrated extensibility. The paper shows that hand-object recordings (HOT3D) enter the pipeline as source clips once a body is fitted to the headset and the MANO hands.

Main Findings

• Dataset scale. The COSMI dataset contains 221,677 sequences and 275 hours of motion from 59 subjects and 36 objects, seven of them articulated, with up to five objects per sequence and left/right hands balanced at 49/51. The abstract describes this as 222k sequences and 275 hours, nearly thirty times the largest multi-object capture — HIMO at 9.44 hours.

• Composition preserves capture quality. Foot sliding is unchanged because composition alters neither the lower body nor the global trajectory. Jerk and penetration change by less than the captured datasets differ among themselves, with human jerk changing the most, by up to half.

• Best text alignment and contact on the all-objects tier. On the all-objects tier (one to five objects, articulated and worn included), COSMI reaches FID 0.049, top-1 R-precision 0.696, top-3 0.900, and MM-Dist 0.827, against MDM at FID 0.148 and PriorMDM at FID 0.077. COSMI has the lowest foot sliding (1.395 against ground-truth data at 1.879), penetration at the ground-truth level (33.96% against data at 34.60%), and the lowest hinge-angle error (0.437) but the smallest hinge sweep, two thirds of the ground truth.

• Best contact on two- and three-object tiers. With two objects, COSMI achieves FID 0.066, T-Acc 0.993, B-Acc 0.962, and penetration 11.94% at the ground-truth level; HIMO reaches FID 0.095, T-Acc 0.818, B-Acc 0.885, and 28.68% penetration. With three objects, COSMI reaches FID 0.157 and T-Acc 0.968; HIMO has the best top-3 R-precision in this tier at 0.516, against COSMI at 0.507.

• Generalization to unseen combinations. On the five held-out combinations, all methods match or exceed their accuracy on the whole test set, with one drop of three points, suggesting they learn the base motion and the hand interaction independently.

• The unseen object is the harder test. The baselines lose four to eight points of temporal contact accuracy on the black chair, whereas COSMI stays within one point of its value on the whole test set, achieving T-Acc 0.975 and B-Acc 0.960.

• Strongest when re-trained on captured data. Re-trained on the HIMO dataset and evaluated with HIMO's evaluator, COSMI has the highest top-3 R-precision in both tiers and, with three objects, also the best FID.

• Ablation results. Auxiliary losses matter most. Part assignment improves both text alignment and contact accuracy. Text tokens and object names improve text alignment. Contact-consistent guidance significantly decreases foot sliding and hand-object gaps without damaging other metrics.

• Efficiency. COSMI has 30.4M parameters, between MDM (20.7M) and PriorMDM (39.6M), and trains in eight hours on one RTX 5090. A window generates in 0.16 s with classifier-free guidance and 1.1 s with contact-consistent guidance.

Methodology in Plain English

The pipeline starts from existing captured single-object datasets (GRAB, BEHAVE, InterCap, OMOMO, ARCTIC, plus CMU walking sequences from AMASS selected with BABEL labels), unified onto a gender-neutral SMPL-X body at 30 fps.

Clips are cut by interaction rules — thirteen of them — that specify which body parts must and must not touch the object, pose constraints such as an upright torso, and the local reference joints relative to which the object is expressed. Contact is defined as proximity below 5 cm to the object surface, and contiguous windows of at least 3 s become atomic clips; there are 1,491 in total.

Because global trajectories do not transfer between bodies, each object is expressed relative to the joint performing its interaction (wrist, foot, head, spine). Inverting that transform with the joints of any other body carries the object over.

A language model, Qwen3, labels each clip: it names the sustained interaction with a verb, writes a sentence, and marks whether the action requires gaze or involves the head — which decides what can happen at once. Every one-handed and one-footed clip is then mirrored to balance the sides (2,663 clips). Reflecting the body is trivial, but a reflected mesh is a different object, so the pipeline keeps the original template and fits a proper rotation and translation that pose it like its reflection.

Composition pairs a target clip (whole-body motion: sitting, moving large objects, foot interactions, walking) with a source clip whose action is added to a free limb. Some pairs are composed simultaneously; two actions that both require gaze or both involve the head are sequenced instead, with a blended transition. Compositions are kept only if physically plausible: every object keeps its rule contacts, the body does not penetrate itself, and no object penetrates another. Composed sequences can serve as targets again, yielding up to five objects per sequence. Captions are assembled from annotations and paraphrased.

The generator, COSMI, is a single transformer that denoises one human token per frame and one object token per four frames over a 120-frame window (4 s at 30 fps). Object slots share input and output projections and differ only in their conditioning, so no parameter depends on the object count and the model is permutation-equivariant. It is conditioned on CLIP text embeddings (both pooled and per-word-piece), the initial poses, each object's name, and a basis-point-set geometry embedding. A lightweight head predicts which joint drives each object from the conditions alone; driven objects are decoded through their joint-relative pose, undriven ones through their absolute pose. Training uses DDPM with a cosine schedule of 1000 steps, min-SNR weighting, and losses over parameters, frame-to-frame differences, contact scores, foot-skating, forward kinematics, and a consistency term relating absolute and relative poses. Sampling uses 100 respaced steps at classifier-free guidance scale 2.5, with gradient steps near the end to enforce contact bounds.

Why This Matters

The paper argues that the combinations to capture grow faster than any recording effort, and that composed data closes the gap the way synthetic and composed data has done before. Instead of recording multi-object activity directly, the dataset grows combinatorially with the clips rather than linearly with recording time, and every further capture multiplies the combinations — full-body datasets through a rule, hand-object recordings once a body is fitted to them.

Real-world applications the paper points to:

• Animation and augmented reality, including composite animation such as sitting down while carrying an object, and clothed-scan rendering as demonstrated in the paper's figures.
• Embodied AI and robotics, where humanoid controllers and robot policies learn from human-object interaction motion.
• Egocentric assistants, which benefit from generated multi-object interaction.
• Avatar driving, since the composed sequences are shown to animate clothed scans registered as SMPL-X+D and would drive any SMPL-X-anchored avatar the same way.

Industry relevance. Any pipeline that needs large volumes of 3D interaction data for training or for character animation can use the composition approach without new motion capture. The released code, models, and dataset pipeline are stated to be forthcoming. The method's single-stage generation for a variable number of objects is contrasted with the concurrent SK-HOI, which generates in three separately trained stages followed by roughly four minutes of test-time optimization per sequence — a practical difference at the scale of a 35k-sequence test set.

Future Directions

• Multi-stage activities. The composition currently covers sustained single interactions, performed at once or in sequence. Extending it to activities such as cooking or assembling, whose steps and objects depend on each other, is named as a promising direction.

• Interactable scenes. Bringing the composed data together with scenes — where a cup is taken from a shelf on the way — is the second named direction.

• Finer finger motion. The contact-consistent guidance closes gaps between fingers and objects, but the finger motion has yet to reach the precision of dedicated grasp synthesis.

• Text as a richer interface. The prompt fixes the objects and actions but not the path of the person or the timing of sequential actions, leaving control over trajectory and temporal ordering open.

Target Audience

Researchers and engineers working on 3D human motion generation, human-object interaction datasets, and diffusion-based motion synthesis will benefit most, particularly those facing data scarcity for multi-object activity. The dataset-composition half of the paper is readable by dataset builders and animation researchers who are not diffusion specialists, while the method and evaluation sections assume comfort with SMPL-X, transformer architectures, and text-to-motion metrics (FID, R-precision, MM-Dist, diversity). Practitioners in character animation, AR/VR, and robot learning from human motion are the likely downstream users.

Authors’ abstract

Generative models of human-object interaction are bounded by the data that exists: everyday activities involve several objects, but most captured datasets record one at a time, as multi-object capture is combinatorially expensive. Our observation is that interactions are local, so single-object captures already contain the parts of multi-object activities. We compose them: contact-consistent clips of single interactions, mirrored to balance the hands, transfer between bodies, and a language model and geometric checks admit only the pairings that are plausible, semantically and physically. Therefore, the dataset grows combinatorially with the clips rather than recording time. The COSMI dataset holds 222k sequences and 275 hours with up to five objects, nearly thirty times the largest multi-object capture, and can be extended by adding datasets or even hand-object recordings. On this data we train the COSMI method, a text-to-interaction diffusion transformer that follows how the data is built: weight-shared object slots generate a variable number of objects, predicted relative to the body parts that move them. On a benchmark with an unseen object and unseen interaction combinations, models trained on the dataset generalize to the unseen combinations. COSMI outperforms baselines in text alignment and contact accuracy, where its margin is largest on the unseen object. Code, models, and the dataset pipeline will be released on the project page: https://ptrvilya.github.io/cosmi.