SOSControl: Enhancing Human Motion Generation through Saliency-Aware Symbolic Orientation and Timing Control

AI summary

Overview

Research area: Text-conditioned human motion generation, specifically controllable motion synthesis using symbolic control signals and diffusion models.

Technical level: Advanced. The paper assumes familiarity with diffusion models, ControlNet adaptation, autoencoders, SMPL body representation, and Labanotation-style symbolic motion description.

Scope: The paper proposes a symbolic scripting notation (SOS) for body-part orientations and motion timing, an automatic pipeline to extract such scripts from motion data, and a ControlNet-based diffusion framework (SOSControl) that generates motions conforming to those scripts.

What This Paper Is About

Text-to-motion systems give users only vague and ambiguous control over how a body moves. Existing alternatives based on joint keyframe locations specify where a joint should be but not how a limb should be oriented, and they can misinterpret an end location as an intermediate waypoint, producing incorrect poses or disrupted timing. The paper's goal is to give animators a sparse, interpretable, programmable symbol script that specifies body-part orientations at salient keyframes, and a generator that reliably respects that script while producing smooth, natural motion.

Key Contributions

1. The Salient Orientation Symbolic (SOS) script — a programmable symbolic framework with a staff-based interface, inspired by Labanotation, that represents body-part orientations and motion timing using orientation symbols annotated at keyframes. The staff has six columns: Root (RT), Left Arm (LA), Left Leg (LL), Right Leg (RL), Right Arm (RA), and Spine (SP), with 8 root direction symbols and 26 body-part orientation symbols (shape denotes direction, shading denotes level).

2. An automatic SOS extraction pipeline that uses temporally-constrained agglomerative clustering to detect frame saliency, then applies a Saliency-based Masking Scheme (SMS) to adaptively filter orientation features below a user-defined saliency threshold, yielding sparse, human-interpretable scripts.

3. The SOSControl framework, which treats orientation symbols in a sparse SOS script as salient constraints and injects them into motion generation via SMS-based data augmentation, gradient-based iterative optimization at diffusion time and test time, and a ControlNet-based ACTOR-PAE Decoder for smooth, natural outputs.

4. Claimed novelty: the first approach to use saliency information from agglomerative clustering for enhanced motion control.

Main Findings

• Control-signal alignment: Under the main SOS-conditioned setting (SOS extracted with a saliency threshold of 0.9), SOSControl achieves an SOS-Acc of 0.988, above TLControl (0.982) and OmniControl (0.873), and far above MDM (0.151), GMD 1-stage (0.113), and GMD 2-stage (0.120).

• Motion fidelity to source motion: SOSControl records L2-Rot6D = 0.325, tied with OmniControl's 0.325 and better than TLControl's 0.341, MDM's 0.351, GMD 1-stage's 0.427, and GMD 2-stage's 0.402.

• Unconditional baseline wins on realism metrics: MDM obtains the best FID (2.592) and MMD (6.001) because it generates without adapting to SOS. Among SOS-conditioned methods, SOSControl reports FID 3.892 and MMD 6.199, versus TLControl (11.132 / 7.066) and OmniControl (3.975 / 6.095).

• SMS data processing matters: Removing SMS processing and replacing it with random masking at a ratio of 0.8 degrades results for every method. For SOSControl, SOS-Acc stays high (0.991) but L2-Rot6D worsens to 0.499, FID to 13.494, and MMD to 6.893. OmniControl collapses to SOS-Acc 0.225, L2-Rot6D 0.505, FID 60.966, MMD 8.483.

• Test-time optimization drives alignment; diffusion-time optimization contributes little: In the ablation, diffusion-time optimization gave SOSControl only 0.531 → 0.535 SOS-Acc, while test-time optimization raised it to 0.988. The paper attributes the weakness of diffusion-time optimization to its adjustments being overwritten during diffusion inference.

• Combining both optimizations: OmniControl performs best with both optimizations (SOS-Acc 0.956, L2-Rot6D 0.323, FID 3.025, MMD 5.988); SOSControl reports 0.988 / 0.325 / 3.892 / 6.199. The paper notes that OmniControl's raw-motion test-time optimization affects only specific keyframes without propagating to adjacent frames, producing inconsistency (illustrated at frame 14).

• Sparsity is user-tunable: Setting the saliency threshold to 0.7 of the maximum saliency selects three LeftArm keyframes in the paper's example. The highest LeftArm saliency in the figure occurs at frame 18, corresponding to a swing peak. Users can adjust thresholds at test time to control script sparsity.

• Timing synchronization: In the qualitative example, the SOS specifies that the Right Leg swings back and peaks at frame 29; OmniControl and the no-SMS variant fail to place the peak at the designated frame, treating the symbol as an intermediate waypoint.

• Additional evaluations exist but are not reported in the provided content: The paper states that module ablations, saliency-threshold analyses, experiments on the BABEL dataset, and user studies on script interpretability and motion quality/control alignment are in the supplementary material. Those results are not included in the available text, and the running-time section is truncated.

Methodology in Plain English

The researchers first build a symbolic notation. From motion data, they extract per-frame orientation features for six body parts as 3D vectors. The Root feature is the horizontal facing direction; the other five are computed with the Pairwise Relative Position Phrase (PRPP) between an end joint (e.g., left wrist) and an anchor joint (e.g., left shoulder) using egocentric reference vectors. These features are represented as o ∈ R^(T×6×3).

To turn continuous features into symbols, they compute a softmax over dot products between the normalized orientation vector and 26 unit-norm direction vectors u ∈ R^(26×3). Softmax keeps the operation differentiable so it can be used inside gradient-based optimization, while argmax is used for symbol recognition and visualization.

To find which frames matter, they apply temporally-constrained agglomerative clustering (scikit-learn, with a grid connectivity matrix from grid_to_graph(n_x=T, n_y=1, n_z=1)) to the frame-level central finite difference of the normalized dot products, so segments only merge with adjacent temporal neighbors. Each merge distance becomes the saliency value for the first frame of the later segment. The Saliency-based Masking Scheme then removes symbols whose saliency falls below a chosen threshold.

Generation is a two-stage diffusion framework. Stage one denoises a periodic latent p produced by ACTOR-PAE (ACTOR's autoencoder plus PAE's periodic parameterization: p = a·sin(f·(N − s)) + b), conditioned on text. Stage two decodes the latent back to motion. ControlNet adaptation is applied to both the Motion Latent Diffusion Model (D⁻ → D⁺) and the ACTOR-PAE decoder (P_D → P_D⁺), mirroring ControlNet and OmniControl. During training, SMS-based mask sampling draws a saliency percentile uniformly from U(0,1) for each of the six body parts, exposing the model to scripts of varying granularity. At inference, gradient descent iteratively refines the latent by minimizing the difference between the masked quantized orientation features of the output motion and the input SOS script, using weights w_d = 300 and w_t = 3000. The ACTOR-PAE decoder's ability to propagate sparse guidance to neighboring frames is credited with preserving motion consistency.

Why This Matters

Impact on research. The paper connects three threads that usually sit apart: symbolic/dance-notation-style motion description, unsupervised saliency detection via agglomerative clustering, and controllable diffusion-based motion synthesis. If the reported results hold, it shows that saliency-aware masking is a viable substitute for the fixed-threshold or random-masking strategies used in prior controlled-generation work, and it supplies a benchmark protocol (SOS-Acc and L2-Rot6D alongside FID and MMD) for orientation-and-timing control. The authors released code at https://github.com/asdryau/SOSControl.

Real-world applications.

• Animation and game production, where animators need quick, precise pose and timing specification without hand-editing keyframes across camera views.
• Robotics and humanoid motion programming, where directional intent and timing at key moments matter more than exact joint coordinates.
• Media content creation and virtual production, where text prompts alone are too vague to match a storyboard.
• Interactive human-AI collaboration tools, where users drag and drop symbols on a staff and refine generation interactively.

Industry relevance. The paper explicitly frames manual keyframe placement with 3D joint locations as time-consuming and impractical for industrial animation pipelines, citing repeated camera-view switching and the need to reason about movement speed and physical balance. A sparse, drag-and-drop symbolic interface with tunable sparsity directly targets that friction, and the reported near-real-time practicality is supported by training that completes within one day on two Nvidia 4090 GPUs.

Future Directions

• Pushing script symbol counts lower. The SMS augmentation samples thresholds uniformly from U(0,1) per body part; how performance degrades at extremely sparse scripts, and whether smarter mask sampling helps, remains open.
• Extending beyond orientation and timing. The notation currently covers root facing direction and five body-part orientation vectors; adding foot contact, global position, or interaction with objects is a natural extension.
• Improving the token-based alternative. The paper notes TLControl's VQ-VAE decoder limits expressiveness through codebook size, suggesting decoder design is a live axis for improving controllable generation.
• Broadening evaluation. The BABEL dataset experiments and user studies are referenced but not detailed in the provided content, and no quantitative statement of inference runtime is available in the truncated text; both merit attention before judging generalization.

Target Audience

Researchers and graduate students working on controllable human motion generation, diffusion-based generative models, and multimodal conditioning; computer animators and technical artists interested in symbolic motion authoring; and robotics or human-AI interaction practitioners who need expressive, low-effort ways to specify body orientation and timing. Readers without a background in diffusion models or SMPL-based motion representation will find the technical sections demanding.

Authors’ abstract

Traditional text-to-motion frameworks often lack precise control, and existing approaches based on joint keyframe locations provide only positional guidance, making it challenging and unintuitive to specify body part orientations and motion timing. To address these limitations, we introduce the Salient Orientation Symbolic (SOS) script, a programmable symbolic framework for specifying body part orientations and motion timing at keyframes. We further propose an automatic SOS extraction pipeline that employs temporally-constrained agglomerative clustering for frame saliency detection and a Saliency-based Masking Scheme (SMS) to generate sparse, interpretable SOS scripts directly from motion data. Moreover, we present the SOSControl framework, which treats the available orientation symbols in the sparse SOS script as salient and prioritizes satisfying these constraints during motion generation. By incorporating SMS-based data augmentation and gradient-based iterative optimization, the framework enhances alignment with user-specified constraints. Additionally, it employs a ControlNet-based ACTOR-PAE Decoder to ensure smooth and natural motion outputs. Extensive experiments demonstrate that the SOS extraction pipeline generates human-interpretable scripts with symbolic annotations at salient keyframes, while the SOSControl framework outperforms existing baselines in motion quality, controllability, and generalizability with respect to motion timing and body part orientation control.