Instruction-Driven 3D Facial Expression Generation and Transition

AI summary

Overview

Research area: Computer vision and graphics — text-conditioned 3D facial expression generation, expression-to-expression transition, and neural face rendering.

Technical level: Advanced. The paper assumes familiarity with conditional variational autoencoders, cross-attention transformers, CLIP text embeddings, and parametric 3D face models (FLAME/DECA).

Scope: The paper proposes a framework that takes a single face photograph plus a text instruction such as "Turn this face from disgust to happiness" and produces a 3D facial expression trajectory with smooth transitions between the two named expressions, rendered as a sequence of facial appearances.

What This Paper Is About

A 3D avatar usually has only a fixed set of cardinal facial expressions, and most existing text-driven methods generate one specific expression (for example, "the person is happy") rather than the motion between two expressions. This paper builds a model that reads a natural-language instruction naming both a starting and an ending expression, generates the corresponding 3D expression and head-pose coefficients, interpolates smoothly between them, and renders the result onto the appearance of the person in the source photograph. The researchers had to first add text instructions to two existing facial image datasets (CK+ and CelebV-HQ) because no dataset paired facial expressions with transition instructions.

Key Contributions

1. An instruction-driven 3D facial expression generation and transition framework that starts from a photograph of a face and follows a text instruction describing a transition between two expressions.
2. The Instruction-Driven Facial Expression Decomposer (IFED), a dual-branch module designed to learn multimodal data and capture the correlation between facial expression features and textual descriptions.
3. The Instruction to Facial Expression Transition (I2FET) method, a conditional variational autoencoder that integrates IFED with a vertex reconstruction loss to refine the semantic comprehension of latent vectors.
4. Evaluation on the CK+ and CelebV-HQ datasets in which the proposed model outperforms the MotionClip baseline used for comparison.

Main Findings

• Outperforms the baseline on CK+: On CK+ the proposed method reached Acc1 of 91.44 ± 0.0024 percent, Acc2 of 84.03 ± 0.0034 percent, and G-mean of 80.30 ± 0.0064 percent, compared with MotionClip at 52.00 ± 0.0058, 20.00 ± 0.0082, and 40.48 ± 0.0060 percent respectively. Ground-truth values for reference were 99.69 ± 0.0001, 99.39 ± 0.0002, and 99.69 ± 9.3e-5 percent.
• Outperforms the baseline on CelebV-HQ: On CelebV-HQ the proposed method reached Acc1 of 58.24 ± 0.0028 percent, Acc2 of 33.45 ± 0.0052 percent, and G-mean of 46.47 ± 0.0065 percent, versus MotionClip at 40.00 ± 0.0039, 13.73 ± 0.0056, and 34.42 ± 0.0069 percent. Ground-truth was 93.67 ± 0.0000, 87.89 ± 0.0000, and 95.04 ± 1.1e-16 percent.
• Rendering quality depends heavily on the renderer: Combined with CVTHead, the method achieved L1 of 0.005, PSNR of 33.74, LPIPS of 0.021, and MS_SSIM of 0.978 on CK+, and L1 of 0.003, PSNR of 37.08, LPIPS of 0.010, and MS_SSIM of 0.990 on CelebV-HQ. Combined with ROME the same metrics were markedly worse (CK+: L1 0.199, PSNR 9.939, LPIPS 0.490, MS_SSIM 0.347; CelebV-HQ: L1 0.168, PSNR 11.316, LPIPS 0.505, MS_SSIM 0.418).
• I2FET with IFED clusters expressions more cleanly: A t-SNE visualization of the learned latent space for seven facial expressions is reported for MotionClip, for I2FET without IFED, and for I2FET with IFED, indicating that the IFED module contributes to separating expression classes in latent space.
• Rotation robustness matters for evaluation: Because the datasets are imbalanced and contain non-frontal heads, the authors tested ResNet-101, MEK, and ResNet-101 with Reweighted Focal Loss (RFL) as the evaluation classifier and selected ResNet-101 with RFL, reporting that it handles both imbalance and rotational transformations better than the alternatives. The supporting ablation numbers are referenced to Section IV-E but are not included in the content available here.
• Not reported: No user study, inference-time measurement, or comparison against a NeRF-based text-driven expression method is reported in the available content.

Methodology in Plain English

The framework has two large parts: a Facial Expression Transition (FET) module that decides what the face should do, and a Face Rendering (FR) module that decides how it looks.

Understanding the text. A frozen CLIP encoder turns the instruction into a 77 × 768 text vector. Separately, a small random latent vector is drawn from a standard normal distribution to represent pose and expression.

Fusing text and geometry (IFED). A dual-branch vision transformer processes the facial parameter vector (the 50-dimensional expression plus the 6-dimensional head pose, reduced to a 53-dimensional input using only the jaw pose) and the text vector in parallel, then uses cross-attention (the CAFT component) so each branch can see the other. The two outputs are layer-normalized, concatenated into a fused vector, and split by two linear projections into a 50-dimensional expression conditioning vector and a 6-dimensional pose conditioning vector.

Generating the expressions (I2FET). A conditional variational autoencoder, built from three multi-layer perceptrons per stream, encodes expression and pose into mean and standard deviation parameters. Latent samples are reparameterized and passed back through IFED together with the text to produce refined conditioning vectors, which the expression and pose decoders use to predict the start and end expression and pose coefficients. Training combines mean-squared error on expression, pose, and vertices with a KL divergence term to keep the predicted distributions close to the ground truth; the vertex term is computed by feeding both predicted and ground-truth parameters through the FLAME head model, which has 5,023 base vertices and shape and expression blendshape bases.

Smoothing and rendering. The source image is encoded with a pre-trained DECA model into shape, expression, pose, and camera parameters. The predicted start and end expressions become anchor points, and a linear interpolation function with a coefficient between 0 and 1 fills in the frames in between so the motion looks smooth and temporally consistent. Those trajectories drive FLAME to produce a mesh sequence, a pre-trained linear deformation model adds hair and shoulder regions, and a pre-trained rendering model (ROME or CVTHead) paints the appearance using texture extracted from the source photograph.

Training setup. I2FET was trained for 200 epochs with a batch size of 128 and a learning rate of 8e-4 using the Adam optimizer. Ten percent of each dataset was held out for testing, another ten percent of the training split served as validation, and every experiment was repeated ten times with average results reported. Because the task is new, MotionClip — originally designed for human motion synthesis — was chosen as the baseline.

Why This Matters

The paper shifts the text-to-face problem from producing a labeled emotion to directing a movement between emotions, which is much closer to how animators and users actually think about performance. It also shows that the geometry of a face can be controlled by language while the appearance comes from a single photo, keeping the same person's identity throughout.

Real-world applications:

• Avatar and social VR: Driving a personalized 3D avatar through typed or spoken emotion descriptions instead of a fixed emoticon menu.
• Game and film production: Rapidly prototyping facial performance beats without hand-keying every expression or capturing a performer.
• Human-computer interfaces and telepresence: Rendering a remote participant's expressions from a photo and a text or speech-derived instruction stream, useful when video bandwidth or camera quality is poor.
• Clinical and therapeutic tools: Practicing or demonstrating emotional expression transitions for people working on facial affect recognition or expression training.

Industry relevance: The framework is designed to plug into existing rendering back-ends — the authors demonstrate it with ROME and CVTHead, and note that FLAME parameters are a common interchange format. That modularity makes the expression-transition module adoptable by studios or platforms that already have a face renderer.

Future Directions

• Handle more than two anchor expressions. The current formulation takes a start and an end expression; extending it to multi-step instructions (for example, neutral to anger to happiness) is a natural next step.
• Replace linear interpolation with a learned transition model. The smoothness between anchors currently comes from a simple linear blend with a coefficient between 0 and 1; a learned dynamics model could produce more believable timing and micro-expressions.
• Reduce dependence on imperfect evaluation classifiers. The authors selected ResNet-101 with RFL because MEK was sensitive to rotation and plain ResNet-101 struggled with imbalance, which suggests the reported accuracies are bounded by the classifier rather than by the generator.
• Improve the rendering back-end. ROME produced far weaker L1, PSNR, LPIPS, and MS_SSIM scores than CVTHead, so closing that gap or finding renderers that handle occlusion, hair, and shoulders better remains open.
• Broaden language diversity. Instructions were generated from five templates with fixed expression names; testing free-form, more varied phrasing would show whether the IFED module generalizes beyond these templates.

Target Audience

Graduate students and researchers in computer vision, computer graphics, and multimodal generative modeling who work on 3D face reconstruction, text-conditioned animation, or avatar systems. It is also useful for technical practitioners in games, VFX, and virtual-reality product teams who need instruction-controlled facial performance and are comfortable reading conditional VAE and transformer architectures. Readers without background in FLAME, DECA, or parametric face models will find the method section demanding.

Authors’ abstract

A 3D avatar typically has one of six cardinal facial expressions. To simulate realistic emotional variation, we should be able to render a facial transition between two arbitrary expressions. This study presents a new framework for instruction-driven facial expression generation that produces a 3D face and, starting from an image of the face, transforms the facial expression from one designated facial expression to another. The Instruction-driven Facial Expression Decomposer (IFED) module is introduced to facilitate multimodal data learning and capture the correlation between textual descriptions and facial expression features. Subsequently, we propose the Instruction to Facial Expression Transition (I2FET) method, which leverages IFED and a vertex reconstruction loss function to refine the semantic comprehension of latent vectors, thus generating a facial expression sequence according to the given instruction. Lastly, we present the Facial Expression Transition model to generate smooth transitions between facial expressions. Extensive evaluation suggests that the proposed model outperforms state-of-the-art methods on the CK+ and CelebV-HQ datasets. The results show that our framework can generate facial expression trajectories according to text instruction. Considering that text prompts allow us to make diverse descriptions of human emotional states, the repertoire of facial expressions and the transitions between them can be expanded greatly. We expect our framework to find various practical applications More information about our project can be found at https://vohoanganh.github.io/tg3dfet/