Mind-to-Face: Neural-Driven Photorealistic Avatar Synthesis via EEG Decoding

AI summary

Overview

Research area: Neural decoding and brain–computer interfaces (BCI) combined with photorealistic 3D avatar synthesis (computer vision, affective computing, 3D Gaussian Splatting).

Technical level: Advanced. The paper assumes familiarity with EEG signal processing, transformer encoders, variational autoencoders, 3D morphable models, photogrammetry, and 3D Gaussian Splatting.

Scope: The paper presents a framework that maps non-invasive 16-channel EEG signals directly to dense 3D facial geometry and renders photorealistic, view-consistent avatars of two subjects using a modified 3D Gaussian Splatting pipeline.

Affiliation: University of Southern California and the USC Institute for Creative Technologies. Project page: https://tianwen-fu.github.io/mind_to_face. License: CC BY 4.0.

What This Paper Is About

Existing expressive avatar and emotion systems depend on visual cues such as cameras, motion tracking, or speech. These approaches break down when the face is hidden behind a head-mounted display (HMD) or when an emotion is felt internally but not expressed outwardly. The goal of Mind-to-Face is to bypass the face entirely: decode brain activity measured by EEG into continuous, subject-specific 3D facial expressions and render them as photorealistic avatars.

Key Contributions

1. A first-of-its-kind neural-to-visual framework. The authors introduce Mind-to-Face, which they describe as the first system to decode non-invasive EEG signals into dense 3D facial geometry and render photorealistic avatars, establishing a continuous neural-to-visual mapping rather than coarse emotion labels.

2. A custom dual-modality capture system and dataset. They design a rig that synchronously records 16-channel EEG and high-speed multi-view facial video while subjects watch emotion-eliciting film clips. The authors state they will publicly release the resulting synchronized EEG-to-3D dataset and describe it as the first of its kind.

3. A dense position-map representation in place of FLAME blendshapes. Instead of predicting a small set of blendshape coefficients, the model predicts a 256×256×3 position map (roughly 65k vertices) that preserves mid-frequency geometric detail such as wrinkles.

4. A modified GaussianAvatars rendering pipeline. The deformation and skinning system of GaussianAvatars is adapted to support the position-map representation, combined with a robust facial tracking pipeline that handles headcap-induced occlusion and extreme expressions.

Main Findings

• EEG alone can drive subject-specific expressions. The framework produces dynamic facial expressions from EEG, including subtle emotional responses, with subject-specific dynamics such as differing smile intensity. The authors state this indicates effective personalization rather than broad categorization.

• Position maps outperform blendshapes in geometric accuracy. Table 1 reports point-to-surface distance within the masked facial region, in millimeters, plus the percentage of vertices below 1 mm and 3 mm. Across all trials except Neutral, the position map–driven avatar achieves lower geometric error than the blendshape-driven avatar. For example, for Subject 1 on Angry, the position map gives a mean of 0.4982 mm with 87.37% of vertices below 1 mm and 99.67% below 3 mm, versus 1.0792 mm, 60.76%, and 93.94% for blendshapes. For Subject 2 on Angry, the position map gives 0.5534 mm, 84.28%, and 99.64% versus 1.6492 mm, 42.79%, and 83.92%.

• The exception is Neutral. For Subject 1's Neutral trial, the blendshape-driven avatar reports a lower mean error (0.3270 mm, 94.99% below 1 mm, 99.95% below 3 mm) than the position map–driven avatar (0.4028 mm, 92.85% below 1 mm, 100.0% below 3 mm). For Subject 2's Neutral trial, the position map is better (0.3245 mm, 95.84%, 99.92%) than blendshapes (1.3065 mm, 52.91%, 89.98%).

• Cross-trial generalization was tested with a held-out sequence. The EMOSTIM mixed-emotion sequence was excluded from training and used as a leave-one-trial-out test. Position-map results for EMOSTIM were 0.5158 mm / 86.76% / 99.16% for Subject 1 and 0.7077 mm / 78.88% / 97.55% for Subject 2; blendshape results were 0.6838 mm / 78.78% / 97.78% and 1.5078 mm / 52.22% / 86.28%.

• The standard GaussianAvatars tracking pipeline does not transfer to this capture setup. Because subjects wear a headcap with sensors and because the emotion-eliciting videos trigger extreme expressions beyond the expressivity of FLAME parameters, the authors report that landmark decoding in the GaussianAvatars pipeline can produce incorrect expression tracking or completely broken geometry. Their pipeline remains robust in these conditions.

• Metrics are otherwise limited to qualitative results and ablations. The authors state there is no direct baseline to compare against because this is a pioneering work; qualitative results for two subjects are shown, and the paper notes that quantitative metrics can be found in the supplementary materials.

Methodology in Plain English

Capture. Sixteen cinema-grade global-shutter cameras, capable of up to 8K resolution at 120 fps, are arranged evenly around the subject at 15-degree intervals. A screen in front of the subject plays emotion-eliciting clips. EEG is recorded with the OpenBCI Cyton-Daisy kit at 16 channels and 125 Hz, with electrodes placed according to the international 10–20 system, and conductive gel applied to keep impedance below 500 kΩ. Cameras and EEG are aligned frame-accurately using a linear signal generated by an Arduino Teensy board, with all modalities temporally aligned via Linear Timecode.

Procedure. Stimuli were selected from the Open Library for Affective Videos (OpenLAV) and emotion-eliciting clips curated by Coan and Allen, grouped into five emotion categories: neutral, disgust, funny, angry, and sad. Each category forms one trial. A separate mixed-emotion video compiled from the EMOSTIM dataset is held out as a leave-one-trial-out test sequence. Each trial is separated by a 30-second rest period to prevent emotional carryover. Each subject completes five stimulus trials, producing roughly 25 minutes of total recording time; about 30 minutes of video per subject is captured in compressed RAW from 16 cameras, amounting to roughly 7 TB of data per subject.

Signal processing. EEG is filtered with a 6th-order Butterworth band-pass filter with a 4–40 Hz passband, then z-scored per channel using training-set statistics. Signals are cut into overlapping windows of size W = 375 samples across C = 16 channels.

Geometry ground truth. For each video frame, a 3D facial mesh is reconstructed. Identity is solved on a neutral frame using the 3D Morphable Model from ICT-FaceModel, then the identity template is fit to multi-view photogrammetry observations while maintaining topology through Laplacian deformation, with optical-flow-based refinement for temporal stability. Rigid Procrustes alignment removes head pose so the decoder only has to learn non-rigid expression deformation. Each aligned mesh becomes a 256×256×3 position map where every pixel stores the (x, y, z) coordinates of the facial surface in UV space.

Decoding. Each EEG window is treated as a 1×16×375 spatiotemporal image and passed through a convolutional-transformer encoder inspired by EEG-Conformer. A temporal convolution with 40 output channels and kernel size 40 is followed by a spatial convolution with kernel size 16 spanning all electrodes; batch normalization, ELU, and average pooling reduce the temporal dimension from 375 to about 19 patches, and a 1×1 convolution projects to an embedding dimension of 40, giving a token sequence of length N = 19. Six transformer layers with pre-normalization, 10 attention heads, feed-forward expansion factor 4, GELU, and dropout p = 0.5 produce a latent embedding. A randomly initialized latent VAE decoder projects the flattened embedding into a B×4×8×8 latent feature map and then, via fully connected layers and two transposed convolution layers, into the final 256×256×3 position map. A binary mask restricts supervision to the inner-face region, and a self-supervised Laplacian term encourages smoothness. The combined loss is L_pm = λ_rec L_rec + λ_smooth L_smooth, where L_rec is per-pixel MSE and L_smooth is the Laplacian smoothing loss within the masked region.

Rendering. The predicted position map is resampled into a mesh according to the template UV layout and rendered with a variant of the GaussianAvatars pipeline. Each triangular face gets one Gaussian with local rotation, position, and scale relative to the face center; during animation the global transform and rotation are updated from the predicted geometry while the local parameters stay fixed as learnable parameters. Colors use 3rd-order spherical harmonics and are alpha-composited. Training optimizes L = (1−λ)L₁ + λL_D-SSIM + λ_pos L_pos + λ_scale L_scale, where L_pos and L_scale regularize spatial shift and Gaussian scale. The authors use the same hyperparameters as documented in GaussianAvatars for 3DGS training. At inference, mouth interiors are refined with a pretrained GFP-GAN model, strictly confined to the mouth interior, which is frequently occluded during capture; the enhancement does not alter the decoded emotion.

Evaluation. For each emotion-specific trial, the final 15% of frames serve as the test set and the remaining frames for training, a split described as following standard signal processing practice to minimize statistical leakage. The EMOSTIM sequence is used only for cross-trial generalization testing.

Why This Matters

Research impact. The paper argues that neural signals carry richer affective and geometric information than previously assumed, and it marries two previously separate lines of work: EEG-based emotion classification (which produces coarse labels) and neural avatar rendering (which consumes visual cues). It also introduces a synchronized EEG-to-3D capture resource that the authors intend to release, and it demonstrates that a dense position-map representation is a better decoding target than a low-dimensional blendshape basis for noisy neural input.

Real-world applications discussed or implied by the paper:

• Emotion-aware telepresence, where an avatar reflects a user's internal state even when their face is hidden.
• Expressive avatar control inside virtual reality, specifically when the face is fully occluded by a head-mounted display.
• Gaming and collaboration scenarios requiring EEG-driven avatars.
• Adaptive avatars that reflect personalized neural patterns.
• Neuro-visual datasets and tools for decoding attention, emotion, and memory into visual form.

Industry relevance. Applications span VR/AR headset ecosystems, virtual production and digital doubles, remote collaboration platforms, affective computing and mental-state monitoring, and clinical or accessibility contexts where motor or facial control is limited. The paper frames the head-mounted-display case as exploratory and states explicitly that it is not intended as a complete VR system.

Future Directions

• Scale the dataset. The authors plan to expand the dataset with more subjects and more diverse stimuli to improve robustness and cross-subject generalization. The current evaluation covers two subjects across six trials, and the paper notes that neural responses are highly individualized, so subject-specific modeling was chosen over cross-subject modeling.
• Incorporate stimulus context. They propose incorporating stimulus context during decoding to better disentangle perceptual and affective signals for more accurate facial synthesis.
• Address the low-dimensional, noisy nature of EEG. The paper identifies the low dimensionality and noisiness of neural signals as the core difficulty in high-fidelity reconstruction; whether richer channel counts or alternate sensing could improve fidelity is left open.
• Establish comparative evaluation. Because the authors state there is no direct baseline for this pioneering setting, standardized comparisons against other decoding targets and encoder architectures remain to be defined.

Not reported in the paper content: rendering speed or frame rate, inference latency, user studies, statistical significance tests, and any quantitative comparison against a third-party neural decoding baseline.

Target Audience

This paper is best suited to researchers and graduate students working on brain–computer interfaces, EEG decoding, affective computing, and neural avatar or head-reconstruction systems. It is also relevant to graphics and VR/AR engineers interested in driving photorealistic avatars from non-visual signals, and to practitioners building multimodal capture rigs who need frame-accurate synchronization between biosignals and high-speed multi-view video. Readers without background in EEG preprocessing, 3D morphable models, and Gaussian Splatting will find the methodology dense, since the paper assumes familiarity with all three.

Authors’ abstract

Current expressive avatar systems rely heavily on visual cues, failing when faces are occluded or when emotions remain internal. We present Mind-to-Face, the first framework that decodes non-invasive electroencephalogram (EEG) signals directly into high-fidelity facial expressions. We build a dual-modality recording setup to obtain synchronized EEG and multi-view facial video during emotion-eliciting stimuli, enabling precise supervision for neural-to-visual learning. Our model uses a CNN-Transformer encoder to map EEG signals into dense 3D position maps, capable of sampling over 65k vertices, capturing fine-scale geometry and subtle emotional dynamics, and renders them through a modified 3D Gaussian Splatting pipeline for photorealistic, view-consistent results. Through extensive evaluation, we show that EEG alone can reliably predict dynamic, subject-specific facial expressions, including subtle emotional responses, demonstrating that neural signals contain far richer affective and geometric information than previously assumed. Mind-to-Face establishes a new paradigm for neural-driven avatars, enabling personalized, emotion-aware telepresence and cognitive interaction in immersive environments.