A Platform for Interactive AI Character Experiences

AI summary

Overview

Research area: Human-Computer Interaction, specifically embodied conversational agents, interactive storytelling, and multimodal AI systems.

Technical level: Intermediate. The paper is written accessibly and surveys each component, but readers will get the most value if they are familiar with concepts such as large language models, speech synthesis, embedding spaces, and real-time rendering pipelines.

Scope: The paper presents a modular, end-to-end platform for building believable, story-driven digital characters, demonstrated through a physical "Digital Einstein" installation that converses with users about Einstein's life and research.

What This Paper Is About

Recent AI advances have solved many individual sub-problems required for lifelike digital characters — language modeling, voice synthesis, facial animation, memory — but no unified system integrates them all while preserving character consistency, personality, and real-time responsiveness. The authors build a modular platform that wires all of these components together behind a single character, then deploy it as a physical booth where visitors hold spoken conversations with a digital Albert Einstein. The goal is to show that believable, story-driven character experiences are achievable today when the pieces are designed to work as one system rather than as isolated research artifacts.

Key Contributions

1. A unified modular architecture for conversational AI characters that integrates speech recognition, LLM-based dialogue, memory, adjustable personality, emotion detection, speech synthesis, facial and body animation, image generation, and real-world sensing into one pipeline with four character states (idle, listening, thinking, speaking).

2. An embedding-based topic steering and memory mechanism that combines 62 curated character topics, 4,402 synthetic conversations, a 3,072-dimensional embedding space, and an exponentially weighted aggregation of recent turns to keep GPT-4o on-theme while permitting smooth transitions between topics.

3. A dual-model conversational design in which GPT-4o handles high-quality cloud interactions and a fine-tuned Llama 3 8B provides a local, privacy-preserving offline fallback, ensuring resilience during service outages.

4. A physical, immersive installation ("Digital Einstein") with themed furniture, spatial audio, hidden microphones and cameras, and five physical sliders that let users adjust personality traits in real time, validated through in-the-wild deployments at two large conferences.

Main Findings

• The system sustains real-time conversation despite heavy computation. Cumulative pipeline latency is 4.7 seconds with GPT-4o and 5.14 seconds with Llama 3, and the character's animated "thinking" state masks this delay effectively enough to keep sessions flowing.

• GPT-4o produced more coherent, on-topic conversations than the local model. With topic steering active, GPT-4o scored low circuitousness (0.02 at both events) and covered 49 and 42 of the 62 Einstein topics at the tech and scientific events, versus only 22 topics for the unsteered Llama 3.

• Llama 3 conversations were more exploratory but less focused. It showed higher semantic speed (1.04), volume (0.70), and circuitousness (0.24), meaning users roamed more broadly across subjects rather than staying within Einstein's domain.

• Real-world engagement varied with venue and audience. The scientific event (SIGGRAPH Asia, 261 sessions) generated longer and richer interactions (5.67 turns, 32.78 words per response) than the tech event (GITEX, 374 sessions, 4.84 turns, 29.35 words).

• Event-specific topics naturally emerged. Beyond the 62 curated Einstein topics, users discussed "GITEX" (241 mentions), "Dubai" (61), "SIGGRAPH Asia" (201), and "Tokyo" (83), demonstrating that the character generalizes to unexpected conversational ground.

• Emotion-aware synthesis and speech-driven facial animation work in tandem. A seven-emotion classification from GPT-4o-mini is passed to both the Azure neural voice (for speaking style) and Audio2Face (for facial expression), while Audio2Face achieved lower lip offset error (μ = 0.005) than the fallback SALSA system (μ = 0.006).

• A camera-based perception layer enables situational awareness. Structural similarity to an empty chair, face detection, age-gender classification, head pose estimation, and face re-identification allow the character to detect whether someone is seated, whether they are attentive, and whether they have been seen before for personalized memory retrieval.

Methodology in Plain English

The researchers treated the digital character as a pipeline of specialized modules rather than a single monolithic model. A camera watches the physical space and decides when someone has sat down; a hidden microphone captures speech, which Azure's speech service transcribes; the transcript is fed into a chatbot along with context about the scene, the user, the current date, and memories from earlier turns.

For the chatbot itself, they used two language models. GPT-4o handles the main cloud-based experience, while a locally deployed Llama 3 8B, fine-tuned on synthetic character conversations, serves as an offline backup. To keep conversations on-character and on-topic, they first generated 71 synthetic dialogues for each of 62 Einstein topics, embedded every conversation turn into a 3,072-dimensional vector, and averaged those vectors into topic representations. During a live conversation, the system embeds each new turn, combines it with recent turns using a decaying weight (so the most recent turns matter most), and compares the result against the topic vectors. If a topic match is strong, the chatbot is nudged toward staying on it; nearby topics are offered as smooth exit routes for natural transition.

Personality is handled separately from the chatbot's base responses. A rewriting step rephrases outputs according to five traits — vibrancy, conscientiousness, decency, artificiality, and neuroticism — that users control with physical sliders on a 3D-printed board. After each response, a smaller model labels the emotional tone, which is fed to both the voice synthesizer (to pick a speaking style) and the facial animation system (to shape expressions). Body motion comes from a curated library of pre-recorded motion-capture clips chosen by character state, while facial animation is synthesized from audio in half-second windows. Midjourney generates illustrative images matching the conversation topic, filtered by CLIP score, and described back to the chatbot for context.

For evaluation, they deployed the finished booth at two large conferences — GITEX GLOBAL 2024 and SIGGRAPH Asia Emerging Technologies 2024 — logging hundreds of real user sessions and comparing conversation metrics (speed, volume, circuitousness, defined by Toubia et al. 2021) across GPT-4o, Llama 3, and the synthetic training data.

Why This Matters

Impact on research. Most prior work optimizes a single component — animation, dialogue, or memory — in isolation. This paper's main contribution to the field is architectural: it shows that the real bottleneck for believable characters is integration, synchronization, and latency management across heterogeneous modules, not any one model. The embedding-based topic-steering scheme and the dual-model resilience pattern are reusable ideas beyond digital characters.

Real-world applications:

• Museums, cultural heritage, and historical sites — interactive guides that embody historical figures and answer questions about their lives and work.
• Education and training — language tutors, patient simulators, or historical role-play partners that adapt personality and emotional tone to the learner.
• Theme parks, retail, and brand experiences — characters that greet visitors, recognize returning guests, and drive themed storytelling in physical spaces.
• Accessibility and companionship — attentive conversational agents for elderly users or isolated individuals, with local processing options that protect privacy.

Industry relevance. The system demonstrates commercially meaningful patterns: cloud-plus-local redundancy for reliability, region-aware deployment for GDPR compliance, real-time performance budgeting across cloud APIs, and a physical hardware interface (sliders) that makes AI control tangible for non-technical users. Theme parks, entertainment studios, and customer experience vendors could adopt these patterns directly.

Future Directions

• Interruption handling. The system cannot yet be interrupted mid-response by the user — a fundamental gap for natural conversation that the authors plan to address.
• Better body animation. The team wants more diverse and lively motion; current motion-capture clip libraries limit expressiveness, and retargeting gesture-synthesis models to a stylized body produced unnatural results.
• Stronger cognitive modeling. The authors aim to move beyond current LLM behavior toward more human-like understanding and reasoning within the character's persona.
• Systematic user studies. The deployment produced extensive logs but no controlled experiment isolating how much each component (memory, personality sliders, emotion, image generation) contributes to engagement and perceived immersion.
• Latency in fast-paced exchanges. Distributed computing still introduces occasional delays that disrupt rapid back-and-forth dialogue, an open engineering problem.

Target Audience

This paper is most useful to HCI and graphics researchers building embodied agents, practitioners architecting multimodal AI systems who need a reference integration design, and developers in entertainment, education, or cultural heritage who want to deploy conversational characters in physical venues. Readers seeking deep technical novelty in any single component (e.g., a new animation model or a novel LLM architecture) will find the paper surveys existing techniques rather than pushing any one of them forward — its contribution is the whole machine, not its parts.

Authors’ abstract

From movie characters to modern science fiction - bringing characters into interactive, story-driven conversations has captured imaginations across generations. Achieving this vision is highly challenging and requires much more than just language modeling. It involves numerous complex AI challenges, such as conversational AI, maintaining character integrity, managing personality and emotions, handling knowledge and memory, synthesizing voice, generating animations, enabling real-world interactions, and integration with physical environments. Recent advancements in the development of foundation models, prompt engineering, and fine-tuning for downstream tasks have enabled researchers to address these individual challenges. However, combining these technologies for interactive characters remains an open problem. We present a system and platform for conveniently designing believable digital characters, enabling a conversational and story-driven experience while providing solutions to all of the technical challenges. As a proof-of-concept, we introduce Digital Einstein, which allows users to engage in conversations with a digital representation of Albert Einstein about his life, research, and persona. While Digital Einstein exemplifies our methods for a specific character, our system is flexible and generalizes to any story-driven or conversational character. By unifying these diverse AI components into a single, easy-to-adapt platform, our work paves the way for immersive character experiences, turning the dream of lifelike, story-based interactions into a reality.