Known-Voice Cloning and the Limits of Personality Transfer for Character Design

AI summary

Overview

• Research area: Voice cloning, human–robot interaction, and social perception (arXiv category: Natural Language Processing, cs.CL), accepted for publication at Robophilosophy 2026 (LNCS volume 17299).
• Technical level: Intermediate. The design is conceptually simple, but readers benefit from familiarity with Big Five personality measures and nonparametric significance testing.
• Scope: Two empirical studies (a university lecture study, n = 17, and a preregistered online study, n = 65) testing whether personality impressions of a familiar person transfer to a virtual character that speaks in a cloned version of that person's voice.

What This Paper Is About

Zero-shot voice cloning can now reproduce a recognizable voice from a short recording and attach it to any virtual character or robot, which raises the concern that familiar voices could be used to manipulate people into treating artificial characters as capable social actors. The authors ask whether hearing a familiar person's cloned voice actually causes listeners to transfer their stored personality impressions of that person onto an embodied character. They test this separately from mere voice recognition, distinguishing between identity activation, trait transfer, and the stronger case of person reconstruction.

Key Contributions

1. Empirical evidence across two contexts that familiar voice cloning yields only small, trait-selective alignment effects rather than robust person-consistent evaluations.
2. A qualitative analysis showing that judgments were often dominated by perceived artificiality and voice–appearance mismatch rather than by the familiar voice itself.
3. A three-level conceptual framework (identity activation, trait transfer, person reconstruction) used to interpret what voice-based familiarity can and cannot do in embodied characters.
4. Practical guidance for responsible character design, identifying boundary conditions under which familiar voice cloning is unlikely to support social evaluation and may backfire.

Main Findings

• Identity activation occurred, but transfer was weak. Participants frequently recognized or commented on the familiar voice, indicating identity activation, but evidence for trait transfer was selective and context-dependent, and there was little to no evidence for person reconstruction.
• Study 1 (lecturer, n = 17) found two significant effects. Wilcoxon signed-rank tests showed significantly smaller person–character distances in the known-voice condition for likability (W = 11.5, p = .010, rank-biserial r = −.75) and agreeableness (W = 16.0, p = .038, rank-biserial r = −.59). Competence and the remaining Big Five dimensions were not significant.
• Study 1 effect sizes were modest and asymmetric. Likability distances were lower in the known-voice condition (M = 1.96, SD = 1.35, Mdn = 2.00) than in the unknown-voice condition (M = 2.56, SD = 1.32, Mdn = 2.60); the same pattern held for agreeableness (M = 1.28, SD = 1.05, Mdn = 1.00 vs. M = 1.74, SD = 0.92, Mdn = 2.00).
• Study 2 (content creators, n = 65, preregistered) found only one effect. Extraversion was the only outcome in the predicted direction (W = 588.0, p = .041), with descriptively lower distances in the known-voice condition (M = 1.02, SD = 0.81, Mdn = 1.00) than in the unknown-voice condition (M = 1.19, SD = 0.84, Mdn = 1.00). Likability, competence, and trust were not significant.
• The Study 1 effects did not replicate. Likability and agreeableness effects from Study 1 did not reappear in Study 2; instead, only extraversion showed the predicted effect.
• Recognition was salient but fidelity was imperfect. Recognition of the cloned voice was the salient theme in the known-voice condition (17 mentions), with participants often acknowledging similarity while emphasizing deviations from the original voice.
• Artificiality dominated impressions. Across both conditions, the dominant qualitative theme was visual rigidity and artificial facial movement (unknown: 15 mentions; known: 13 mentions), described as limited expression, stiffness, and in some cases uncanny or unsettling.
• Small asymmetries in qualitative comments. Comments describing the voice as comparatively natural or pleasant were more frequent in the unknown-voice condition (5 vs. 1 mention), while voice–appearance mismatch was reported more often in the known-voice condition (3 vs. 1 mention).
• Traits differ in how voice-diagnostic they are. The authors suggest extraversion is closely tied to acoustic parameters such as vocal effort, intensity, and dynamic variability, whereas traits like agreeableness or likability may depend more on broader behavioral cues.
• Familiarity can raise scrutiny. Because familiar listeners hold robust mental representations of familiar voices, a cloned voice that diverges from that stored prototype may shift attention toward authenticity evaluation and away from person-based impression formation.

Methodology in Plain English

Participants first rated a real, familiar person from memory to capture their baseline internal person model. They then evaluated the same virtual character in two conditions: a known-voice clone and a cloned unknown voice. Visual appearance and spoken content were held constant, making voice the only manipulated factor. Alignment was operationalized as smaller absolute distances between the person ratings and the character ratings, indicating that the character was being rated more like the remembered person.

Study 1 was exploratory: 18 completed questionnaires were collected in a university teaching context, with one participant excluded for not knowing the lecturer, leaving 17 valid responses (mean age 22.19 years, SD = 2.99, range 20–32). Participants used their own smartphones and headphones during a lecture or tutorial. The familiar person was their lecturer.

Study 2 was preregistered and online: of 208 participants who started the survey, 65 valid responses remained after preregistered exclusions for incomplete responses, failed attention checks, and insufficient creator recognition (mean age 27.9 years, SD = 6.38, range 19–52). Participants were recruited through creator-related online communities, primarily Reddit, supplemented by mailing lists and social media posts. Familiarity was verified by correct identification of both the creator's real name and artist name, plus content consumption within the previous three months.

Voice clones were generated with zero-shot systems: OpenVoice in Study 1 and XTTSv2 in Study 2. The virtual character and lip synchronization were created with MeLaX. Stimuli were roughly 14 s in Study 1 (Oldenburg sentences) and roughly 43 s in Study 2 (trivia statements), with the longer stimuli chosen to support more reliable personality judgments following participant feedback from Study 1. Personality was measured with BFI-K (21 items) in Study 1 and BFI-10 in Study 2; likability and competence items came from established person-perception scales, and Study 2 additionally measured trust. Condition order was randomized in both studies, and the memory-based familiar-person rating always came first. Analysis used paired t-tests or Wilcoxon signed-rank tests, with one-tailed tests in Study 2 per the preregistered directional plan; neither study corrected for multiple comparisons. Open-ended responses from Study 2 were analyzed with qualitative content analysis. Voices were cloned with permission from the voice owners or from explicitly authorized materials, and all participants gave informed consent.

Why This Matters

• Impact on research: The work moderates a common assumption that cloned familiar voices straightforwardly make artificial characters appear socially continuous with the people they resemble. It reframes familiar voices as weak, context-dependent identity cues rather than reliable carriers of personality, and it argues that multimodal coherence (voice, appearance, behavior) constrains whether such cues shape evaluation at all.
• Human–robot interaction design: Designers considering a known voice for a robot should expect mismatch awareness rather than smooth personality transfer, especially if the visual carrier does not fit the voice.
• Education and healthcare: The authors specifically discuss vulnerable users in these domains, noting these users might have more difficulty distinguishing reality from simulation or be more affected by uncanny impressions, and they discourage default use of familiar voices there.
• Voice-loss and speech-impairment tools: The paper acknowledges beneficial uses, such as giving autonomy to people with speech impediments or voice loss by letting them communicate in a voice they identify with, as well as users choosing a voice for a robot they personally own.
• Consent and platform governance: The authors call for transparent consent mechanisms for voice cloning, the ability for voice owners to track, update, and retract consent, monitoring of platforms providing these technologies, and better deepfake detection for listeners.
• Industry relevance: For developers of zero-shot voice cloning and virtual character pipelines, the findings suggest that voice choice alone is a weak lever for shaping user perceptions, and that investment in visual fidelity and audiovisual coherence may matter more than voice identity.

Future Directions

• Exam

Authors’ abstract

Recent advances in zero-shot voice cloning systems make it possible to reproduce a recognizable voice from a short recording and attach it to arbitrary virtual characters. Prior work suggests that familiar vocal identity can carry over associated personality impressions to new targets such as robots or virtual characters. However, these works have mainly examined familiarity and personalization effects rather than directly testing whether a cloned familiar voice systematically transfers personality impressions to an embodied virtual character. Such transference raises the concern that cloned voices could be used to manipulate people into engaging with virtual characters as capable social actors. Across two studies, we investigated whether personality impressions transfer when a virtual character speaks with the cloned voice of a familiar person. Study 1 examined this question in a real-world university setting using the cloned voice of a lecturer familiar to the participating students. Study 2 examined the same question online using voices of social media content creators and members of their communities. Drawing on the concepts of identity activation, trait transfer, and person reconstruction, we show that evidence of personality transfer was limited in both studies. Observed effects were selective and did not generalize across traits or study contexts. In the online study in particular, recognizable voice identity cues activated expectations about the speaker but did not reconstruct the broader person impression. Qualitative responses further indicated that judgments were frequently shaped by perceived artificiality and by whether voice and appearance fit together. Our findings moderate common concerns that cloned voices straightforwardly manipulate people into perceiving artificial characters as capable social actors.