FURINA: A Fully Customizable Role-Playing Benchmark via Scalable Multi-Agent Collaboration Pipeline

AI summary

Overview

Research area: Natural Language Processing — specifically evaluation of large language models (LLMs) in role-playing (RP) conversational agents.

Technical level: Advanced. The paper assumes familiarity with LLM-as-a-judge evaluation, multi-agent pipelines, pairwise comparison scoring, and hallucination taxonomy.

Scope: The paper introduces FURINA-Builder, a multi-agent pipeline that automatically constructs customizable role-playing benchmarks at arbitrary scale, and FURINA-Bench, the bilingual benchmark built with it, along with an evaluation of cutting-edge LLMs on it.

What This Paper Is About

Existing role-playing benchmarks are static: they use fixed character sets, fixed dialogue structures, and fixed evaluation dimensions, so they go stale quickly and cannot be adapted to a user's specific NPC designs or application scenarios. The authors' goal is to make benchmark construction itself dynamic — a pipeline where users can plug in arbitrary characters, scenes, prompt formats, and evaluation criteria, and have the benchmark generated automatically. To demonstrate the pipeline works, they use it to build FURINA-Bench and evaluate a wide range of modern LLMs on it.

Key Contributions

1. FURINA-Builder, described as the first multi-agent collaboration pipeline for automatically constructing fully customizable RP benchmarks at arbitrary scales, with human evaluation justifying the builder design.
2. FURINA-Bench, a comprehensive RP benchmark built with that pipeline, combining both established and synthesized test characters in group-chat scenarios with fine-grained, dimension-specific evaluation criteria. A preliminary analysis shows it provides clearer model separability.
3. Extensive evaluation of cutting-edge LLMs on FURINA-Bench, producing key insights plus analysis of model performance and limitations.
4. Definition and measurement of role-playing hallucination subtypes — Established-character (EC) hallucination and Synthesized-character (SC) hallucination — mapped onto the factuality/faithfulness taxonomy.

Main Findings

• Best English and Chinese models differ: o3 achieves the strongest overall English RP score (43.98) and leads English Context Reliance, Conversational Ability, and Preference Alignment; DeepSeek-R1 achieves the strongest overall Chinese RP score (73.38). On Chinese, Qwen3-32B/235B-thinking perform best across most RP dimensions.
• Qwen3 shows scaling benefits: weighted average rises from Qwen3-8B (13.39 English / 45.63 Chinese) to Qwen3-32B (14.80 / 63.10) to Qwen3-235B-A22B (22.77 / 59.97).
• Chinese results are stronger at smaller scales: Qwen3 models deliver significantly stronger Chinese results even at smaller sizes, which the authors attribute to language-specific training data.
• Established characters beat synthesized ones, for every model: English gaps range from +0.04 (Qwen3-8B) to +19.02 (DeepSeek-R1); Chinese gaps range from +2.93 (Qwen3-8B) to +17.24 (Qwen3-32B-thinking).
• Reasoning amplifies the established/synthesized gap: reasoning models show significantly larger gaps, which the authors attribute to reasoning prioritizing pre-existing character knowledge over prompt-provided detail, while weakening instruction-following.
• Model scale does not monotonically reduce hallucination: the authors state hallucination rates do not follow a monotonic trend with scale, suggesting training data composition matters more than raw capacity.
• Reasoning raises RP performance but increases hallucination: all Qwen3 variants show higher hallucination rates in thinking mode than in no-thinking mode, for both SC and EC, with sharp SC increases at 8B and 32B.
• A Pareto frontier exists between RP performance and reliability: high-performing models such as DeepSeek-R1 often compromise reliability, highly reliable models such as GPT-4o use conservative strategies that suppress performance, and Claude-4-Sonnet in thinking mode achieves the best balance among evaluated models.
• Judge reliability validated: GPT-4.1 dimension selection achieves 0.892 average accuracy on 1000 manually annotated samples (CR 0.909, FR 0.862, RR 0.891, CA 0.908, PA 0.888); on 400 samples, GPT-4.1's Pearson correlation with human scores averages 0.6275, above DeepSeek-R1-0528 (0.5808) and DeepSeek-V3-0324 (0.4982).
• Stronger separability than the GCA baseline: FURINA-Bench evaluation yields a steeper slope across four models than GCA evaluation.

Methodology in Plain English

The pipeline has four parts.

Test character. A character is defined as a set of key–value profile attributes, each tagged public or private. Private attributes stay hidden from other characters during the dialogue, which makes the interaction more realistic and harder. A separate LLM-based pipeline can also synthesize characters from scratch.

Character–scene pool. The authors curated 6,556 scenario fragments from a bilingual corpus of 80 Chinese and 100 English books. Each scenario bundles a background, a motivation, an original reference dialogue from the source text, and the scene characters. Character profiles can shift with time so the same character may have different profiles. Users can add their own scenarios to the pool.

Simulation. A director model decides who speaks next or when to stop. When it is the test character's turn, the reply prompt contains only the public attributes of the scene characters. Scene characters, driven by a scene model, additionally receive the motivation and original dialogue. A dynamically weighted random selection algorithm adjusts how often each evaluation dimension appears so that under-represented dimensions get balanced coverage.

Selection. For every test character turn, both a source model and a stronger base model generate candidate responses. A judge model picks the appropriate evaluation dimension, then uses chain-of-thought reasoning to rate the two candidates on a 5-point Likert scale from 1 (strongly prefers source) to 5 (strongly prefers base). If the score is 3 or below, the source response is kept; if above 3, the base response replaces it. This keeps dialogue quality high and fixes a single dimension per test utterance, avoiding cross-dimension interference. The dimension-utterance pairing is then frozen for benchmark construction.

Benchmark configuration. FURINA-Bench used source models spanning mid-scale open-source models (roughly 24B–70B) to large frontier proprietary systems (GPT, Claude, Gemini series). Qwen3-235B-A22B served as both scene character model and director model; GPT-4.1 served as both base model and judge model. Construction ran until each character hit a minimum threshold of τ = 10 per evaluation dimension, with additional LLM-based and rule-based filtering.

Five evaluation dimensions: Context Reliance (CR), Factual Recall (FR), Reflective Reasoning (RR), Conversational Ability (CA), and Preference Alignment (PA).

Evaluation of models. Each test model and the base model (GPT-4.1) respond to the same prompt, augmented with a response strategy matching the pre-assigned dimension. Judgments are made bidirectionally to offset ordering bias, and averaged via an unbalanced scoring function f: {1,2,3,4,5} → {3, 1, 0.5, 0, 0}. A response earns credit only if it matches or exceeds the base response. Overall performance is the total score divided by 3N. Confidence intervals are computed by bootstrapping with 1000 resamples.

Hallucination measurement. An automatic checker (e.g., Qwen2.5-32B-Instruct) detects hallucination-related keywords in the chain-of-thought judgments produced during evaluation; their occurrence probability serves as the hallucination-rate metric.

Dataset composition. FURINA-Bench is bilingual and was built from 20 test characters: 5 synthesized Chinese, 5 synthesized English, 5 established Chinese, and 5 established English. These characters interact with 1,471 unique roles, producing 1,459 high-quality multi-party dialogues containing 7,181 test utterances. All five dimensions are balanced across languages with over 500 examples each. Table 1 in the paper lists FURINA-Bench at 1,494 conversations with an average of 19.8 turns, and notes it is the only listed dataset supporting established and synthesized characters, dynamic personas, perspective misalignment, multi-character conversations, scenarios, and strategies. Chinese evaluation excluded Llama3.1-8B, Llama3.1-70B, and CoSER-70B due to poor English performance and limited Chinese training; Gemini models were excluded due to red-teaming behavior.

Why This Matters

Impact on research. The paper reframes RP benchmarking from a fixed artifact into a configurable process. It also introduces two operational hallucination subtypes (EC and SC) tied to specific evaluation dimensions, and documents a performance–reliability trade-off that gives future work a concrete target to beat.

Real-world applications:

• Evaluating and selecting models for character-based chat platforms such as Character.ai or Xingchen, where NPC quality must be assessed against a specific design rather than a generic rubric.
• Testing task-specific NPCs in games or simulations before deployment, using the target designer's own character cards and scenarios.
• Auditing role-playing agents for persona drift and hallucination in immersive virtual worlds populated with many interacting NPCs.
• Building internal, domain-specific RP evaluation suites without depending on a fixed public benchmark.

Industry relevance. The trade-off between RP performance and reliability has direct product implications: shipping a more capable, more hallucination-prone role-play model can degrade user experience, so teams need per-dimension reliability metrics rather than a single aggregate score. The base-model choice also matters — the authors note GPT-4.1 as the English baseline is stronger than the Chinese baseline, which depresses English test scores overall.

Future Directions

• Surpassing the Pareto frontier: the paper explicitly asks how to achieve simultaneous gains in RP performance and reliability, suggesting pre-training data curation, post-training with diverse role types, and instruction-sensitive reasoning as promising directions.
• Mitigating the reasoning/instruction-following trade-off: reasoning models gain RP performance but lose on synthesized characters that depend on instruction-following, so recovering that ability is an open problem.
• Understanding hallucination sources: since hallucination rates are not monotonic in model scale, the role of training data composition versus capacity needs further study; the paper reports checking hallucination-related keywords in CoT judgments but the full statistics sit in an appendix truncated here.
• Extending the builder: users can add custom scenarios and designate private attributes, but the paper does not report how well the pipeline generalizes to entirely new scenario domains or languages beyond Chinese and English.

Target Audience

Researchers and engineers working on LLM evaluation, conversational agents, and character/NPC systems will benefit most. It is also useful for practitioners building role-playing products who need to understand why a model that scores well on a benchmark may still hallucinate, and for anyone designing automatic benchmark-construction pipelines using multi-agent collaboration. Readers need a working understanding of LLM-as-a-judge evaluation and pairwise scoring to follow the results tables and the Pareto analysis.

Authors’ abstract

As large language models (LLMs) advance in role-playing (RP) tasks, existing benchmarks quickly become obsolete due to their narrow scope, outdated interaction paradigms, and limited adaptability across diverse application scenarios. To address this gap, we introduce FURINA-Builder, a novel multi-agent collaboration pipeline that automatically constructs fully customizable RP benchmarks at any scale. It enables evaluation of arbitrary characters across diverse scenarios and prompt formats, as the first benchmark builder in RP area for adaptable assessment. FURINA-Builder simulates dialogues between a test character and other characters drawn from a well-constructed character-scene pool, while an LLM judge selects fine-grained evaluation dimensions and adjusts the test character's responses into final test utterances. Using this pipeline, we build FURINA-Bench, a new comprehensive role-playing benchmark featuring both established and synthesized test characters, each assessed with dimension-specific evaluation criteria. Human evaluation and preliminary separability analysis justify our pipeline and benchmark design. We conduct extensive evaluations of cutting-edge LLMs and find that o3 and DeepSeek-R1 achieve the best performance on English and Chinese RP tasks, respectively. Across all models, established characters consistently outperform synthesized ones, with reasoning capabilities further amplifying this disparity. Interestingly, we observe that model scale does not monotonically reduce hallucinations. More critically, for reasoning LLMs, we uncover a novel trade-off: reasoning improves RP performance but simultaneously increases RP hallucinations. This trade-off extends to a broader Pareto frontier between RP performance and reliability for all LLMs. These findings demonstrate the effectiveness of FURINA-Builder and the challenge posed by FURINA-Bench.