VIBE: A VAD-Informed Benchmark for Entity-Centered Affective Profiling of Large Language Model Outputs

AI summary

Overview

Research area: Natural language processing, specifically LLM evaluation, measurement methodology, bias/fairness measurement, and dimensional affect modeling (Valence–Arousal–Dominance).

Technical level: Intermediate. The paper is written around a measurement contract rather than a novel model architecture, but it relies on correlation coefficients, Euclidean distances in VAD space, variance decomposition (η²), bootstrap confidence intervals, and inter-annotator agreement statistics.

One-sentence scope: VIBE is a benchmark and reporting framework for profiling the affect that LLM-generated text expresses toward named socially salient targets in Valence–Arousal–Dominance space, released through a structured "Affective Passport" artifact.

What This Paper Is About

When an LLM writes about a political figure, a country, a religion, or a historical event, it does more than state facts: the text also conveys whether that target seems favorable or threatening, calm or conflictual, powerful or vulnerable. Prior work measures parts of this with scalar favorability scores, sentiment labels, or emotion benchmarks, but none combines target-directed VAD attribution with an explicit scorer contract and a standard reporting format. VIBE's goal is to define what exactly is being measured, who is doing the measuring, under which elicitation protocol, and within which interpretation limits — and then to show empirically that the current practice of using a single scalar score loses real information.

Key Contributions

1. A measurement contract. VIBE separates generation from external scoring, distinguishes three scoring modes (scalar target favorability, response-level VAD, and target-directed VAD), and reports every profile through an Affective Passport that records scorer identity, coverage, protocol, and interpretation limits.
2. Quantified substitution costs across 2,613 targets and six models: scalar favorability correlates with valence at r = 0.875 but does not subsume arousal and dominance, and response-level VAD diverges from target-directed VAD with a mean L2 distance of 0.240.
3. Protocol-drift quantification showing that situation framing shifts affective profiles far more than model identity does — η²_family = 0.141 versus η²_model = 0.010 — so cross-model comparisons require protocol metadata. The paper describes this as protocol choice explaining 14× more drift variance than model choice.
4. The Affective Passport, a condition-explicit JSON artifact tying each profile to its scorer, protocol, coverage, and limits, with a builder script (scripts/build_passport.py) and a canonical rebuild command (bash scripts/build_paper.sh).

Main Findings

Valence is cross-validated, arousal and dominance are not. Judge–human agreement on valence is r_V = 0.944 and inter-scorer agreement is r_V = 0.954. Arousal and dominance are described as single-scorer directional estimates, not point-precise, consistent with human inter-annotator agreement of r_A^hh = 0.495 and r_D^hh = 0.702 (and r_V^hh = 0.798). The paper attributes this ceiling partly to the emotion-label-to-VAD mapping used on the human side, where 28 emotion labels map to only 23 distinct VAD points.

Scalar favorability does not subsume arousal and dominance (H1). Under brief evaluative prompting on 15,678 rows with 100% join coverage, scalar–valence alignment reaches r = 0.9491, yet within-bin arousal/dominance standard deviation is 0.1806, the mean absolute dominance residual from the 0.5 midpoint is 0.2786, and 41.19% of near-neutral-valence rows still carry non-neutral arousal or dominance.

Irreducibility persists across elicitation settings. Under open descriptive prompting on 15,671 rows, r = 0.8749 with a hidden-affect share of 26.68%. In the Buyl replication track of 129,181 usable scored rows (99.9930% coverage), r = 0.7586, within-bin standard deviation 0.1870, and 42.56% of rows with valence in [0.45, 0.55] still show non-neutral arousal or dominance under the primary scorer.

Hidden affect is operationally defined and illustrated. A row has hidden affect when valence is in [0.45, 0.55] and arousal ≥ 0.70, dominance ≥ 0.75, or dominance ≤ 0.25. The worked example is "Mongol invasions and conquests," scoring scalar = 0.5, valence = 0.5, arousal = 0.8, dominance = 1.0. The share stays ≥ 0.35 under every alternative band and threshold variant tested, but drops to below 1% under strict cross-scorer agreement.

The effect is strongest for political persons (H1.6). Per-family hidden-affect shares run from 0.770 for political persons (N = 2,688) and 0.667 for historical figures (N = 1,284) down to 0.149 for abstract phenomena (N = 1,164), with a corpus mean VAD residual of 0.438 across 13 families.

Response-level and target-directed VAD are different contracts (H2). Across 15,626 paper-grade pairs, the mean Euclidean VAD distance is 0.240 (95% bootstrap CI [0.238, 0.242]), with a 90th percentile of 0.376. Per-axis mean absolute deltas are |Δ_V| = 0.095, |Δ_A| = 0.091, and |Δ_D| = 0.170 — dominance carries the largest contract gap. The 52 excluded pairs (0.33%) were flagged by low coverage, refusal, or missing evidence.

Situation framing dominates protocol drift (H3). Across 342,779 rows and five protocol families: situation/factor drift has mean 0.342 and max 0.632; instruction language mean 0.028 / max 0.056; task regime mean 0.151 / max 0.151; role-play turns mean 0.085 / max 0.158; dialogue topic mean 0.032 / max 0.052. The paper concludes that an Affective Passport should record protocol and high-drift condition tags before model identity.

Cheaper scorers were tested and rejected as primary instruments. NRC-VAD lexical means track the LLM judge weakly (r ≈ 0.60 against the Qwen judge) and are not target-directed; cross-scorer Qwen–GPT agreement is r = 0.95 with mean VAD distance ≈ 0.72. Pretrained VAD regressors lack target-visible attribution.

Inter-scorer agreement is uneven across axes. Qwen and the swap scorer agree on valence (r = 0.954) but less on arousal (r = 0.418) and dominance (r = 0.573) across 13,065 paired rows.

Entity bank and design. The v2 bank contains 2,613 unique targets across 13 families (raw: 2,846; 233 removed), built from Wikidata SPARQL queries with a sitelinks threshold and Wikipedia pageview salience over a 12-month window. 40% carry the standard sensitivity label and 60% are flagged as politically, geopolitically, socially, or historically sensitive; political persons, ideologies, and geopolitical events make up 39% of targets.

Methodology in Plain English

The researchers first built a bank of 2,613 socially salient entities from Wikidata, spanning families such as political persons, countries, religions, organizations, and historical events, and deliberately skewed it toward contested targets. They then had six instruction-tuned generators — GigaChat 3.1 10B A1 8B BF16, YandexGPT 5 Lite 8B Instruct, Ministral 3 14B Instruct 2512 BF16, Gemma 4 26B FP8, Granite 4.1 8B, and Qwen 3 6/35B A3B — produce text about each target under controlled conditions (default generation: temperature 0.0, top-p 0.95, seed 13, max tokens 512, parallelism 8).

Separately, external LLM judges scored that text. The key design move is that generation and scoring are separate stages with a declared scorer identity. Three scoring modes were used: a single positive–negative favorability score, a VAD score of the whole response with the target hidden from the scorer, and a VAD score directed specifically at the named target. Primary scoring used a Qwen3.6 judge for five of six generators; to limit same-family self-judgment, Qwen generator outputs were scored by a Gemma-4 judge and Gemma outputs by Qwen, with a GPT-4o-mini cross-check reported separately.

The empirical work is organized as three hypotheses. H1 tests whether a scalar favorability score can stand in for a full VAD profile, across a three-step elicitation ladder of brief evaluative prompting, open descriptive prompting, and a Buyl replication at scale. H2 compares response-level and target-directed VAD on the same text using paired records from the same scorer. H3 varies five protocol dimensions and measures Euclidean distance between mean VAD vectors for factor pairs. Human calibration covered 325 targets with four annotations each, and all headline metrics were given 2,000-resample bootstrap confidence intervals at the row level.

Why This Matters

Impact on research. The paper reframes entity-centered affect evaluation as a measurement-contract problem rather than a leaderboard problem. It provides evidence that a widely used scalar practice loses arousal and dominance signal, that a three-axis score still diverges depending on whether the target is visible to the scorer, and that elicitation protocol explains more variance than model choice. This directly challenges cross-model comparisons made without protocol metadata and gives evaluators a concrete reporting artifact, the Affective Passport, modeled on data statements, model cards, and datasheets.

Real-world applications:

• Bias and representation audits. Organizations auditing how a deployed model describes religious groups, nationalities, or social groups can use target-directed VAD to catch framings — such as low dominance or high arousal — that a favorability score would report as neutral.
• Model selection and procurement. Because situation framing produces mean drift of 0.342 versus a model-identity η² of 0.010, buyers need to know that reported differences may reflect the test protocol rather than the model.
• Media and content moderation tooling. Distinguishing whole-response tone from target-directed framing helps identify text that reads calmly overall while depicting a named entity as threatening or powerless.
• Evaluation infrastructure and documentation. The Affective Passport schema, entity bank, frozen scorer prompts, and build script offer a template for releasing affective measurements with provenance and stated limits.

Industry relevance. Any team shipping LLM-generated text about people, institutions, or countries faces the framing question the paper formalizes. The finding that dominance carries the largest gap between scoring contracts (mean |Δ_D| = 0.170) is especially relevant to applications where the depiction of agency and power matters, such as news summarization, policy briefs, and educational content.

Future Directions

• Human validation beyond target-directed VAD. The paper states that scalar favorability and the H2 response-level divergence lack direct human validation; human calibration currently covers target-directed VAD only, on 325 targets with a Qwen3.6 judge.
• Extending H3 beyond VIBE-owned protocol families. The H3 layer covers five VIBE-owned protocol families with one primary judge, and the H3.2 instruction-language result is reported as aggregate drift only, explicitly not as per-target invariance or full multilingual robustness.
• Addressing the dominance construct gap. The H1.5 hidden-affect share drops from at least 41% under the primary scorer to below 1% under strict cross-scorer agreement in the neutral-valence bin, which the paper identifies as a dominance construct gap rather than random noise.
• Personalization and persona slices. The paper names persona slices and full human evaluation as remaining future work, and the released prompt/role infrastructure includes nine role prefixes and seven prompt templates that the reported experiments used only in part.

Target Audience

This paper is most useful to researchers and practitioners in LLM evaluation, fairness and bias measurement, and computational social science who need to report affective measurements of model outputs with defensible scope. It also suits NLP engineers building evaluation pipelines, model-card and documentation authors, and policy or trust-and-safety teams who need to interpret claims about how a model treats a named entity. Readers should be comfortable with correlation statistics, confidence intervals, and the VAD dimensional model of affect; the paper's conceptual contribution about separating generation from scoring is accessible without that background.

Authors’ abstract

Large language models routinely describe socially salient targets, including political figures, countries, religions, organizations, historical events, and social groups, encoding affective framing alongside factual content: a target may appear favorable or threatening, calm or conflictual, powerful or vulnerable. Existing work captures parts of this space through sentiment, favorability, and emotion benchmarks, but none combines target-directed VAD attribution, an explicit scorer contract, and a passport reporting format. We introduce VIBE, a benchmark for entity-centered affective profiling of LLM outputs in Valence-Arousal-Dominance (VAD) space. Its core contribution is a measurement contract: VIBE separates generation from external scoring, distinguishes scalar favorability, response-level VAD, and target-directed VAD, and reports profiles through an Affective Passport. Three empirical layers support the contract. H1 shows scalar favorability does not subsume arousal and dominance: valence findings are cross-validated (rV = 0.944 judge-human, rV = 0.954 inter-scorer); arousal and dominance are single-scorer directional estimates, not point-precise, consistent with known inter-annotator difficulty on these axes (rA = 0.495, rD = 0.702 among human annotators). H2 shows whole-response and target-directed VAD are different contracts: the same text can carry one affective tone overall while representing the named target differently. H3 is a protocol-drift diagnostic: elicitation conditions shift profiles, motivating context metadata in every affective report. These results motivate entity-centered affective profiling as a documented practice: profiles should be released with scorer identity, coverage, protocol, and interpretation limits.