Value-Rich, Differently Organized: Measuring Value Expression and Organization in AI-Generated Debates

AI summary

Overview

Research area: AI/NLP — evaluation of LLM-generated argumentation and debate, with a focus on computational modeling of human values in text.

Technical level: Intermediate. The conceptual ideas are accessible, but the metric suite (entropy, Jensen–Shannon divergence, paired effect sizes) assumes some familiarity with NLP evaluation methods.

Scope (one sentence): The paper introduces a Schwartz Basic-10-based diagnostic framework that separates value expression (which values a debate invokes) from value organization (how those values are distributed and sustained across turns), and uses it to compare AI-generated debates against matched human debate references.

What This Paper Is About

Existing evaluation of AI-generated debate concentrates on argument quality — coherence, factuality, stance consistency, persuasiveness — and leaves the value structure of the arguments largely unexamined. Separately, work on values in NLP treats values as labels attached to isolated texts, so it never asks how value appeals are organized across a multi-turn exchange. The paper closes that gap by asking how AI-generated debates compare with matched human debates not only in which values they invoke, but in how those values are spread across utterances, stances, and whole debates.

Key Contributions

1. A diagnostic distinction between value expression and value organization — separating the breadth and reference-alignment of invoked values from how value mass is concentrated and carried across turns.
2. A Schwartz-grounded, multi-level representation of debate values — each argumentative unit (a stance-specific claim paired with an extracted premise) receives a soft distribution over the ten Basic-10 values, aggregated to utterance, stance, and full-debate levels.
3. A metric family covering value coverage, top-k overlap, Jensen–Shannon divergence, entropy at three levels, plus cross-turn stability, drift, and dominant-value persistence.
4. A paired benchmark and analysis (60 topics from the Chinese ORCHID corpus, five generation systems) showing that AI debates activate as many or more values than human references while differing in local concentration and cross-turn organization, with results surviving surface controls and supported by source-blind human judgment.

Main Findings

• AI debates are broadly value-expressive. Human references activate 4.883 Basic-10 values on average (threshold τ = 0.05), while AI systems activate between 5.4 and 5.6: GPT-5.4 5.600, Qwen3.6-Plus 5.567, R-Debater 5.520, Claude-3.6 Sonnet 5.450, DeepSeek-V3.2 5.400. No system shows reduced value coverage.

• Broader coverage does not mean closer alignment. Top-3 overlap with the matched human reference ranges from 0.572 to 0.667 and JSD from 0.102 to 0.133. DeepSeek-V3.2 is closest (JSD 0.102, overlap 0.667), but no system reproduces the reference profile.

• The strongest difference is local value diffusion. Human utterance-level entropy is 1.252 versus 1.470–1.553 for AI. DeepSeek-V3.2 shows the largest paired gap (ΔH_utt = +0.301, d_z = 1.301), then Qwen3.6-Plus (+0.265, d_z = 1.018); other systems have d_z between 0.848 and 0.971.

• Aggregation shrinks the gap. Stance-level entropy is 1.541 for humans versus 1.592–1.756 for AI (largest: Claude-3.6 Sonnet, ΔH_sta = +0.215, d_z = 0.828). At debate level, humans are at 1.651 versus 1.712–1.761 for AI, and GPT-5.4's difference is not significant (+0.062, Holm p = 0.139, d_z = 0.228). Full-debate averaging can mask local diffusion.

• Cross-turn trajectories are smoother for AI. Human stability is 0.831 versus 0.915 (Claude-3.6 Sonnet) to 0.952 (Qwen3.6-Plus). Human drift is 0.213, while Qwen3.6-Plus, DeepSeek-V3.2, and R-Debater show 0.047, 0.051, and 0.078 with perfect persistence (1.000 vs. 0.983 for humans, computed at k = 3). The authors stress near-perfect persistence may also mean reduced adaptive movement, not superiority.

• Differences survive surface controls. Adjusted paired differences remain positive for all dependent variables; the utterance-level effect is the most robust (ΔH_utt = +0.252, 95% CI [+0.205, +0.300]). Generation-system effects are significant only for stability (p = .003), while motion-type effects are significant for all dependent variables.

• Source-blind judges separate clarity from focus. With 120 university debate judges, human references score highest on human-debate likeness (6.42 ± 0.38; best AI: GPT-5.4 at 5.09 ± 0.47, DeepSeek-V3.2 at 4.88 ± 0.49). But AI scores high on value clarity: GPT-5.4 at 6.01 ± 0.46 and DeepSeek-V3.2 at 5.93 ± 0.49, both above the human mean of 5.77 ± 0.61. Humans lead on value focus (5.91 ± 0.56) and are chosen in 74% of pairwise focus comparisons.

• The annotation pipeline is validated. SAPE-Agent reaches observed agreement 0.8612, Cohen's κ 0.7124 (micro) and 0.6897 (macro), Krippendorff's α 0.7035, with premise rates 0.3840 (human) versus 0.4075 (SAPE-Agent). VA-Agent reaches precision 0.74, recall 0.91, F1 0.82, versus F1 0.25 for BERT, 0.20 for SVM, and 0.64 for direct LLM annotation (GPT-4o) — these are annotation baselines, not debate generators.

• A qualitative case confirms the pattern. On "Love is a necessity for humanity," human turns concentrate on Universalism and Benevolence while R-Debater activates care, pleasure, conformity, achievement, security, autonomy, and stimulation within the same motion.

Methodology in Plain English

The researchers took real competitive debate transcripts in Chinese from the ORCHID corpus (1,218 debates, 476 topics, 2,436 stance-specific summaries, 14,133 annotated utterances) and sampled 60 topics. For each topic, one real human debate served as the reference, and the same topic was given to five generation systems — GPT-5.4, Claude-3.6 Sonnet, Qwen3.6-Plus, DeepSeek-V3.2, and the debate-specific R-Debater — producing debates matched on topic, stance setting, language, turn structure, and approximate length.

Each transcript was broken into turns, and a "Stance-Aware Premise Extraction Agent" pulled out verbatim premise spans conditioned on the topic, stance, and stance-specific claim. Each premise was paired with that claim to form an argumentative unit. A second agent then scored every unit with a soft distribution over Schwartz's ten basic values (Self-Direction, Stimulation, Hedonism, Achievement, Power, Security, Conformity, Tradition, Benevolence, Universalism), averaged over repeated runs and normalized. These unit-level distributions were averaged up to turn, stance, and whole-debate levels.

Metrics were then computed on those distributions: how many values clear a threshold (coverage), how much the top-k values agree with the human reference (overlap), how far the full distributions diverge (JSD), plus entropy at each level for concentration and stability, drift, and persistence for cross-turn behavior. Because debates are paired by topic, the unit of analysis is the within-topic difference between an AI debate and its human reference, tested with Wilcoxon signed-rank tests, Holm correction, and Cohen's d_z — then re-tested with regressions controlling for length, argumentative-unit count, lexical diversity, BLEU similarity, and Self-BLEU repetition, with system and motion-type (fact, value, policy) fixed effects. Finally, 120 debate judges rated debates source-blind on 7-point scales and made pairwise comparisons on which debate better maintains a focused value core.

Why This Matters

Impact on research. The paper reframes debate evaluation: presence of values is not the same as organization of values. It gives the argument-mining and value-modeling communities a shared, theory-grounded measurement layer that connects two literatures that had been running in parallel — debate generation evaluation and value identification in argumentative text — and shows that aggregate metrics can hide turn-level effects.

Real-world applications:

• Auditing AI debate and persuasion systems for how they allocate value appeals, not just whether they sound persuasive.
• Debate training and coaching tools that show speakers where their value core drifts or diffuses across a round.
• Argument quality assessment in education, distinguishing clear value framing from scattered value invocation in student writing.
• Content review and red-teaming of generated persuasive text, where value framing choices are the thing being examined.

Industry relevance. Teams building debate, persuasion, or advocacy agents can use these metrics as a diagnostic layer alongside existing quality checks. The finding that clearer value expression does not imply better value focus is directly actionable for product evaluation: a model can score well on clarity while still organizing its appeals very differently from a human debater.

Future Directions

• Beyond the Basic-10 space. The authors note their framework cannot capture every culturally specific, ideological, religious, rhetorical, or domain-specific value; other value frameworks may reveal complementary patterns.
• Beyond Chinese, fixed-turn debates. The benchmark is Chinese and uses a fixed-turn matched design that simplifies real interaction — interruptions, audience feedback, identity cues, spontaneous strategic shifts. The cross-lingual analysis in the appendix is described as exploratory.
• Reducing annotation error on hard cases. Implicit, metaphorical, and culturally specific value appeals may still be misread by the automatic premise and value agents.
• Turning descriptive metrics into evaluative ones. Entropy, stability, drift, and persistence are diagnostic rather than quality scores — high entropy could mean diffusion or could mean inclusive, multi-perspective argumentation. Establishing when a given value pattern is rhetorically desirable remains open.

Target Audience

Researchers and practitioners in NLP evaluation, argument mining, and computational social science; builders of debate, persuasion, or advocacy agents who need richer diagnostics than coherence and persuasiveness scores; and argumentation scholars interested in operationalizing value-based argumentation theory at scale. Readers without an NLP background can follow the conceptual distinction and the headline results, but the metric definitions and statistical analysis reward some familiarity with distribution comparison and paired testing.

Authors’ abstract

Current evaluation of LLM-generated debate largely focuses on argumentative quality, leaving the value structure of generated arguments underexamined. We introduce a diagnostic metric framework for comparing how values are expressed and organized in human-reference and AI-generated debates. The framework separates value expression, the range of recognizable values a debate invokes, from value organization, how those values are distributed and sustained across turns. Grounded in Schwartz's Basic-10 value theory, it represents debates as multi-level value distributions over argumentative units, utterances, stances, and full debates, enabling metrics for value coverage, alignment, diffusion, and cross-turn dynamics. Across multiple generation systems, AI-generated debates are broadly value-expressive, often activating as many or more value dimensions than matched human references, but they organize these values differently. Their utterances are more locally value-diffuse, and source-blind judges distinguish value clarity from value focus. These findings show that debate evaluation should consider not only value presence, but also value organization. Code and supplementary materials are available at https://github.com/Maoyuan-li/Value-Expression-and-Organization.