ParliaBench: An Evaluation and Benchmarking Framework for LLM-Generated Parliamentary Speech

AI summary

Overview

Research area: Natural Language Processing — LLM evaluation, benchmark construction, computationally grounded political text generation.

Technical level: Intermediate. The paper combines standard NLP generation metrics, sentence-embedding geometry, parameter-efficient fine-tuning (QLoRA), LLM-as-a-judge grading, and statistical significance testing, but it presents each component in accessible terms.

Scope: The paper builds and releases a UK Parliament speech dataset, a multi-dimensional evaluation framework with two new embedding-based political authenticity metrics, and baseline benchmark results from five fine-tuned LLMs generating 27,560 evaluated speeches.

What This Paper Is About

Generating realistic parliamentary speech is harder than ordinary text generation, because a speech must not only read well but also sound like it comes from a specific party and sits at a specific point on the political spectrum. Existing NLP metrics measure surface similarity, fluency, or semantic closeness to a reference, but they do not measure political authenticity — whether a "Labour" speech actually sounds Labour rather than Conservative. The authors address this by creating ParliaBench: a curated UK parliamentary dataset, an evaluation framework spanning linguistic quality, semantic coherence and political authenticity, and two novel embedding-based metrics intended to quantify ideological positioning.

Key Contributions

1. A curated parliamentary speech dataset. A four-step processing pipeline applied to the UK subset of the ParlaMint corpus (raw version) produced a final training dataset of 447,778 speeches from 1,901 unique speakers across 11 political affiliations, enriched with speaker identity, party, political orientation, chamber, session date, parliamentary section, and EuroVoc topic classification.

2. A multi-dimensional evaluation framework. The framework combines computational metrics (PPL, Dist-N, Self-BLEU, GRUEN, BERTScore, MoverScore) with LLM-as-a-judge scoring across six dimensions (J_Coh, J_Conc, J_Rel, J_Auth, J_PolApp, J_Qual), organized into three assessment categories plus a cross-context consistency layer.

3. Two novel political authenticity metrics. Political Spectrum Alignment (PSA) measures alignment with an expected left–right orientation using orientation centroid embeddings and a normalized ideological distance term; Party Alignment (Party Align) measures cosine similarity between a generated speech and a party-specific centroid.

4. Baseline benchmarks from five fine-tuned LLMs. Five architectures were fine-tuned with QLoRA and used to generate speeches, yielding a released generated dataset of 27,560 fully evaluated speeches (29,220 passed the generation-validity filter; 30,000 were planned) and baseline performance numbers for all metrics.

Main Findings

• Fine-tuning broadly improves quality, but not universally. Pairwise t-tests confirmed statistical significance in 45 of 70 metric comparisons (p < 0.05). Llama and YI showed the widest gains, with improvements marked as significant on 11 of 14 metrics (79%) in Table 2.

• Fine-tuned models also became more consistent. Cross-context stability analysis (Stability = 100 / (1 + CV)) yielded composite stability scores of 91.4–96.2 for fine-tuned models. Mistral had the highest consistency (96.2), while Llama combined strong performance with stability (95.1).

• The new political authenticity metrics responded strongly to fine-tuning. All five models significantly improved on PSA (p < 0.001) with effect sizes from d = 0.141 to d = 1.045. Party Align improved for 4 of 5 models, with effect sizes d = 0.099 to d = 1.221.

• The two novel metrics are discriminative. Party Align distinguishes parties and PSA distinguishes political orientations, both at p < 0.001, which the authors present as evidence that these dimensions are not captured by conventional NLP metrics.

• Some architectures showed quality trade-offs. Gemma 2 9B's fine-tuned perplexity was higher than its baseline (102.382 ± 60.592 vs 89.783 ± 47.619) and its judge-assessed relevance fell (7.502 vs 7.782), while several other metrics improved significantly (BERTScore 0.497 vs 0.443; MoverScore 0.589 vs 0.542; J_Auth 4.215 vs 3.842; J_PolApp 7.314 vs 6.513; J_Qual 4.976 vs 4.469). Qwen2 7B also showed category-level trade-offs; Mistral 7B v0.3's J_Auth (3.986 vs 4.234) and J_Qual (3.745 vs 4.182) did not improve significantly.

• Largest single perplexity gain. Llama 3.1 8B dropped from 61.150 ± 35.890 (baseline) to 31.623 ± 8.251 (fine-tuned), the paper's headline illustration of fine-tuning impact.

• Party-level performance tracks data abundance and coherence, not party size alone. Conservative (58.9% of training data) and Labour (24.3%) achieved stable performance across models, while minor parties varied more. Mistral struggled with Non-Affiliated (0.436) and Qwen2 excelled with Bishops (0.664); YI was robust across parties (0.614–0.633).

• Party difficulty was uniform. Party difficulty scores ranged narrowly from 0.382 to 0.456 with no statistically significant differences between parties.

• Topic performance varied. Science scored lowest on average (0.516); Economics (0.610) and European Union (0.606) scored highest. Science and Geography were the most difficult topics, while Finance, Business and Economics were the least difficult.

• Ideological coverage is skewed and correlates with performance. Centrist positions (Centre-left 0.607, Centre-right 0.551) dominate the dataset at 88% and scored higher. Gemma and Qwen achieved their highest scores on Right positions, and Mistral underperformed consistently, which the authors attribute to architecture rather than ideology.

Methodology in Plain English

Building the dataset. The authors started with the raw UK portion of ParlaMint and ran it through four steps: parsing the XML to isolate individual speeches, attaching metadata (speaker, party, chamber, date) and aligning each speech date with the speaker's affiliation history at that moment, filtering out procedural noise and keeping only substantive content, and assigning topic labels. For topic labelling they converted ParlaMint's CAP categories into EuroVoc categories, using direct semantic mapping where the categories matched cleanly (reported as 16 categories) and automated classification from Bocchi et al. (2024) where they did not (reported as 6 categories). Parties with fewer than 1,000 speeches were dropped, taking 28 affiliations down to 11.

Choosing models and fine-tuning. Five architectures were chosen to represent different design philosophies: Mistral 7B v0.3, Llama 3.1 8B, Gemma 2 9B, Qwen2 7B and YI 6B. Each was fine-tuned with QLoRA (r = 16, α = 16, 2 epochs) using SFTTrainer from TRL on an 80%–20% train-test split, with each model's own chat template framing political metadata as input.

Generating the evaluation set. The plan was 30,000 speeches (3,000 per model-type combination across 10 model-type combinations), sampled from the held-out test set with 90% generic instruction prompts carrying political context and 10% specific parliamentary questions. Generation used nucleus sampling (temperature = 0.7, top-p = 0.85, repetition penalty = 1.2), with invalid outputs regenerated up to 3 attempts. Keeping only speeches produced successfully by all 10 model-type combinations left 29,220, and removing speeches with incomplete judge ratings left 27,560 fully evaluated speeches. The same prompts were used for every model so comparisons are like-for-like.

Scoring. Computational metrics are deterministic: perplexity against GPT-2 as a fixed reference (lower is better), Dist-N for lexical diversity, Self-BLEU for output variety, GRUEN for grammaticality and coherence, and BERTScore and MoverScore against the top-5 human speeches matching the same context. Judge metrics used Flow-Judge-v0.1, a 3.8B-parameter evaluation model whose architecture derives from Phi-3.5-mini instruct — chosen so the judge shares no architecture or training data with the evaluated models — scoring each speech 1–10 on six dimensions in batches of 32. All metrics, computational and judged, were normalized to a 0–1 scale so that consistency could be compared across them.

Measuring political authenticity. Both new metrics represent each political group as a centroid embedding computed with sentence transformers. PSA finds the closest orientation centroid by cosine similarity, then multiplies that similarity by a distance penalty: max(0, 1 − Δφ/12), where orientations are mapped to numbers from Far-left = −6 through Centre = 0 to Far-right = +6, so the maximum possible distance is 12 and the score lands between 0 and 1. Parties with unknown orientation are excluded. Party Align is simply the cosine similarity between a generated speech and its expected party centroid.

Why This Matters

Parliamentary debate is a domain where sounding wrong is as damaging as being incoherent, and no existing benchmark tested that. The paper argues it is the first benchmark resource addressing the specialized challenge of parliamentary speech generation, combining data, metrics, and baselines. Its novel metrics give researchers a measurable handle on ideological positioning rather than relying on subjective reading, and the released generated-speech dataset plus fine-tuned models make results reproducible.

Real-world applications the framework could support:

• Computational political science — quantifying how closely a real or synthetic speech matches party positions and the left–right spectrum over time, including around events like Brexit or the COVID-19 pandemic response covered in the dataset.
• Synthetic data generation for research — producing large volumes of party- and topic-labelled political text with measurable authenticity, for studying framing and rhetoric without exposing real speakers.
• Model auditing and bias assessment — detecting whether a model systematically drifts toward particular parties or orientations when asked to speak as others, using Cross-Context Stability and the alignment metrics.
• Tooling for civic technology and education — illustrating parliamentary style and conventions in training or simulation settings, subject to the paper's stated requirement that outputs be labelled AI-generated.

Industry relevance: the benchmark offers a reproducible evaluation harness for anyone fine-tuning or serving models in political, governmental, or news domains where ideological consistency and party-appropriate tone matter, and its two embedding metrics are cheap, deterministic, and easy to fold into existing evaluation pipelines.

Future Directions

• Multilingual extension to European parliamentary systems, which the authors list as a first priority.
• Human evaluation protocols to validate the automated judgments and the new metrics against expert readers — a gap the paper explicitly names in its limitations.
• Systematic assessment of political bias and perspective maintenance across viewpoints.
• Targeted data collection for under-represented parties, since Party Align performance tracked data abundance and ideological coherence, leaving open whether more data would close the gap for minor parties.
• Closing the evaluation gaps the paper does not cover: argument structure and factual verification against parliamentary records, plus the LLM-as-a-judge biases the authors acknowledge from using a single judge model.

Target Audience

Researchers in computational linguistics and NLP benchmark design; computational political scientists studying parliamentary discourse, ideology, and party rhetoric; and machine learning engineers building or auditing LLMs for political, governmental, or public-affairs applications. It is also directly useful to practitioners who need a ready-made evaluation protocol for domain-specific fine-tuning, since the paper supplies datasets, fine-tuned models, metric definitions, and full prompts under a CC BY license on Hugging Face, with resources on a project website and GitHub. Readers who want argument-level or factuality evaluation, human validation, or deployment guidance will need to look elsewhere: the authors state the work is intended for research and educational purposes, not for deployment in actual democratic processes.

Authors’ abstract

Parliamentary speech generation presents specific challenges for large language models beyond standard text generation tasks. Unlike general text generation, parliamentary speeches require not only linguistic quality but also political authenticity and ideological consistency. Current language models lack specialized training for parliamentary contexts, and existing evaluation methods focus on standard NLP metrics rather than political authenticity. To address this, we present ParliaBench, a benchmark for parliamentary speech generation. We constructed a dataset of speeches from UK Parliament to enable systematic model training. We introduce an evaluation framework combining computational metrics with LLM-as-a-judge assessments for measuring generation quality across three dimensions: linguistic quality, semantic coherence, and political authenticity. We propose two novel embedding-based metrics, Political Spectrum Alignment and Party Alignment, to quantify ideological positioning. We fine-tuned five large language models (LLMs), generated 28k speeches, and evaluated them using our framework, comparing baseline and fine-tuned models. Results show that fine-tuning produces statistically significant improvements across the majority of metrics and our novel metrics demonstrate strong discriminative power for political dimensions.