"As Eastern Powers, I will veto." : An Investigation of Nation-level Bias of Large Language Models in International Relations

AI summary

Overview

Research area: Natural Language Processing — bias evaluation of Large Language Models (LLMs) in the International Relations (IR) domain.

Technical level: Intermediate. The paper is readable for general NLP readers, but its three-test evaluation framework, the weighted-F1 metric, and the RAG-plus-Reflexion debiasing pipeline assume some familiarity with LLM evaluation and prompting.

Scope: This paper builds a United Nations Security Council (UNSC)-grounded dataset and a three-part evaluation framework to measure nation-level bias in LLMs, with a focus on the five permanent UNSC members (P5), and proposes a debiasing method combining Retrieval-Augmented Generation (RAG) with Reflexion-based self-reflection.

What This Paper Is About

Most research on bias in language models has examined demographic attributes such as gender or race, and very little work looks at bias directed at whole nations. The authors argue this gap matters because LLMs are increasingly proposed for simulations, decision support, and policy analysis in international relations. Their goal is to define and quantitatively probe nation-level bias — the discrepancy between a country's real-world characteristics or behavior and the judgments an LLM makes about that country — and then to test whether such bias can be reduced.

Key Contributions

1. A multi-faceted evaluation framework with three distinct tests for nation-level bias in the IR domain — a Direct Question-Answering (DirectQA) test, an Association Test (AT), and a persona-assigned Vote Simulation — accompanied by a publicly released dataset grounded in real UNSC records.

2. A comprehensive cross-model evaluation of nation-level bias, showing that bias varies by model and by task, i.e., that it is multidimensional — the same pattern previously observed in demographic-bias research.

3. A debiasing framework for the IR domain that integrates external knowledge (RAG over past resolutions and voting records) and Reflexion-based self-reflection, reducing nation-level bias and improving prediction performance for GPT-4o-mini and Llama-3.3-70B.

4. A released dataset and code (dataset under CC BY-NC 4.0; code at a public GitHub repository) built from UNSC resolutions from 2013 to 2024, comprising 515 adopted resolutions, 66 non-adopted resolutions, and associated meeting transcripts, plus a domain keyword pool of 41 keywords grouped into 7 thematic categories.

Main Findings

• General bias trends exist but vary by model: Across models, positive bias appeared toward the United Kingdom, France, and the United States, and negative bias toward Russia, while bias toward China varied. Llama appeared neutral toward Russia, unlike GPT.

• DirectQA — General Irresponsibility: GPT and Mistral produced the highest proportions of neutral responses, refraining from naming any country. The U.K. and France were least frequently labeled "irresponsible." Russia received the highest irresponsibility scores for Mistral and Qwen. The U.S. ranked first under Llama and second across the other models, while China's scores varied.

• DirectQA — Function-Specific: GPT and Mistral's robustness declined relative to the General test, though GPT still produced neutral answers most often. France and the U.K. occupied the lowest irresponsibility ranks (fourth and fifth) across all functions, and Russia was most frequently classified as "irresponsible" across all models. The U.S. recorded higher irresponsibility scores than China on most function-specific dimensions. Model- and topic-specific deviations appeared: GPT and Qwen ranked China second in the "Non-Military Measures Against an Aggressor" function, while the others ranked it third, and Llama ranked the U.S. above Russia in "Adjust Disputes, Recommend Settlement."

• DirectQA summary counts: Positive bias toward the U.K. and France and negative bias toward Russia and the U.S. were observed, with Russia and the U.S. scoring first or second 43 and 32 times respectively, out of 44 combinations. Qwen showed the most polarized distribution among the five nations, while Llama and Mistral displayed relatively balanced distributions across the U.S., Russia, and China, and GPT achieved the highest overall robustness.

• Association Test: The U.S., U.K., and France achieved average ATS values above zero across every model, with the U.K. and France generally below the U.S., while Russia and China registered negative ATS values in all cases. Except in the "Armament" and "International Law" categories, the U.S. attained the highest ATS in every remaining category regardless of model. Across all models, the U.S., U.K., and France showed positive bias in 26, 28, and 21 out of 28 combinations respectively, while Russia and China showed predominantly negative bias in 28 out of 28 combinations. GPT produced the most extreme span of ATS values, followed by Qwen; Llama and Mistral yielded relatively balanced association patterns.

• Vote Simulation — statistical comparison: All models cast "favour" votes for the U.S., U.K., and France more often than the ground truth. Behavior for Russia and China varied by model: GPT cast "against" votes more often than the ground truth, Qwen cast "favour" votes more often than the ground truth, Llama most closely matched Russia's actual record but still overvoted "favour" for China, and Mistral registered "abstention" more frequently than the ground truth for Russia and China. GPT exhibited a distinct polarity bias between Western nations (the U.S., U.K., France) and non-Western nations (Russia, China).

• Vote Simulation — weighted F1 (Table 2): GPT-4o-mini scored 60 (US), 43 (UK), 49 (FR), 41 (RU), 28 (CN); Llama-3.3 scored 54, 41, 49, 72, 50; Mistral-Small scored 44, 51, 56, 44, 38; Qwen-2.5 scored 48, 50, 52, 60, 59. GPT achieved its highest performance on the U.S. persona and performed worst on China; Llama and Qwen were stable across all five personas, with Llama achieving the highest weighted F1 score overall on the Russia persona; Mistral was strong for the U.K. and France but poor for the U.S., Russia, and China.

• Bias and performance are linked: GPT's least extreme statistical profile for the U.S. corresponded to the highest performance among the models, while GPT's dominant negative bias toward Russia was matched by its poorest performance on Russia — indicating that both positive and negative biases can degrade performance. Llama's simulation for Russia, which statistically aligned most closely with the true vote distribution, also attained the highest performance score across all models and nations.

• Reasoning models show reduced bias and better performance: The two reasoning-oriented models evaluated, o3-mini and DeepSeek-R1, achieved high performance across most personas compared to the basic LLMs, with DeepSeek-R1 achieving the highest scores for four of the five personas.

• Bias is multidimensional within a model: The DirectQA experiment revealed negative bias against the U.S. across all models, while the AT and implicit-bias experiments showed positive bias toward the U.S. for the same models. Qwen showed strong negative bias toward China in DirectQA and the AT but strong positive bias in the implicit-bias experiment.

• Debiasing results (Table 3, weighted F1 × 100): With RAG and Reflexion, GPT-4o-mini changed by −1 (US, to 59), +17 (UK, to 60), +3 (FR, to 52), +18 (RU, to 59), +16 (CN, to 44); Llama-3.3 changed by +2 (US, to 56), +6 (UK, to 47), −1 (FR, to 48), −18 (RU, to 54), +2 (CN, to 52); Mistral-Small changed by −5 (US, to 40), −5 (UK, to 46), −8 (FR, to 48), −7 (RU, to 37), +5 (CN, to 43); Qwen-2.5 changed by −1 (US, to 47), 0 (UK, 50), −4 (FR, to 48), −2 (RU, to 58), −7 (CN, to 52). The authors attribute the degradation for Mistral and Qwen to increased prompt length impairing comprehension, noting that prior studies show the GPT series performs better than Mistral and Qwen series in long-context settings.

• Experimental reliability: Temperature was fixed at 0 and each experiment was repeated three times. Inter-run agreement was assessed with Fleiss' kappa (κ > 0.40), multi-χ² tests (thresholds χ² < 15.507 and χ² < 9.488), and a Friedman χ² test (threshold χ² < 5.991). Following Landis and Koch (1977) interpretation guidelines, 90% of function-specific DirectQA tests, 100% of the testable AT, and 97% of Vote Simulation tests met their statistical criteria.

Methodology in Plain English

The authors first assembled a real-world dataset from the UN Digital Library covering 2013 to 2024: full texts of resolutions, voting outcomes and adoption statuses, and official statements by national representatives explaining their votes. They also extracted 41 frequently occurring keywords from the resolutions and grouped them into 7 thematic categories. They focused on the P5 (the U.S., U.K., France, Russia, and China), reasoning that permanent members generate a substantial and consistent record — P5 members have 66 voting records on non-adopted resolutions, whereas non-permanent members have only 10 on average.

They then probed bias in three ways. In the DirectQA test, the model is asked which of two P5 members is more irresponsible, both generally and in specific UNSC functions; each pairing is asked twice with the nation order reversed to counter positional bias, and a "win rate" style irresponsibility score records how often each country is chosen. In the Association Test, the model ranks the P5 for each UNSC keyword, offering a rationale from which the authors infer whether the association is positive or negative; country order is randomized. In the Vote Simulation, the model adopts the persona of a nation's representative and votes "favour," "against," or "abstention" on a resolution — only non-adopted resolutions are used, because a single "against" vote from any P5 is a veto, so adopted resolutions contain no recorded "against" votes from permanent members. Simulated votes are compared to real records both statistically and through a confusion-matrix weighted F1 score.

Four base models were chosen to represent their home states: GPT-4o-mini and Llama 3.3-70B (U.S.), Mistral-Small (France), and Qwen 2.5-72B (China). Two reasoning models, o3-mini and DeepSeek-R1, were added later to test whether reasoning ability mitigates bias.

Because no neutral national ground truth can be established for DirectQA or the Association Test — annotating one could itself be biased — the authors treat those tests as detecting pronounced and consistent skews across many prompts rather than measuring a precise deviation from truth. For Vote Simulation, the official UNSC voting records provide the factual benchmark.

The proposed debiasing framework works purely through prompt engineering, with no parameter tuning. A custom retriever finds thematically similar past resolutions from adopted and non-adopted pools; the model makes rehearsal votes on those past resolutions, then reflects on its guesses by comparing them with the actual votes and the accompanying speech by the nation's representative; the accumulating history of rehearsal votes and reflections is finally supplied alongside the target resolution for the final vote.

Why This Matters

Impact on research: The paper extends bias evaluation from individual demographics to nations, and shows that nation-level bias is multidimensional — the direction and magnitude of bias toward a country shifts with the model and the evaluation context. This implies bias tests must be tailored to the downstream task, and that reporting prediction performance alone is not enough when deploying LLMs in IR.

Real-world applications:

• Diplomatic simulation games and crisis-escalation exercises that use LLM agents to role-play states.
• Decision-support and policy-analysis tools that summarize or predict positions in multilateral bodies.
• Benchmarks and audits for AI systems used in government and international-organization settings.
• Human-in-the-loop review pipelines where domain experts vet LLM outputs before they inform any position.

Industry relevance: Any organization deploying LLMs for geopolitical analysis, forecasting, or negotiation support needs to know that measured accuracy can coexist with systematic favoritism, and that favoring a country is not automatically safer than disfavoring it — both were associated with degraded performance in this study. The debiasing framework is also attractive practically because it improves outputs through prompt engineering alone, without retraining.

Future Directions

• Establishing and justifying a reasonable "unbiased" status for each nation, which the authors state is currently infeasible for the DirectQA and Association Tests and leaves quantitative bias measurement in those tests impossible.
• Improving the debiasing framework so it does not degrade performance for models such as Mistral and Qwen, where the authors suspect increased prompt length exceeds long-context comprehension capacity.
• Extending the analysis beyond the P5 to other UNSC members and other international institutions, given the data imbalance the authors cite as justification for focusing on permanent members.
• Determining whether the bias–performance relationship generalizes across more reasoning models and tasks, given that DeepSeek-R1 reached the highest scores on four of five personas.

Target Audience

This paper is most useful to NLP researchers working on bias, fairness, and evaluation; AI safety and alignment practitioners concerned with geopolitical applications; and policy analysts, diplomats, or international-organization staff who are considering LLM-based tools for IR tasks. Readers interested in retrieval-augmented generation and self-reflection prompting will also find the debiasing pipeline relevant, though the paper is best read alongside its appendix for implementation details.

Authors’ abstract

This paper systematically examines nation-level biases exhibited by Large Language Models (LLMs) within the domain of International Relations (IR). Leveraging historical records from the United Nations Security Council (UNSC), we developed a bias evaluation framework comprising three distinct tests to explore nation-level bias in various LLMs, with a particular focus on the five permanent members of the UNSC. Experimental results show that, even with the general bias patterns across models (e.g., favorable biases toward the western nations, and unfavorable biases toward Russia), these still vary based on the LLM. Notably, even within the same LLM, the direction and magnitude of bias for a nation change depending on the evaluation context. This observation suggests that LLM biases are fundamentally multidimensional, varying across models and tasks. We also observe that models with stronger reasoning abilities show reduced bias and better performance. Building on this finding, we introduce a debiasing framework that improves LLMs' factual reasoning combining Retrieval-Augmented Generation with Reflexion-based self-reflection techniques. Experiments show it effectively reduces nation-level bias, and improves performance, particularly in GPT-4o-mini and LLama-3.3-70B. Our findings emphasize the need to assess nation-level bias alongside performance when applying LLMs in the IR domain.