TradeVerse: A Longitudinal Benchmark of Political Negotiation in International Trade

AI summary

Overview

Research area: Natural Language Processing / benchmark construction for large language models, at the intersection of computational political science and computational trade policy.

Technical level: Intermediate. The reader needs familiarity with LLM evaluation (precision/recall/F1, BLEU, ROUGE, BERTScore) and with benchmarking conventions, but no background in economics or trade law.

Scope (1 sentence): The paper introduces TradeVerse, a benchmark of 1,170 longitudinal World Trade Organization (WTO) trade concerns reconstructed from 6,933 meeting records, and evaluates six contemporary LLMs on three annotation-free tasks: HS chapter prediction, respondent identification under anonymization, and final-statement generation.

What This Paper Is About

Existing benchmarks for political and institutional texts evaluate LLMs on isolated documents or single decision points, whereas real negotiations are longitudinal: parties argue over many rounds, sometimes for years, and each turn depends on everything said before it. The authors build a benchmark from WTO Specific Trade Concerns, where member states challenge one another's trade measures repeatedly, so that a model must track the evolution of a dispute across meetings rather than read one document. The goal is to test whether current LLMs can reason over the temporal, institutional, and strategic structure of international trade negotiations.

Key Contributions

1. A longitudinal benchmark of authentic multi-party WTO negotiations. TradeVerse spans five WTO groups and more than three decades of proceedings (1995–2026), reconstructed from 1,170 trade concerns and 6,933 meeting records, with 26,219 statements and 68 distinct responding members.

2. A structured representation of each concern as a sequence of interventions by raiser, supporter, and respondent members, with only concerns discussed in at least two rounds retained (mean 5.9 meetings per concern, maximum 52).

3. Three annotation-free tasks — HS chapter prediction, respondent identification, and final-statement generation — whose ground truth is recovered directly from official WTO proceedings, requiring no manual annotation.

4. An evaluation of six contemporary LLMs that surfaces systematic limitations: over-prediction of product categories (high recall, low precision), fluent but lexically and semantically imprecise diplomatic generation, and a respondent-identification disparity favouring Western members that persists under full anonymization.

Main Findings

• Respondent identification is accurate overall but uneven by bloc. Under partial masking (ST-1), accuracy ranges from 82.63% (GPT-OSS-120B) to 92.86% (GLM-5.2); under full masking of all country identities (ST-2), it ranges from 80.92% (GPT-OSS-120B) to 93.21% (GLM-5.2). Accuracy is nearly identical across the two settings, and in two cases (GLM-5.2, Kimi-K2.7-Code) marginally higher when all names are hidden, indicating that models infer the respondent from the substance of the dispute rather than from surrounding participant identities.

• A consistent Western vs. non-Western gap. In the main analysis, every model identifies Western respondents more accurately than non-Western ones at p < 0.01, with gaps (Δ) ranging from 6.51 to 16.68 percentage points; for several models the gap widens when all identities are masked. In the appendix ablation that removes all cases where the EU is the respondent (leaving 942 concerns), the gap persists at roughly 5 to 14 percentage points and is significant at p < 0.05 in all but one of the twelve model–setting combinations. The authors attribute this tentatively to differing amounts of training exposure to Western-bloc countries, and state that investigating the asymmetry is beyond the scope of the paper.

• HS chapter prediction shows high recall, low precision. Across all six models, recall substantially exceeds precision, meaning models recover relevant chapters but over-generate plausible neighbouring categories. Micro F1 ranges from 47.36 (Llama-3.3:70B-Instruct) to 65.63 (GLM-5.2); macro F1 is lower, ranging from 15.68 (Llama-3.3:70B-Instruct) to 46.55 (GLM-5.2). DeepSeek-V4-Pro has the highest micro precision (62.52) but a much lower micro recall (63.63) and macro F1 (25.89).

• Generated statements are fluent but non-specific. Task 3 results are led by GLM-5.2 across lexical and semantic metrics (BLEU-4 11.31, ROUGE-L 23.00, BERTScore-F1 60.49); GPT-OSS-120B is weakest (BLEU-4 2.91, ROUGE-L 14.31, BERTScore-F1 53.23). All models overshoot the average ground-truth word count of 167.26, with GLM-5.2 the most conservative at 177.78. The paper notes that generated statements are diplomatically appropriate yet share little specific content with the interventions actually made.

• Longer negotiation histories improve generation. The number of rounds correlates positively with BERTScore-F1 for all models; Spearman ρ is statistically significant at the 1% level for five models (0.205 to 0.314) and at the 5% level for GPT-OSS-120B (ρ = 0.070, p = 0.02).

• Prior history matters for five of six models. A matched-pair ablation withholding all prior meetings shows significant BERTScore-F1 gains from full history for GLM-5.2, DeepSeek-V4-Pro, Llama-3.3-70B-Instruct, Nemotron-3-Ultra-550B, and Kimi-K2.7-Code (Cohen's d between 0.32 and 0.75, Wilcoxon p < 0.001). GPT-OSS-120B shows no measurable effect (95% CI includes zero, d = 0.05, p = 0.31); the authors cautiously link this to that model's weaker general performance rather than to a deficiency in the data's longitudinal structure, and flag it as a hypothesis rather than a settled conclusion.

• Corpus composition. The five groups are TBT (538), SPS (397), CTG (133), CMA (68), and IL (34); TBT and SPS together account for 79% of concerns and 73% of statements. The EU has the highest number of responses (224 in the main text; 228 reported in the appendix), followed by India (118). The HS system has 97 principal chapters globally, of which the corpus covers 89, dominated by agricultural foods and products followed by electronic items. Concerns have a mean span of 2.5 years, a maximum span of 20 years, and a few exceeding 15 years. Thread lengths are heavy-tailed: median 1,562 words, mean over 4,500, longest 108,511 words, with 12.7% exceeding 8,000 words and the top percentile exceeding 50,000 words.

Methodology in Plain English

The authors scraped the public WTO Trade Concerns Database, which records concerns raised by member states against other members. Data collection proceeded in two phases: first they gathered structured metadata (meeting IDs, committee, member countries and their roles, number of rounds), and then they downloaded the meeting minutes documents, which are Word files with country names written in different colours. They parsed these documents to separate countries, roles, and statements. To preserve the longitudinal character of the data, they discarded any concern with fewer than two meetings.

Each concern is represented as a sequence of rounds; each round contains statements from a raiser, optionally one or more supporters, and a respondent. Three tasks are then defined over these sequences. Task 1 asks the model to output the set of HS chapters under discussion given the full history — a multi-label problem, since a concern may involve multiple products. Task 2 asks the model to name the responding member from a transcript with identities masked, under two settings: only the respondent masked (ST-1) and all countries masked (ST-2). Task 3 asks the model to generate the respondent's concluding statement given all prior rounds plus the raiser and supporter statements of the final round, with the respondent's own statement withheld.

Six LLMs were run through the Together API in a zero-shot setting at temperature 0.0: Nemotron3-Ultra:550B, Llama-3.3:70B-Instruct, GLM-5.2, DeepSeek-V4-Pro, GPT-OSS-120B, and Kimi-K2.7-Code. Evaluation uses micro and macro precision, recall, and F1 for Task 1; accuracy for Task 2; and BLEU-4, ROUGE-L, and BERTScore for Task 3, with BERTScore computed on an A100 GPU using microsoft/deberta-large-mnli. Because only the TBT and SPS groups carry HS codes, Task 1 uses 724 of the 1,170 meetings; after removing concerns with missing respondent statements, Task 3 uses 1,101 of 1,170.

Why This Matters

Impact on research. TradeVerse is presented as the first benchmark to investigate the potential of LLMs in understanding longitudinal political trade negotiations. It differs from existing political benchmarks (UNBench, UNSC-Bench, European Parliament work) by requiring a model to follow the same dispute across successive meetings rather than answer a single static question, and from trade-focused resources such as TradeGov, which focuses on legal question answering over trade laws. The paper also contributes the first structured reconstruction of WTO trade concern proceedings as multi-party dialogues with rounds, product codes, and domains, which the authors argue is useful to computational social scientists studying dispute evolution, coalition formation, and diplomatic language.

Real-world applications (as raised by the paper):

• Negotiation research: studying how LLMs behave in negotiation is an active area, and TradeVerse offers an authentic real-world setting for it.
• Multi-agent simulation: the authors note it would be interesting to investigate multiple agents, each representing a different country and role, communicating with each other.
• Social science analysis: researchers can study how disputes evolve over time, how coalitions form, and patterns of participating countries at a scale hard to accumulate manually.
• Institutional NLP evaluation: the benchmark provides a setting for studying geopolitical bias and strategic language generation in authentic international discussion.

Industry relevance. The tasks map onto practical needs for reasoning over long, evolving institutional text — tracking which product categories are implicated in a dispute, inferring a party's identity or position from anonymized argumentation, and drafting contextually appropriate responses. The documented Western-bloc advantage in respondent identification is relevant to anyone deploying such models on international regulatory or policy content, since it indicates uneven performance across the membership even when explicit identity cues are removed.

Future Directions

• Explaining the geopolitical disparity. The authors state that investigating the asymmetry in model performance is beyond the scope of this research and reserve it for future work; one tentative explanation they offer is differing amounts of training data across countries.
• Testing the GPT-OSS-120B null result. The absence of a measurable benefit from prior history for one model is left as an open hypothesis, with architectural or training differences noted as alternatives that are not ruled out.
• Multi-agent negotiation. Moving from a single LLM performing a series of tasks to multiple agents representing different countries and roles communicating with one another.
• Expanding task scope. The benchmark restricts itself to HS chapter prediction rather than the full 97-chapter hierarchy with its sub-branches, and to a subset of the corpus for two of the three tasks, leaving room for broader coverage.

Target Audience

Researchers building or auditing LLM benchmarks for institutional, legal, and political text; computational social scientists studying trade disputes, coalition formation, and diplomatic language; practitioners applying LLMs to international regulatory or policy analysis who need to know where model performance is uneven; and NLP researchers interested in long-context reasoning and generation, where the corpus's heavy-tailed thread lengths (median 1,562 words, longest 108,511 words) make it directly relevant.

Authors’ abstract

LLMs are increasingly being applied to tasks involving institutional and political texts, but existing benchmarks evaluate them on isolated documents or single tasks. In realpolitik, negotiations are longitudinal data, where participating parties can align or argue over multiple iterations and each turn is an outcome of the previous turns, hence, understanding one turn requires tracking everything before it. We introduce TradeVerse, a benchmark built from the World Trade Organisation (WTO) specific trade concerns, where member states challenge one another and exchange arguments over multiple rounds, sometimes for years. We, in TradeVerse, reconstruct minutes of $1170$ meetings, spanning across 5 groups and $89$ product groups and define three tasks: first, the system has to analyze the longitudinal meeting records and predict the harmonized system codes (HS chapters) of the products under discussion in the particular meeting, second, we examine whether the system, upon analyzing the anonymized content of the meeting, can guess the name of the responding country and third, we ask the system to play the role of the responding country and provide the statement for the very last round. All labels are recovered directly from the proceedings, requiring no manual annotation. Our experiments highlight the challenges these tasks pose for current LLMs. To the best of our knowledge, TradeVerseis the first benchmark to investigate potential of LLMs in understanding longitudinal political trade negotiations.