SwissGov-RSD: A Human-annotated, Cross-lingual Benchmark for Token-level Recognition of Semantic Differences Between Related Documents

AI summary

Overview

Research area: Natural Language Processing — cross-lingual semantic similarity, token-level sequence labeling, and benchmark construction for evaluating large language models and encoder models.

Technical level: Intermediate. The task setup and evaluation metrics are explained clearly enough for readers with basic NLP background, while the fine-tuning and prompting configuration details will be most useful to practitioners.

Scope: This paper introduces SwissGov-RSD, a human-annotated, multi-parallel benchmark of naturally occurring semantic differences between English, German, French, and Italian government documents, and benchmarks 6 large language models plus 12 encoder models on it.

What This Paper Is About

When the same content is published in several languages, the versions are supposed to mean the same thing — but translations and unsynchronized updates mean they often do not. This paper collects such mismatches from the Swiss government portal admin.ch and has human annotators mark, token by token, exactly where the meanings diverge. It then tests how well current automatic systems can detect those differences, and finds that they perform far worse on this naturalistic data than on the synthetic benchmarks they were previously evaluated on.

Key Contributions

1. A new dataset. The authors construct and release SwissGov-RSD, described as the first human-annotated, document-level, cross-lingual dataset for token-level semantic difference recognition, covering three language pairs (English–German, English–French, English–Italian) drawn from naturally occurring content on admin.ch.

2. A broad benchmark of automatic approaches. They evaluate 6 LLMs and 12 encoder models using 3 different approaches (unsupervised alignment, few-shot prompting, and fine-tuning) under differing degrees of supervision, on both SwissGov-RSD and the synthetically constructed iSTS-RSD benchmark.

3. Evidence of a synthetic-to-realistic performance gap. They report a considerable gap between performance on the naturalistic and synthetic datasets, with differences in Spearman correlation of up to 78.

4. An analysis of where systems break down. They examine output-format failures and performance as a function of token position in long documents, showing that LLMs underlabel differences and degrade more than encoder models as inputs grow longer.

Main Findings

• Naturalistic data is much harder than synthetic data. Across all model types and language pairs, performance drops substantially when moving from iSTS-RSD to SwissGov-RSD. The fine-tuned GPT-4o-mini model illustrates this most starkly: it reaches the highest scores on iSTS-RSD (81.6 EN–DE, 79.9 EN–FR, 78.2 EN–IT) but falls to among the lowest on SwissGov-RSD (3.9, 1.1, and 5.3 respectively), suggesting overfitting to the data augmentation patterns.

• Unsupervised approaches are relatively robust. On SwissGov-RSD, the unsupervised methods outperform all few-shot and fine-tuned models. The highest unsupervised SwissGov-RSD scores include LaBSE at 20.6 (EN–DE) and 25.6 (EN–IT), Qwen3-Embedding 4B at 20.1 (EN–DE) and 20.3 (EN–IT), and Qwen3-Embedding 8B at 12.1 (EN–FR). The authors suggest specialized models may suffer from out-of-domain effects, since SwissGov-RSD differs from iSTS-RSD in naturalness, domain, length, and label distribution.

• Encoder models remain competitive. Among long-context models, Qwen3-Embedding achieves the strongest results, followed by EuroBERT 610M. LaBSE and XLM-R + SimCSE score highest in the unsupervised setting, but their limited input length requires document segmentation — and the authors note that segmentation effectively reduces the task to shorter units and avoids long-context challenges, which may itself explain the higher scores.

• DiffAlign with ModernBERT underperforms, but fine-tuning fixes it. The unsupervised DiffAlign approach works poorly with ModernBERT, likely because of its English-centricity. Once fine-tuned, however, it surpasses LLMs used with few-shot prompting in the naturalistic setting while being more efficient in time and computation.

• LLMs are highly limited on this task. For iSTS-RSD, fine-tuned GPT-4o-mini performs best, while few-shot prompted LLMs fail to beat the unsupervised baseline; reasoning models bring only little improvement. Proprietary models generally outperform open-weight alternatives. LLMs also produce formatting errors — in particular, sequence-length mismatches — at much higher rates on SwissGov-RSD than on iSTS-RSD. On SwissGov-RSD, Llama-3.1 405B Instruct failed to produce the correct number of labels for 7.00% of samples and fine-tuned GPT-4o-mini for 7.40%, compared with 0.07% for each on iSTS-RSD.

• Long documents expose weaknesses. Analyzing correlation by token position shows a slight downward trend for all systems as documents grow longer, but encoder-based systems — specifically unsupervised DiffAlign — appear more robust than LLMs. Llama-3.1 405B is the most sensitive, often failing to produce complete output for inputs exceeding 1200 tokens. LLMs also tend to underlabel semantic differences, most noticeably toward document ends or in the second input document.

• The dataset is more skewed than prior benchmarks. Roughly 10–17% of tokens are labeled as semantically different, against approximately 25–30% in the iSTS-RSD dataset.

• Inter-annotator agreement is limited but consistent with prior work. Token-level Spearman correlation between annotators ranges from 43.61 (EN–FR) to 65.09 (EN–IT), with EN–DE at 56.14. The authors note these values are consistent with agreement reported for other cross-lingual, pairwise span-level annotation tasks. One below-average EN–DE annotation run was replaced, raising Spearman correlation from 9.4 to 56.14.

Methodology in Plain English

Building the dataset. The authors crawled admin.ch and six of its subdomains, keeping only pages that existed in English plus German, French, and Italian and that contained at least three sentences of natural language in the right language. Romansh was excluded due to limited availability. After filtering, 235 multi-parallel documents remained. Each language pair was assigned two undergraduate or bachelor's students in computational linguistics who were native or highly proficient speakers. Annotators marked spans of text where meaning differs, using a five-point scale where 1 means minimal difference and 5 means complete semantic dissimilarity, with unannotated text treated as semantically equivalent (label 0). Annotation ran in two phases: a trial phase on three shared text pairs with automated and manual feedback, then a main phase in which each annotator handled half the pairs plus 25 overlapping samples used for validation. Annotators were paid approximately $35 per hour and spent between 20 and 35 hours on the task. After removing 11 documents flagged as faulty, the final dataset contains 224 multi-parallel documents totaling 886 texts.

Testing systems. Three families of approaches were compared. The unsupervised baseline, DiffAlign, encodes each document separately with a pretrained encoder and scores each token by how poorly it aligns to the other document (1 minus the maximum cosine similarity to any token in the other text) — it needs no labeled data. Few-shot prompting gives an LLM a natural-language instruction, a JSON output format, and three in-context examples, and asks it to label every token on a 0-to-5 scale. Fine-tuning applies either supervised fine-tuning for LLMs or a linear regression layer over concatenated encoded sequences for encoder models, trained with binary cross-entropy.

Training data. Fine-tuning used the iSTS-RSD training splits (based on human-annotated data from SemEval-2016 Task 2) plus 1500 human-validated paraphrases from PAWS as negative examples, augmented into synthetic "documents" following Vamvas and Sennrich (2023). Because iSTS-RSD contains labels only in English, the authors extended it to English–Italian using DeepL and used GPT-4o-mini for direct LLM-based label projection onto the German, French, and Italian sides. They generated 560 augmentations for LLM training and 10,000 for encoder training.

Evaluation. Systems were scored with token-level Spearman correlation between gold labels and predictions, with Kendall tau-b reported as a check showing no substantial differences. iSTS-RSD evaluation was restricted to 100 samples per difficulty setting and language to reduce computational cost. Malformed LLM outputs were handled by extracting JSON heuristically and padding or truncating to the gold sequence length; out-of-range labels were left unaltered.

Why This Matters

Research impact. The paper argues that synthetically augmented benchmarks can mask a failure to generalize — strong scores on constructed data do not translate to realistic, naturally occurring differences. It provides the community with a naturalistic, human-annotated, document-level, cross-lingual test set that fills a gap left by sentence-level NLI and STS datasets, and by comparable corpora such as SwissAdmin, the Bulletin Corpus, and 20min-XD that lack token-level semantic annotations. It also points toward limitations of synthetic benchmarking in general.

Real-world applications.

• Auditing multilingual public-sector and corporate websites to confirm that language versions actually convey equivalent content.
• Catching unsynchronized content updates, where one language version is revised and the others are not.
• Machine translation evaluation and quality estimation, where the same token-level difference signal is used to localize errors.
• Content alignment and generation evaluation pipelines that need fine-grained comparisons rather than a single similarity score.

Industry relevance. Organizations publishing regulated or safety-relevant information across languages — governments, healthcare, financial services, and any company with multilingual product documentation — can use this task to detect discrepancies that would otherwise go unnoticed. The finding that fine-tuned encoder models can beat LLMs at lower computational cost is directly relevant to teams choosing where to spend inference budget.

Future Directions

• Extending beyond high-resource, Latin-script, English-centric pairs. The authors note that generalizability to typologically distant languages and non-Latin scripts remains untested in a realistic setting, and that the dataset's multi-parallel nature would allow annotation of other language combinations — at high annotation cost.
• Improving annotation reliability. The limitations section argues that multiple annotators per document pair across all samples would be ideal, but resource constraints forced a two-annotator setup with overlap-based quality checks.
• Developing specialized approaches for cross-lingual document-level RSD. The conclusion calls for new methods that align better with human judgment and are applicable in real-world scenarios.
• Preserving structured output stability for long inputs. For LLMs, the authors suggest future work should investigate strategies that maintain structured sequence-labeling output quality at extended input lengths, and that unsupervised encoders or in-domain fine-tuning may be promising for long documents.

Target Audience

Researchers and practitioners working on cross-lingual semantic similarity, token-level sequence labeling, machine translation evaluation, and multilingual content alignment. It is also relevant to NLP benchmark designers interested in the gap between synthetic and naturalistic evaluation, and to engineers building multilingual publishing or documentation pipelines who need to detect divergences between language versions. The supervised fine-tuning setups and prompting configurations make the experimental section directly useful to model practitioners.

Authors’ abstract

Recognizing semantic differences across documents is crucial for text generation evaluation and content alignment, especially in cross-lingual settings. However, as a standalone task, it has received little attention. We address this by introducing SwissGov-RSD, the first naturalistic, document-level, cross-lingual dataset for semantic difference recognition. It encompasses a total of 224 multi-parallel documents in English--German, English--French, and English--Italian with token-level difference annotations by human annotators. We evaluate a variety of open-source and closed-source large language models as well as encoder models across different fine-tuning settings on this new benchmark. Our results show that current automatic approaches perform poorly compared to their performance on monolingual, sentence-level, and synthetic benchmarks, revealing a considerable gap for both LLMs and encoder models. We make our code and dataset publicly available.