MultiGhostBench: A Multilingual Benchmark for Long-Form LLM-Generated Text Attribution under Distribution Shifts

AI summary

Overview

Research area: Natural Language Processing, specifically authorship attribution (AA) of LLM-generated text, multilingual benchmarking, and out-of-distribution (OOD) generalisation.

Technical level: Intermediate. Readers should be comfortable with classification metrics (macro-F1), Transformer encoders, token-level statistics, and the notion of distribution shift.

One-sentence scope: The paper introduces MultiGhostBench, a benchmark of 928 long-form, LLM-generated books in six languages and three scripts, and uses it to test whether existing authorship attribution methods can identify the generating LLM under domain, author, and language shifts.

What This Paper Is About

Most work on LLM-generated text asks a binary question: human or machine? A harder question is authorship attribution, meaning which specific LLM wrote a given text. Existing benchmarks for this task are mostly English-only, built on short texts, use relatively outdated models, or evaluate only a narrow set of conditions. MultiGhostBench addresses this gap by providing long-form multilingual books written by five recent LLMs and evaluating attribution methods under three distinct distribution shifts: unseen genres (OOD-Domain), unseen generators (OOD-Author), and unseen languages (OOD-Language).

Key Contributions

1. A new multilingual long-form benchmark. MultiGhostBench contains 928 books averaging 59.6K words, generated by five recent LLMs (gemini-pro, gemini-flash, deepseek-v3.2, qwen3-235b, and gpt-oss) across six languages (Italian, Spanish, German, English, Chinese, Russian) from four language families and three scripts (Latin, Chinese characters, Cyrillic).

2. Three OOD evaluation dimensions in one benchmark. The authors combine OOD-Domain (unseen genres from Project Gutenberg), OOD-Author (leave-one-generator-out, an open-set setting) and OOD-Language (train on one language, test on another). They state this is the first benchmark designed to jointly evaluate multilingual long-form LLM AA under multiple distribution shifts.

3. A comprehensive comparison of detector families. The paper evaluates metric-based methods (rank, entropy, gLTR), supervised methods (n-gram, bert-aa, detective) and fingerprint-based methods (trace variants) under low-resource (1–5 training books per LLM) and high-resource (10–30 books per LLM) regimes.

4. Diagnostic analysis of why transfer succeeds or fails. Using UMAP projections of xlm-roberta embeddings and trace fingerprints, the authors show that Transformer models retain generator-related information across languages to varying degrees, while fingerprint and metric-based signals are largely language-dependent.

Main Findings

• No universal winner in-language. Across languages, data regimes, and the ID, OOD-Domain and OOD-Author settings, no single method is consistently best. xlm-roberta is competitive in ID and OOD-Domain, n-gram frequently wins in the high-resource regime (especially OOD-Author), and trace variants top several language-specific configurations. rank, entropy and gLTR consistently perform poorly, indicating that individual token-level statistics provide insufficient signal.

• More training data helps, but unevenly. The high-resource setting improves the best achievable performance across all ID and OOD-Domain configurations, and in several cases the best detector approaches or reaches perfect macro-F1. Under OOD-Author, however, xlm-roberta performs worse in the high-resource setting than the low-resource one for Italian, Chinese and English, while detective improves across all languages and n-gram benefits most, becoming the best OOD-Author detector in five languages and second-best in Russian. The authors suggest xlm-roberta may become overconfident on unseen-author texts, assigning them to known classes instead of rejecting them.

• Cross-language transfer is only viable for Transformer detectors. In cross-language evaluation, xlm-roberta achieves the strongest overall performance and detective also transfers substantially better than the rest, despite generally scoring lower. Metric-based, n-gram and fingerprint-based methods reach near-zero performance in most cross-language configurations, suggesting their signals are largely language-dependent. trace also drops sharply under OOD-Language.

• Chinese is the hardest target language. Averaging across source languages, transfer to Chinese yields macro-F1 of 0.508 in the low-resource setting and 0.582 in the high-resource setting, the lowest average in both regimes. Russian, also non-Latin, is the easiest target in the high-resource regime at 0.841 (and 0.717 low-resource), so script differences alone do not explain the Chinese difficulty.

• Source-language strength depends on the resource regime. In the low-resource setting, Spanish is the strongest source language (average 0.866 across target languages), followed by Chinese (0.823), while German is weakest (0.435). In the high-resource setting, Italian becomes strongest (0.894), followed by Spanish (0.819), and Russian becomes weakest (0.410); Chinese falls markedly from 0.823 to 0.629.

• Language-family proximity helps but does not decide outcomes. Spanish-to-Italian reaches 0.926 in the low-resource setting and 0.857 high-resource, while Italian-to-Spanish reaches 0.750 low-resource and 0.981 high-resource, the highest score reported in that table. The Germanic pair is weaker and less symmetric: German-to-English is 0.420 low-resource and 0.673 high-resource, while English becomes the strongest source for German in the high-resource setting at 0.916.

• Representation spaces tell a consistent story. For xlm-roberta fine-tuned on Italian, macro-F1 falls from 0.956 in Italian ID to 0.676 under OOD-Domain, and the UMAP plots show less compact, more overlapping clusters. Generator-specific clusters largely persist on Spanish texts and remain visible but broader and less separated on Chinese texts. In the trace fingerprint space, Italian ID-test and OOD-Domain samples stay close to the training distribution, whereas fingerprints from other languages occupy clearly displaced regions, including Spanish.

• Generated books resemble human books on several surface metrics. Across languages the generated books show textual properties comparable to human-written books, particularly in lexical diversity and redundancy, but with higher Self-BLEU and lower perplexity (measured with mGPT), indicating greater inter-book lexical similarity and higher predictability for the reference model. No eligible human references were available for Chinese and Russian, so those languages were compared with LLM-generated books in other languages.

• Human evaluation supports quality with caveats. A small-scale evaluation of gemini-pro books in the Literature & Fiction genre found generally fluent and coherent books with strong passage-level scores, while narrative-level quality was less consistent across languages for subjective dimensions. Overall row values spanned 3.16 to 4.66 across the reported dimensions.

Methodology in Plain English

Building the dataset. The authors reuse a multi-stage book generation pipeline that mimics how humans write: constraints such as genre and time period are drawn from Project Gutenberg, and each book is expanded segment by segment, where each new segment is conditioned on the outline, the previous segment, and a running summary of the narrative. This keeps books globally coherent while allowing them to exceed the length of a single generation step. Prompts were translated into each target language and verified by native-speaker volunteers. Each of the five LLMs generated books in each of the six languages.

Cleaning and validation. Invalid control characters, markup and non-textual symbols were removed, while language-specific punctuation, diacritics and non-Latin scripts were preserved. Following prior work, 1.5K tokens were removed from the start and end of every book to reduce shortcut cues such as author names, titles or publication years. GlotLID was used as a sanity check, and all books matched their intended target language.

Defining the shifts. For OOD-Domain, genres are split into in-distribution and out-of-distribution subsets per generator so that no genre appears in both. For OOD-Author, one LLM is held out at a time; methods train on the remaining models and are tested on the held-out one, which makes the setting open-set. For OOD-Language, methods train on one source language and are tested on a different target language; the authors deliberately do not combine language shift with domain shift so that the two effects are not conflated.

Evaluation protocol. Detectors were calibrated on the development set by choosing a confidence threshold that balances attribution and rejection, and the same threshold was applied across ID, OOD-Domain, OOD-Author and OOD-Language because in practice one would not know whether a test document is in or out of distribution. Predictions below the threshold count as missed attributions for known authors and as correct rejections for unseen authors. Performance is reported as macro-F1, giving equal weight to all generating LLMs.

Methods compared. Metric-based detectors (rank, entropy, gLTR) use token-level probabilities from the multilingual reference model mGPT. Supervised detectors include n-gram (with Jieba segmentation for Chinese), bert-aa (using xlm-roberta as the multilingual encoder) and detective (with a multilingual sentence-embedding model). Fingerprint-based detectors are trace variants (trace_rank-js, trace_entr-js, trace_entr-norm), which model transitions between token-rank or entropy statistics; the original GPT-2 evaluator was replaced with Gemma. The authors also used UMAP to visualise xlm-roberta embeddings and trace fingerprints.

Why This Matters

Impact on research. The paper shows that strong in-language attribution scores do not automatically imply robustness: performance drops under domain and generator shift, and most method families collapse entirely when the language changes. It also provides a reusable testbed with explicit OOD dimensions, which lets future work report transfer results in a comparable way rather than on isolated, English-only, short-text setups.

Real-world applications:

• Publishing and editorial verification. The paper cites the emergence of LLM-ghostwritten books; a benchmark covering book-length texts and multiple languages supports workflows that check whether a submitted manuscript was machine-generated and, if so, by which model family.

• Platform content provenance and transparency. Knowing which generator produced a text can inform labelling, disclosure and accountability practices for large volumes of multilingual content.

• Journalism and fact-checking. Attribution supports investigations that need to distinguish outputs of different models rather than merely flag "AI involvement".

• Multilingual model governance and auditing. The source/target language asymmetries reported here, for example Chinese as the hardest target, give auditors concrete evidence that a single English-centric detector will not suffice across markets.

Industry relevance. The finding that supervised Transformer detectors transfer across languages while lightweight statistical and fingerprint methods do not is directly relevant to teams choosing detectors for multilingual deployments: the cheaper, more interpretable options may need per-language retraining, revision or replacement. The open-set, threshold-calibrated protocol also mirrors production constraints, where a detector must reject text from a model it has never seen rather than force it into a known class.

Future Directions

• Extend coverage to more languages. The authors state the benchmark does not cover low-resource languages, languages underrepresented in LLM pre-training data, or languages with substantially different grammatical and morphological characteristics.

• Add more and future LLMs. The current benchmark uses five models; extending to additional and future generators is described as an important direction.

• Move beyond single-author, non-adversarial settings. Mixed authorship (a book written by multiple authors) and adversarial generalisation such as obfuscation attacks remain open directions.

• Explain and mitigate cross-language degradation. The paper notes that the distinctive NGD values for Chinese and the Russian PPL variation may reflect distributional properties relevant to transfer, but states that these descriptive metrics do not directly establish the source of the performance gap. It also suggests that training with more data may cause overfitting to source-language patterns for Chinese and Russian.

• Broaden human evaluation. The human evaluation covers a small subset of generated books and is limited to gemini-pro literature texts, so validation across more models, genres and languages remains open.

Target Audience

This paper is most useful for researchers and engineers working on LLM-generated text detection and authorship attribution, particularly those building or evaluating multilingual systems. It also benefits benchmark designers and evaluation specialists who need to test robustness under distribution shift, applied practitioners in publishing, platform trust and safety, and fact-checking who need to know which detector families survive a language or model change, and multilingual NLP researchers interested in how language-family relatedness and script differences affect transfer of learned representations.

Authors’ abstract

While existing work on LLM authorship attribution (AA) has made progress, available benchmarks remain limited, often focusing on English, controlled settings, or relatively outdated models, with the few multilingual studies considering only relatively short texts. We introduce MultiGhostBench, a multilingual benchmark comprising 928 books generated by five recent LLMs across six languages and three scripts, with an average length of approximately 59K words per book. The benchmark supports evaluation under domain, author, and language shifts. Evaluation of representative AA methods shows that no single method consistently performs best across settings, and performance generally degrades under distribution shifts. Transformer-based detectors can retain generator-related information across languages, although transfer effectiveness varies by language pair, whereas statistical and fingerprint-based detectors are more language-dependent. We envision MultiGhostBench as a valuable resource for the development and evaluation of robust AA methods. The dataset and code can be found at https://github.com/GrecoMT/MultiGhostBench.