Think Globally, Group Locally: Evaluating LLMs Using Multi-Lingual Word Grouping Games

AI summary

Overview

Research area: Natural Language Processing / multilingual evaluation of large language models, specifically abstract reasoning benchmark design.

Technical level: Intermediate. Readers should be comfortable with benchmark evaluation metrics (F1, clustering scores), embedding-based similarity, and standard LLM model families, but the paper's core game setup is explained in plain terms.

Scope: The paper introduces GlobalGroup, a multilingual word-grouping game benchmark derived from the New York Times Connections, built from author-created word groupings in English, Spanish, Chinese, Hindi, and Arabic (plus English translations of the non-English sets) and used to measure language- and culture-related performance disparities across six open- and closed-source LLMs.

What This Paper Is About

LLMs often perform better on reasoning tasks in one language than another, even when the underlying content is equivalent, and prior work has mostly tested this on knowledge-heavy tasks such as commonsense or math. This paper targets the less-studied case of abstract reasoning, where a model must spot and build patterns through lateral thinking rather than apply a formula or recall facts. The authors build a multilingual, game-based benchmark called GlobalGroup to measure that skill across languages, and they add difficulty measurements so that games of comparable difficulty can be compared in a more controlled way.

Key Contributions

1. A new multilingual abstract-reasoning benchmark. GlobalGroup is inspired by NYT Connections: a model receives a word pool and must form equal-sized groups of words plus a topic connecting each group. The benchmark covers five linguistic backgrounds (English, Spanish, Chinese, Hindi, and Arabic), each in the native language and in English translation (es, es-en, zh, zh-en, hi, hi-en, ar, ar-en, en).

2. Author-created word groupings plus a Connections-derived comparison set. Native-speaker annotators produced groups of 4 words linked by a topic, labeled as culturally-related or non-culturally-related. After a quality check, this yielded 48 en, 48 es, 80 zh, 49 hi, and 40 ar groupings. A separate dataset collected all groupings from 511 Connections games (2,044 groupings total).

3. Difficulty measurements for controlled reasoning evaluation. The paper proposes and tests three difficulty metrics: group size and group count (game structure), Adjusted Rand Index (ARI) of semantic clustering of group words, and word overlap between groups measured via GPT-4o-mini candidate proposals.

4. A two-part evaluation scheme that scores both groupings and topics. Grouping quality is measured with group-level F1, and topic prediction with a boolean "Topic Achieved" (TA) score based on FastText embedding cosine similarity.

Main Findings

• English translations generally help. For es, hi, and ar, both F1 and TA scores were better when groupings were translated into English across all models. Chinese behaved differently: closed-source models did better in Chinese than in the zh-en translation, while open-source models (except Aya-8B) did better in English.

• Some translated subsets reached very high F1. For example, GPT-4 reached 0.920 F1 on es-en, 0.935 on zh-en, 0.964 on hi-en, and 0.957 on ar-en, while on the untranslated sets it scored 0.892 (es), 0.971 (zh), 0.963 (hi), and 0.877 (ar).

• A multilingual training focus can close a size gap. Aya-8B, a small open-source multilingual model, performed comparably to far larger LLMs across scored metrics and dataset backgrounds, while Llama3-8B and Mistral-7B performed similarly poorly.

• Model size matters. Going from Llama3-8B to Llama3.1-70B produced general F1 gains (many subsets above 90% F1) and smaller performance deficits between English and non-English subsets, indicating size affects the magnitude of language-related bias.

• NYT Connections games are harder than the in-house games. F1 and TA scores on nyt-shuf and nyt-seq were much lower across models; for instance GPT-4 scored 0.813 F1 on nyt-shuf and 0.707 on nyt-seq versus 0.943 on the in-house en set. The in-house en games performed similarly to "Yellow" Connections groupings, the easiest grouping type. nyt-seq scored lower than nyt-shuf for all six models, which the authors suggest reflects deliberately designed group overlap.

• Non-culturally-related groupings score higher. The mean F1 difference (non-culturally-related minus culturally-related) across all subsets excluding nyt-seq and nyt-shuf was 0.050 for GPT-3.5-Turbo, 0.032 for GPT-4, 0.044 for Llama3-8B, 0.045 for Llama3.1-70B, 0.086 for Mistral-7B, and 0.065 for Aya-8B.

• Closed- and open-source models separate geometrically. In a PCA of models represented by their mean 4×4 results per dataset subset, closed-source models clustered together and open-source models formed another cluster, also seen with K-means at k=2. Hindi, Chinese, and Arabic — all non-Latin-script subsets — had the largest weights on the first two principal components.

• Group count, ARI, and word overlap track difficulty; group size does not. Increasing group count lowered F1 (GPT-3.5-Turbo 0.930 to 0.842 from 2 to 4 groups; GPT-4 0.962 to 0.912). Higher ARI raised F1 (GPT-3.5-Turbo 0.924 to 0.951 across the three ARI bins; GPT-4 0.969 to 0.990). Higher word overlap lowered F1 (GPT-3.5-Turbo 0.932 to 0.831 across the overlap bins; GPT-4 0.977 to 0.859). Group size patterns were inconsistent, with GPT-3.5-Turbo peaking at group size 3 (0.890) while GPT-4 rose with size (0.925 to 0.949).

• Topic scoring setup. The TA score uses FastText embeddings with a cosine similarity threshold of 0.3, chosen after comparing BERTScore and FastText at thresholds from 0.1 to 0.7 in 0.1 increments and validating against human annotations of 200 Chinese grouping attempts; the chosen setting attained a Randolph's Kappa of 0.55.

Methodology in Plain English

The authors turned a familiar puzzle into a benchmark. Annotators who are native speakers of each language created small groups of four words joined by a shared topic, deliberately making some topics culturally related (for example, characters from Don Quixote or American holidays) and others not (for example, aquatic animals or parts of a train). The groupings were then translated into English by the same annotators, with verification assistance from Google Translate and DeepL, so the same content could be played in two languages.

From these groupings, games are assembled by sampling groups and then sampling words from each group, giving a word pool that the model must partition. The default game is 16 words in four groups of four, but the authors also built 2-, 3-, and 4-group games with 2, 3, or 4 words per group, producing nine settings for each of the nine language/translation subsets; each setting has 600 games split into 300 development and 300 test games. The Connections-derived data was split into nyt-seq (three settings, 211 development and 300 test games) and nyt-shuf (nine settings, 600 games split evenly). Within each game, the authors ensured no repeating words or topics, so each game has a single correct solution.

Six models were evaluated zero-shot with prompts giving the rules, the shuffled word pool, and an output-format example: GPT-3.5-Turbo and GPT-4 (closed source), and Llama-3-8B-Instruct, Llama-3.1-70B-Instruct, Mistral-7B-Instruct-v0.2, and Aya-23-8B (open source). Responses were parsed with regular expressions and, when formatting failed, reformatted by prompting GPT-4o-mini. Scoring uses group-level F1, which credits partially correct groups rather than requiring an exact match, and the TA score for topics. For difficulty, the team measured semantic cohesion of true groups by clustering word embeddings and computing ARI, measured word overlap by having GPT-4o-mini propose candidate groupings without size constraints and averaging set intersections across them, and sliced results by group size and group count.

Why This Matters

The paper argues that abstract reasoning — the kind needed for everyday out-of-the-box thinking — has been largely left out of multilingual LLM evaluation, which has focused on commonsense, math, and other knowledge-dependent tasks. By providing groupings in five languages with matched English translations, it gives a tool for isolating language effects from content effects, and the difficulty metrics let researchers compare models on games of equivalent difficulty rather than on raw averages.

Real-world applications:

• Model selection for multilingual deployment. The findings on where smaller open-source models match larger closed models (notably on Arabic tasks) inform cost-versus-performance decisions for organizations serving non-English users.
• Multilingual training-data strategy. The Aya-8B result is presented as evidence that a multilingual-focused training paradigm can let a small model compete with much larger ones, which is relevant to teams allocating compute and data budgets.
• Benchmark design for low-resource languages. The dataset construction and annotation procedures are documented so the community can extend coverage to additional languages.
• Bias auditing and fairness reporting. The language and culture performance gaps provide a template for flagging unequal reasoning capability across user populations.

Industry relevance centers on LLM developers and evaluation teams: the benchmark offers a reproducible way to test whether a model's reasoning transfers across languages, and the difficulty metrics offer a way to make those comparisons fairer than raw accuracy alone.

Future Directions

• Expand beyond five languages. The authors explicitly acknowledge that thousands of languages and many cultural backgrounds are unrepresented and invite community contributions using their annotator request and dataset construction procedures.
• Broaden annotator demographics. The paper notes that its annotators do not reflect the complete demographics of speakers of a given language, so some bias may have been introduced, and it calls for wider representation.
• Further validate and refine difficulty metrics. Group size did not behave as a clean difficulty signal, and the authors report only three features that correlate with performance, leaving room for metrics that better explain model failures.
• Investigate the Chinese-language anomaly. Chinese was the one language where translation to English did not consistently help, which the authors attribute tentatively to training-data volume rather than to the language itself — an open question for training-paradigm research.

Target Audience

This paper is most useful to NLP researchers working on multilingual evaluation, abstract reasoning benchmarks, or LLM fairness across languages; to benchmark designers who need difficulty-controlled comparison methods; and to practitioners selecting or fine-tuning models for non-English user bases. Readers interested in culturally grounded evaluation and annotation methodology will also find the dataset construction and limitation discussion relevant.

Authors’ abstract

Large language models (LLMs) can exhibit biases in reasoning capabilities due to linguistic modality, performing better on tasks in one language versus another, even with similar content. Most previous works evaluate this through reasoning tasks where reliance on strategies or knowledge can ensure success, such as in commonsense or math tasks. However, abstract reasoning is vital to reasoning for everyday life, where people apply "out-of-the-box thinking" to identify and use patterns for solutions, without a reliance on formulaic approaches. Comparatively, little work has evaluated linguistic biases in this task type. In this paper, we propose a task inspired by the New York Times Connections: GlobalGroup, that evaluates models in an abstract reasoning task across several languages. We constructed a game benchmark with five linguistic backgrounds -- English, Spanish, Chinese, Hindi, and Arabic -- in both the native language and an English translation for comparison. We also proposed game difficulty measurements to evaluate models on games with similar difficulty, enabling a more controlled comparison, which is particularly important in reasoning evaluations. Through experimentation, we find English modalities largely lead to better performance in this abstract reasoning task, and performance disparities between open- and closed-source models.