CulturalRiddles: A Multicultural Benchmark of Riddles

AI summary

Overview

Research area: Natural Language Processing — multilingual and multicultural evaluation of large language models (LLMs), specifically benchmark construction.

Technical level: Intermediate. Readers should be familiar with the general idea of LLM benchmarks and accuracy-based evaluation, but no specialized math or modeling background is needed.

Scope: The paper introduces CulturalRiddles, a human-authored, community-sourced benchmark of culturally grounded riddles spanning 62 cultures, and reports how 28 LLM configurations perform on it under automatic and human evaluation.

What This Paper Is About

Most benchmarks used to test how well language models understand culture are built in English, cover a narrow set of cultures, and mostly probe whether a model can recall isolated facts. That leaves open the question of whether a model can actually use cultural knowledge the way people do — in everyday reasoning, wordplay, and shared references. The authors address this by building a benchmark out of riddles, a form of expression that is deeply tied to a specific culture and that cannot be solved by memorized facts alone.

Key Contributions

1. A new multicultural benchmark built from riddles. CulturalRiddles consists of original riddles written entirely by humans rather than generated or translated, covering 62 cultures across 45 countries and 51 languages.

2. A participatory, community-driven construction process. Rather than collecting data from existing sources, the benchmark was assembled through a community effort in which contributors authored riddles rooted in their own cultures.

3. A task designed to test applied cultural reasoning. The riddles are intended to require abductive and compositional reasoning, factual and social knowledge, language skills, and familiarity with culturally specific associations — not simple fact recall.

4. A large evaluation of current models plus released artifacts. The authors evaluate 28 LLM configurations using both automatic scoring and large-scale human evaluation, and they release all artifacts along with an open contribution process for extending the benchmark's language and culture coverage.

Main Findings

• Frontier models struggle badly on some cultures: The strongest models reach at most 22% accuracy in some communities, according to the abstract. Per-model and per-culture breakdowns are not provided in the abstract.

• Gaps persist despite broad language coverage in current LLMs: The framing of the paper is that expanding language coverage has not translated into comparable cultural understanding; the riddle results are presented as evidence of that gap.

• Factual knowledge is not sufficient: Because the items are built to demand reasoning and culturally specific associations, the benchmark is positioned as measuring something that fact-oriented, English-centric benchmarks miss. The abstract does not report separate scores isolating each skill.

• Evaluation combines automatic and human judgment: The authors describe both automatic scoring and "large-scale human evaluation," but the abstract gives no detail on agreement rates, human accuracy, or how the two compare.

Methodology in Plain English

The authors did not scrape or machine-translate existing riddle collections. Instead, they organized a community effort in which many contributors each wrote original riddles grounded in their own culture and language, producing a set that spans dozens of cultures. Because a good riddle usually depends on a pun, a local reference, or a chain of inference rather than a single lookup, answering one requires the model to combine clues and draw on cultural knowledge the way a member of that community would.

The team then ran 28 different LLM setups through the benchmark and scored the results in two ways: automatically, and with human judges assessing model outputs at scale. They also committed to releasing the data, the prompts/artifacts, and a process for new contributors to add cultures and languages. The abstract does not state how many riddles are in the benchmark, how cultures or languages were selected, what the prompt format is, or which specific models were tested.

Why This Matters

Impact on research. The paper argues that cultural evaluation of LLMs is currently skewed toward English and toward factual recall. A human-authored, riddle-based benchmark offers a complementary test that targets reasoning and culturally specific associations, and the open contribution model suggests a way to keep expanding coverage rather than freezing it at publication time.

Real-world applications:

• Consumer assistants used outside English-speaking markets, where misunderstanding a local reference or idiom produces confidently wrong or unintentionally rude answers.
• Localization and content tools, such as translation and marketing systems that need to preserve humor, wordplay, and cultural nuance rather than just literal meaning.
• Educational technology deployed in multilingual classrooms, where explanation quality depends on culturally familiar framing.
• Model selection and procurement, giving organizations a way to check whether a vendor's claims of multilingual support extend to cultural competence in the specific communities they serve.

Industry relevance. The reported ceiling of 22% accuracy in some communities is a practical warning for anyone deploying a model globally on the assumption that strong benchmark scores in English imply broad cultural competence. Because evaluation here relies partly on human judgment, the work also speaks to teams building evaluation pipelines that go beyond automatic accuracy.

Future Directions

• Expand linguistic and cultural coverage. The authors explicitly invite open contributions to grow the benchmark beyond its current 62 cultures, which raises open questions about quality control and consistency as authorship broadens.
• Diagnose why models fail. The abstract reports low accuracy but not the error structure; follow-up work could separate failures of language fluency, missing cultural facts, and weak multi-step reasoning.
• Clarify human versus automatic evaluation. With both evaluation modes in play, a natural next question is where automatic scoring diverges from human judgment, and whether automatic metrics are trustworthy for culturally loaded items.
• Connect benchmark scores to deployment outcomes. Whether riddle performance predicts real-world failures in multilingual products is untested here and would be a valuable next step.

Target Audience

Most useful to NLP researchers working on multilingual and cultural evaluation, benchmark designers, and teams auditing LLMs for deployment in non-English-speaking markets. It is also relevant to linguists, folklorists, and community contributors interested in representing cultural knowledge in machine learning datasets, and to practitioners who need a concrete argument for why English-centric benchmark scores are not enough.

Authors’ abstract

As LLMs expand their language coverage, evaluating how well they understand and represent diverse cultures becomes increasingly important. However, many benchmarks remain English-centric, with limited cultural coverage and a focus on factual knowledge over its application in real-world cultural contexts. We propose CulturalRiddles, a benchmark of original, culturally grounded riddles created entirely by humans through a participatory community effort, spanning 62 cultures (45 countries, 51 languages). These riddles are deeply rooted in their respective cultures and require abductive and compositional reasoning, alongside detailed factual and social knowledge, language skills, and an understanding of culturally specific associations. We benchmark 28 LLM configurations using both automatic and large-scale human evaluation. Even frontier models achieve at most 22% accuracy in some communities. We publicly release all artifacts alongside an open contribution process to expand the benchmark's linguistic and cultural coverage.