LC-Eval: A Bilingual Multi-Task Evaluation Benchmark for Long-Context Understanding

AI summary

Overview

• Research area: Natural Language Processing — long-context language model evaluation, bilingual (Arabic/English) benchmarking.
• Technical level: Intermediate. The paper assumes familiarity with LLM evaluation concepts (benchmarks, ROUGE/BLEU, LLM-as-a-judge) but explains its own tasks and metrics clearly.
• Scope: The paper introduces LC-Eval, a human-validated bilingual benchmark of 7,903 samples covering four long-context tasks in Arabic and English across contexts from 4K to over 128K tokens, and reports baseline results for eight language models.

What This Paper Is About

Long-context language models (LCLMs) can now ingest very long inputs, but the benchmarks used to test them are mostly English-only, often rely on multiple-choice questions or short answers, and score open-ended text with exact-match metrics like BLEU and ROUGE that ignore meaning. Arabic in particular lacks a dedicated long-context benchmark. The authors build LC-Eval to fill that gap: a bilingual, four-task, human-validated evaluation suite that stresses deep reasoning, document tracing, and cross-lingual information extraction, together with a semantic, entity-relationship-based scoring method for open-ended answers.

Key Contributions

1. A large-scale dataset of 7,903 samples spanning context lengths from 4K to over 128K, targeting deep reasoning, document tracing, and bilingual information extraction (Table 1).
2. Parallel Arabic and English datasets for every task, enabling comparative assessment of LCLM performance across languages and text genres.
3. An entity-relationship-based evaluation approach for open-ended question answering that judges overlap by conceptual meaning rather than lexical matching (Section 5.2.1).
4. Multiple complementary metrics — entity-relationship recall and F1, recall@k, and accuracy — for comprehensive assessment (Section 5.2).

Main Findings

• Four tasks defined: multi-document question answering, bilingual question answering, claim verification within a paragraph, and multiple-choice question answering based on long contexts.
• Dataset composition (Table 1): AR Multidoc QA 1180 samples; EN Multidoc QA 1186; AR Bilingual QA 1194; EN Bilingual QA 1191; AR Claim Verification 400; EN Claim Verification 400; AR MCQs 1200; EN MCQs 1152. Average word counts range from 6,006 (AR Multidoc QA) to 57,667 (EN Claim Verification).
• Token fertility differs sharply by language (Table 2): Arabic requires roughly 1.995 to 2.350 tokens per word depending on the tokenizer, versus about 1.262 to 1.417 for English. Consequently, 128K tokens corresponds to roughly 64K words for Arabic but about 106K words for English.
• GPT-4o led on multi-document QA with the highest entity-relationship evaluation in both Arabic (70.55) and English (77.81), though Command-r-plus-08-2024 had higher entity-relationship recall than four other LCLMs yet failed to retrieve the correct document IDs, lowering its average recall.
• Information tracing is the weak point: most models degrade as word count increases. Standard deviation of entity-relationship recall across word-count bins was 2.29% to 11.49% for Arabic and 4.72% to 19.36% for English, with recall standard deviations typically much higher than entity-relationship recall ones.
• Semantic scoring correlates with lexical scoring: Pearson correlation between entity-relationship recall and ROUGE-L was 0.77 for Arabic and 0.94 for English in multi-document QA; in bilingual QA it ranged from 0.73 to 0.87 against BLEU and ROUGE-L.
• Bilingual QA: Qwen2.5-14B obtained the highest correct-language accuracy for both Arabic and English, while GPT-4o had the highest entity-relationship recall. Arabic standard deviations across context lengths were 0.84% to 8.64% for most LCLMs, with Llama-3.3-70B at 24% or higher; English ranged from 4.56% (Llama-3.3-70B) to 25.21% (Llama-3.1-8B Instruct).
• Claim verification: accuracy sometimes dropped sharply when claims were presented as a paragraph rather than sentence by sentence, and sometimes rose. Standard deviations across word-count bins were close for Arabic and English, ranging from 1.91% to 7.11%.
• MCQs: all models scored higher on English MCQs than on the other three tasks, and some models (Llama-3.1-8B and Phi-3.5-mini) showed a substantial Arabic–English disparity. Accuracy moved unpredictably across word-count bins with no consistent trend.
• Overall language gap: all LCLMs performed better in English than in Arabic.
• Human check of the entity-relationship metric: using 50 randomly selected multi-document QA samples per model, human evaluations correlated strongly with the automatic scores for the best-performing models (GPT-4o, Qwen2.5-14B, Command-r-plus-08-2024, and Phi-3.5-mini).
• Memorization control: GPT-4o averaged 75.88 when context was provided versus 56.41 without context — a 20-point gap that was even larger in Arabic than in English.

Methodology in Plain English

The authors assembled source text from 2024 Wikipedia dumps, WikiNews, WikiHow, and WikiBooks in both languages, English books from Project Gutenberg, Arabic books from the Hindawi Organization, and articles from the Saudi Press Agency. They filtered out potentially harmful content with a custom word-based dictionary, then sampled to spread data evenly across sources and context-length bins. Questions, answers, claim paragraphs, and MCQs were initially generated with GPT-4/GPT-4o using task-specific prompts, with generated items scored by GPT-4o at four temperatures (0.0, 0.3, 0.6, 0.9) on Accuracy, Grammar and Syntax, Cultural Sensitivity, and Safety, keeping only items scoring at least 9 with an average majority vote of at least 9.0. Three human annotators then reviewed every sample, with acceptance requiring agreement from at least two; 1,300 high-quality instances each were sent to human validation for multi-document QA and bilingual QA. For open-ended answers, the authors used GPT-4o to extract entities and their relationships from both gold and model responses, then measured recall and F1 of the shared relationships based on conceptual meaning rather than wording. They evaluated eight models — Llama-3.1-8B Instruct, Llama-3.3-70B Instruct, Qwen2.5-14B Instruct, Command-r-plus08-2024 Instruct, Phi-3.5-mini Instruct (five open-source 128K-context models), AceGPT-v2-32B Instruct and Qwen2.5-72B Instruct (two open-source 32K-context models), and the 128K-context GPT-4o API — measuring context by word count so tokenizer differences would not distort comparisons.

Why This Matters

• Impact on research: LC-Eval provides the Arabic long-context evaluation resource the authors say did not previously exist, and its entity-relationship metric offers an alternative to exact-match and similarity scoring that the authors argue is more reliable for open-ended generation.
• Enterprise search and RAG over Arabic documents: the benchmark exposes how poorly models trace which source document an answer came from, a core requirement for retrieval-augmented systems.
• Multilingual customer support: bilingual QA tests the common real-world case where the source context and the user's question are in different languages.
• Fact-checking and news monitoring: the claim verification task mirrors checking statements against long source material, relevant to media and regulatory workflows.
• Document-heavy professional domains: long books and reports are used as context; MCQ and multi-document QA reflect legal, educational, and technical document review.
• Industry relevance: the benchmark provides a public, human-validated yardstick (hosted on Hugging Face) for vendors and enterprises choosing models for Arabic and mixed Arabic–English deployments, and the consistent English-over-Arabic gap signals where investment in Arabic model training is needed.

Future Directions

1. Reduce GPT-4o creator bias: because the initial data was generated with GPT-4o and then human-validated, GPT-4o may have an evaluation advantage over other models.
2. Enlarge per-bin sample sizes so that randomness in model performance is eliminated.
3. Validate the summaries used to generate MCQs and claim paragraphs, since their content was not validated and may introduce inaccuracies.
4. Improve domain balance, as the dataset lacks sufficient datapoints per individual domain to support domain-level conclusions.
5. Align human evaluation of entity relationships exactly with the automatic entity-relationship recall method for a more direct comparison, and address the fact that the metric does not penalize repeated output — a model can reach 100% entity-relationship recall while repeating itself.

Target Audience

NLP and LLM evaluation researchers, particularly those working on long-context and multilingual benchmarking; Arabic NLP researchers and developers building Arabic or bilingual models and applications; model evaluation and quality teams in industry who need a public, human-validated test set for 4K-to-128K-token contexts; and practitioners deploying retrieval-augmented or document-heavy systems in Arabic or mixed Arabic–English settings.

Authors’ abstract

Recent advancements in Large Language Models (LLMs) have demonstrated sophisticated capabilities, including the ability to process and comprehend extended contexts. These emergent capabilities necessitate rigorous evaluation methods to effectively assess their performance in long-context understanding. In this paper, we present \textbf{LC-Eval}, a bilingual, multi-task evaluation benchmark designed to evaluate long-context understanding in English and Arabic, targeting context lengths ranging from 4k to over 128k tokens. LC-Eval introduces four novel and challenging tasks: multi-document question answering, bilingual question answering, claim verification within a paragraph, and multiple-choice questions based on long contexts. These tasks are designed to assess LLMs' abilities in deep reasoning, document comprehension, information tracing, and bilingual information extraction and understanding. The benchmark includes datasets in both Arabic and English for each task, allowing for a comparative analysis of their performance across different text genres. Evaluations were conducted on both open-weight and closed LLMs, with results indicating that LC-Eval presents significant challenges. Even high-performing models, such as GPT-4o, struggled with certain tasks, highlighting the complexity and rigor of the benchmark.