CorporateBench: Large-Scale Q&A Benchmarking with Temporal Knowledge Bases

AI summary

Overview

Research area: Benchmarking and evaluation of large language models on enterprise-scale, multi-document question answering; synthetic document generation from temporal knowledge bases.

Technical level: Intermediate. The paper is readable without deep technical background, but it assumes familiarity with retrieval-augmented generation (RAG), knowledge graphs, and standard QA scoring metrics such as F1.

Scope: The paper introduces CorporateBench (CB), a human-validated Q&A benchmark built from four procedurally generated synthetic companies whose corpora scale from 354 to 232,693 documents, and reports how five contemporary LLMs perform on extraction and QA tasks over those corpora.

What This Paper Is About

Evaluating LLMs on realistic corporate knowledge work is difficult because companies cannot share internal communications, and existing synthetic benchmarks are too simple or too small. CorporateBench addresses this by first generating a temporally evolving knowledge base for a simulated company, then sampling a large corpus of emails whose statements are guaranteed not to contradict that knowledge base. The goal is to measure how well models retrieve and reason across hundreds of thousands of mutually consistent documents, with answers computed deterministically from the underlying knowledge base rather than written by hand.

Key Contributions

1. Comprehensive company simulation. The authors adapt knowledge-base-to-corpus generation to produce corporate document collections that preserve temporal sequencing and role-respecting communication directions across teams and departments.
2. Deeply interconnected evidence sets. Answering typical queries requires integrating evidence across many documents; CB achieves 87.6 documents per question, compared with 1.5 for the closest benchmark (EKRAG, Yu et al., 2025), and is compared against nine previous benchmarks in evidence and question counts.
3. Enterprise-scale tasks with deterministic labels. A five-task suite split into extraction and QA categories, with labels computed directly from the knowledge base via verified SPARQL queries, enabling reproducible evaluation.
4. A released dataset and codebase. Four synthetic companies totalling 263,466 documents, released at https://huggingface.co/datasets/epiq-ai-labs/corporatebench.

Main Findings

• Scale versus accuracy. Across all three QA types, the best KB (SQL-grounded) scores exceed the corresponding best RAG scores, and the gap widens with scale: on KB QA the KB–RAG difference increases from 0.24 for the small company to 0.37 for the extra-large company.
• Extraction degrades unevenly. Entity extraction remains stable across scales (F1 0.715–0.824), while relationship extraction degrades from 0.419–0.845 (S) to 0.205–0.494 (XL). Temporal relationships are hardest, dropping from F1 0.142–0.470 (S) to 0.062–0.173 (XL).
• Two error patterns identified. Entity extraction errors cascade into relationship extraction, and contradictory relationship descriptions accumulate across documents, with different models struggling with different relationship types.
• Topic classification shows opposing trends on the two larger companies. On Biocure (L), LLMs reach F1 0.872–0.950 zero-shot and 0.877–0.982 with 50 examples, approaching the TF-IDF+LR baseline's 0.993 at 10K examples with 200x less data. On Pound (XL), LLMs reach 0.357–0.531 zero-shot and 0.598–0.751 with 50 examples, below the baseline's 0.983 at 100K examples. LLMs performed better on Biocure (31 categories) than on Pound (17 categories), which the authors attribute to ambiguous boundaries created by Pound's hierarchical topic merging.
• KB QA and Topic QA degrade with corpus size; Integrated QA does not. Integrated QA results slightly improve with scale for both RAG and KB methods, which the authors suggest reflects richer cross-source context for synthesis.
• Model-level differences. Both GPT-5 models outperform others with RAG, while Claude Sonnet 4.5 excels at KB querying. GPT-5.1 shows an early-stopping problem — returning an empty string after a tool response — occurring 191 times versus 56 times with RAG, which partly explains the gap between settings.
• Human validation. Across 1,000 judgements on a balanced set built by inverting 100 sampled emails with Claude Sonnet 4, non-expert reviewers correctly identified the intended relationship 76.2% of the time.
• Overall conclusion. Contemporary LLMs are marginally competent at these enterprise tasks, with QA performance decreasing as corpora scale up to and beyond 10^5 documents.
• Note on coverage. Table 3 reports KB evaluation results for only three models (Haiku 4.5, GPT-5 Nano, Gemini 2.5 F-L), not all five evaluated models.

Methodology in Plain English

The pipeline works from ground truth outward rather than from documents inward.

1. Build a knowledge base. A company C = (D, T, E) is defined as the union of all employees, organized by teams and departments, where each team belongs to exactly one department and each employee belongs to exactly one team, except for department leads and executives. The ontology has two base classes, Entity and Relationship, extended with organizational entities (companies, departments, teams, employees) and work matter (projects, meetings, tasks), linked by seven predicates: MemberOf, ReportsTo, WorksOn, WorksAt, Attends, Organize, and BelongsTo, serialized in Turtle and validated with an OWL parser.
2. Generate a company. Department and team counts scale logarithmically with target headcount, targeting an average team headcount of 5–10 and department headcount of 8–68 when scaling from 10^1 to 10^4 employees, with C-suite size from 1 to 13 executives. Work is simulated over one quarter (90 days): each day, each employee is assigned a task with probability P = 1/45, so each completes 1 to 3 tasks by quarter's end. Meetings are simulated across six types (direct reports, team meetings, task collaboration sessions, executive meetings, project reviews, department meetings) respecting roles and reporting structures. At the end of the quarter, roughly 10% additional employees are hired, forcing team splits.
3. Annotate entities. Each company gets a one-paragraph "story" (origin, motto, focus, industry), and Claude Haiku 4.5 is prompted recursively down the hierarchy to produce contextually appropriate properties; aliases for employees and projects are added for diversity. Triples are stored as N-Quads and validated against the ontology.
4. Generate documents. Emails substantiate one or more relationships, with sender, recipient, subject, body, and timestamp taken from the KB. Explicit "evidence strings" reveal the relationship; there are 3,217 such strings across the seven Relationship types, split into 360 strings per temporal bound and 40 strings per level of difficulty. Meetings use a separate set of evidence strings binned by progress up to 25%, 50%, 75%, and 100%. Claude Haiku 4.5 fills in the remainder of each email around the evidence string.
5. Add lexical diversity. Formatting follows the Enron corpus; topics are sampled from a Dirichlet distribution with α = 0.75 from 3,000 business topics and 3,000 non-business topics, with meeting topics from a separate set of 60; each employee gets one of sixteen Myers–Briggs personalities, and each document is written formally or casually.
6. Define tasks. Two extraction tasks (KB Evaluation, scored by F1 over ingested triples; Topic Classification, macro F1 with 31 classes for Biocure and 17 for Pound on a stratified test set of 1000 documents per corpus) and three QA tasks (KB QA, Topic QA, Integrated QA) generated from 250 manually written question templates, instantiated via verified SPARQL queries. Answers come in five types (string, date, integer, boolean, sets of strings), scored by exact match except sets, which use F1. Models are run in a RAG setting (pgvector with text-embedding-3-small) and a KB setting (SQL access to the ground truth KB).

Why This Matters

Impact on research. CorporateBench targets a specific gap: long-context benchmarks built on books or scripts have too few documents to count as enterprise scale, while existing enterprise benchmarks often transform non-corporate data or are generated by prompting LLMs without any guarantee that outputs match the desired world state. Generating documents by sampling from a validated knowledge base gives arbitrary scale with cross-document logical consistency, and the deterministic labels make the benchmark reproducible as context windows grow.

Real-world applications:

• Benchmarking retrieval and reasoning systems for enterprise search over email archives.
• Evaluating knowledge-base construction (entity and relationship extraction) from messy corporate communications, a prerequisite for many internal AI deployments.
• Testing temporal reasoning, such as determining when an employee joined, changed teams, or began a project, from onboarding and handover messages.
• Setting expectations for RAG deployment size: the paper's finding that RAG may suffice for small corpora but degrades relative to direct KB access at scale is directly relevant to system architecture decisions.

Industry relevance. Enterprises evaluating LLM assistants, and vendors selling retrieval systems into regulated industries such as pharma and finance, can use CB as a cheap, non-confidential proxy for internal document collections. The paper notes the benchmark uses "lighter" models specifically because evaluating a model on the full 263,466-document set can incur significant API costs.

Future Directions

• Multi-channel communication. Extending the benchmark beyond email to channels such as Slack and Microsoft Teams, and to document types like technical specifications, financial reports, spreadsheets, and presentations, each with distinct structural characteristics.
• Longer time horizons. Moving past the 90-day simulation window to model strategic pivots, mergers and acquisitions, cultural shifts, and gradual accumulation of institutional knowledge.
• Reducing LLM-generated artifacts. Since documents are produced by Claude Haiku 4.5, writing style and vocabulary may carry systematic bias; the authors describe their tasks as a "lower bound" on difficulty, to be solved before messier human-written corpora.
• Understanding the Integrated QA inversion. Why Integrated QA improves with scale while KB QA and Topic QA degrade is not resolved; the authors offer richer cross-source context as a hypothesis.
• Addressing early stopping. GPT-5.1's tendency to return an empty string after tool responses (191 occurrences in the KB setting) points to a tool-use failure mode worth studying separately.

Target Audience

LLM evaluation researchers and benchmark builders working on long-context and multi-document reasoning; practitioners building retrieval-augmented or knowledge-base-grounded assistants for enterprise document collections; and industry teams in regulated sectors (pharma, finance, media, tech) who need a non-confidential proxy for internal corporate data. Readers interested in synthetic data generation from knowledge graphs and in ontology-grounded document synthesis will also find the pipeline design relevant.

Authors’ abstract

LLMs are increasingly able to answer complex questions about enterprise-scale document collections. But evaluation is hard: companies don't want to share internal communications, and synthetic datasets have been overly simple. We present CorporateBench (CB), a human-validated multi-task Q&A benchmark whose scale approaches the conditions LLMs encounter in corporate communication networks, with evaluation corpora surpassing 230,000 documents. CB evaluates LLMs across two dimensions (information extraction and knowledge base querying) through four synthetically generated firms ranging from 12 to 10,000 employees. Each corpus is sampled from a temporally evolving knowledge base describing a consistent world, guaranteeing cross-document logical consistency even across hundreds of thousands of documents. We evaluate five LLMs on CB, revealing increasingly poor performance as input size approaches realistic scales. CB provides LLM developers a metric for corporate communication reasoning, filling a crucial gap in the benchmarking ecosystem.