A Framework for Generating Valid Context-Specific Benchmarks through Expert Guidance

AI summary

Overview

Research area: LLM evaluation methodology, benchmark dataset construction, measurement validity, human-centered AI.

Technical level: Intermediate — assumes familiarity with LLM benchmarking, embeddings, and distributional distance metrics, but the framework itself is conceptual and domain-agnostic.

Scope: The paper proposes and empirically validates an end-to-end framework — a structured expert-elicitation schema plus four quantitative dataset quality metrics — for generating context-specific LLM evaluation datasets that are both scalable and valid.

What This Paper Is About

Most public LLM benchmarks measure "universal" capabilities and are not grounded in any particular deployment context, which raises validity concerns: benchmark scores may not reflect what actually matters to stakeholders in a specific setting. Building benchmarks with domain experts yields high-quality but slow and expensive datasets, while purely synthetic generation scales well but produces unrealistic, redundant, or out-of-scope examples. This paper closes that gap by using structured expert input to steer synthetic data generation toward more valid evaluation datasets.

Key Contributions

1. A schema for eliciting context. A lightweight structured template that captures the minimum information needed to define an evaluation task, organized around three components from prior validity work — population, concept, and instance — decomposed into eight fields (deployment population, contextual constraints, capability, definition, constituents, what is always present, varying factors, and seed examples).

2. Four operationalized dataset quality criteria grounded in measurement validity. Coverage and diversity together operationalize content validity; content realism and stylistic realism together operationalize ecological validity. Each is defined as a concrete, computable metric bounded in [0,1].

3. A guided generation pipeline. The filled schema is inserted into a general system prompt plus a domain-specific user prompt, instructing an LLM to emit benchmark examples as CSV rows that also label each example's constituent factor values (enabling automated coverage scoring).

4. Empirical validation and ablation guidance. A think-aloud study with domain experts plus a systematic ablation over 1,013 schema-field configurations showing which expert inputs matter most for which quality dimension — providing practical resource-allocation guidance.

Main Findings

• Experts strongly preferred schema-guided examples. Five of six social workers preferred the schema-generated dataset overall. Likert ratings rose from 3.0 to 4.2 for writing style and from 3.0 to 3.3 for content realism (the baseline content rating in the table is 3.3, schema 4.3), both on a 1–5 scale.

• Coverage improved substantially. Schema-generated datasets covered 0.23 of enumerated constituent-factor combinations versus 0.16 for the baseline — both under 25%, but a meaningful relative gain. The baseline entirely missed contexts experts considered important, such as interpreting student behavior through a child's "brainstate," a framing rooted in the organization's shared training.

• Content realism improved. Quantitative content realism rose from 0.73 to 0.77. Experts attributed this to the schema examples being more "socially-complex," involving multiple interconnected events and actors, rather than isolated incidents.

• Diversity showed no difference. DCScore was 0.13 for the schema method versus 0.14 for the baseline, and experts could not reliably assess distributional properties from five examples alone.

• Stylistic realism was mixed. Qualitative ratings favored the schema method (4.2 vs 3.0), but the automated metric moved the other direction (0.56 vs 0.60). The authors attribute this to the seed set used for scoring — curated collaboratively by several workers — not reflecting how individual workers actually write.

• The schema method is robust to seed-example scarcity. Baseline content realism and coverage improved as more seed examples were supplied (1 through 16), but schema-generated dataset quality stayed consistent even with zero seed examples, suggesting the schema itself encodes sufficient guidance.

• Different schema fields drive different quality dimensions. Systematized instance and constituents (especially actors and observations) drove coverage, raising it by as much as 12% when present. The deployment population field contributed most to content realism (+11.87% on average) despite being one of the shortest fields to complete. Capability and systematized concept mattered for defining the evaluation but did not need to be passed to the generator.

• Seed examples create a coverage trade-off. Counterintuitively, including seed examples reduced coverage by an average of 3.44%, likely because models anchor on the provided examples and fail to explore the full constituent space.

• Depth beats breadth for central constituents. Providing detailed values for the constituents most central to the deployment context is more valuable than specifying all constituents at equal depth; less detailed constituents can still be scored via high-level or post-hoc categories.

Methodology in Plain English

The researchers first built a structured questionnaire — the schema — that domain experts fill out to describe their evaluation context: who will use the system, what capability or risk is being measured, how that capability is broken into component parts, what every example must contain, what varies across examples, and a handful of real prompts written by practitioners.

They then defined four ways to score a generated dataset. Coverage measures whether the dataset hits enough examples of each combination of component factors, using a smooth penalty so that total absence of a group hurts most and partial representation is interpolated. Diversity uses DCScore, an embedding-based method that measures how many semantically distinct examples a dataset contains. Content realism embeds generated and seed examples and computes a Sinkhorn distance between the two distributions, normalized against two anchors — how far apart seed examples are from themselves, and how far they are from random noise. Stylistic realism measures the average cosine distance between each generated example and all seed examples in a style-focused embedding space.

To generate data, they wrote a general system prompt (domain-agnostic) plus a domain-specific user prompt containing the filled schema, and asked the model to output CSV rows with both the example text and its constituent labels. They compared this against a few-shot baseline: a two-sentence domain overview plus a small number of expert-written seed examples. Generation used GPT-5.4 mini, GPT-5.5, Claude Haiku 4.5, and Claude Sonnet 4.6 at temperature 0.7.

For validation, they ran a think-aloud study with six social workers from a US school social work organization, who reviewed five examples each from the baseline and the schema method, rated style and content realism on Likert scales, and explained their reasoning. The first author then did bottom-up thematic coding of the transcripts. Finally, they ablated the pipeline: varying the number of seed examples from 1 to 16, and testing all subsets of schema fields (1,013 configurations) to measure each field's marginal contribution.

Why This Matters

This paper reframes benchmark construction as a measurement validity problem rather than a data quantity problem, and shows that a relatively small amount of structured expert input can substitute for large amounts of manual dataset curation. It gives practitioners a concrete vocabulary and set of instruments for arguing that an evaluation dataset is the right one for their context — a step beyond simply asserting that domain experts were consulted.

Real-world applications:

• Small, resource-constrained organizations adopting LLMs (nonprofits, clinics, school districts) that need evaluation data tailored to their workflows but cannot afford extensive expert dataset authoring.
• Regulated or sensitive domains such as education, child welfare, and mental health, where evaluating a model on unrealistic or out-of-scope inputs could obscure failure modes or lead to harmful deployments.
• Retrospective auditing of existing benchmarks. Because the schema can be completed at any point, practitioners can apply it to interrogate or compare candidate datasets for a given evaluation need rather than building new ones from scratch.
• Enterprise AI procurement and model selection. Organizations comparing vendor models for a specific internal use case can generate a small context-specific benchmark to distinguish models that universal leaderboards treat as equivalent.

Industry relevance: The framework directly addresses a common enterprise pain point — public benchmarks do not predict performance on a company's actual use case. The ablation results are particularly actionable: teams with limited access to domain experts now have evidence-based guidance on which schema fields to prioritize (deployment population for realism, systematized instance and constituents for coverage) and know that the method tolerates having zero seed examples. The authors also caution that the automated metrics are diagnostics, not substitutes for expert judgment — a restraint worth noting for teams tempted to fully automate evaluation dataset creation.

Future Directions

• Empirical validation beyond the single case study. The authors demonstrate structural domain-independence with two additional unrelated use cases in an appendix and a field-by-field mapping showing existing domain-specific frameworks can be expressed as schema configurations, but leave full empirical study of those domains to future work.

• Fuller treatment of metric assumptions. Both the content and stylistic realism metrics rely on a seed set that represents how the target population writes — an assumption the authors show can fail in practice. Developing metrics that do not depend on a representative seed set, or that explicitly quantify that assumption's violation, remains open.

• Scaling and sample size. Results are reported at n = 100 examples, which the authors acknowledge is small relative to typical NLP benchmarks; they show quality is stable from n = 10 to 100 but do not test larger datasets. Whether the approach holds at benchmark scale is unresolved.

• Extending the criteria set. The four criteria are framed as necessary but not sufficient. Additional validity concerns — such as construct-irrelevant variance or differential functioning across subpopulations — may warrant further operationalized metrics as the literature develops.

Target Audience

AI evaluation researchers and benchmark designers working on context-specific or domain-grounded evaluation; HCI and responsible-AI researchers interested in participatory or expert-in-the-loop dataset design; practitioners in small organizations deploying LLMs who need evaluation data but lack large annotation budgets; and industry ML teams who need to justify why a context-specific evaluation dataset is more appropriate than a general-purpose leaderboard for their deployment decision.

Authors’ abstract

This paper presents an end-to-end approach for generating context-specific large language model (LLM) benchmark datasets by combining expert input with synthetic data generation. Existing benchmark construction methods often trade off validity and scalability: datasets designed with domain experts can produce high-quality evaluations but are slow and costly to create, while synthetically generating data may scale efficiently but often results in unrealistic, redundant, or out-of-scope examples. To address this gap, we introduce a schema eliciting key information about the goals, scope, and context of an evaluation task, and use this information to guide synthetic data generation. We further define four criteria grounded in measurement validity for assessing dataset quality: coverage, diversity, content realism, and stylistic realism. Using these criteria, we show how expert-informed scaffolds can guide synthetic data generation toward more valid benchmarks. Through quantitative evaluations and a real-world case study with domain experts, we demonstrate that our approach improves benchmark data quality over existing methods while preserving validity. We additionally analyze how different types of schema information affect different dataset quality criteria, and provide practical guidance on which information to prioritize collecting under resource constraints.