AREAs-Lab: An Interactive Environment for AI-driven Requirement Elicitation for AI Systems

AI summary

Overview

Research area: Human-Computer Interaction / NLP — interactive AI agents, requirement elicitation, LLM-based benchmarking and evaluation.

Technical level: Intermediate. The paper is readable without deep technical background, but it assumes familiarity with LLMs, prompt-based agents, and standard classification metrics (Precision, Recall, F1).

Scope: The paper introduces AREAs-Lab, a controlled, domain-agnostic testbed and synthetic benchmark for studying how an AI assistant can turn vague user goals into complete, actionable task requirements for AI systems by interrogating both the user and the underlying dataset.

What This Paper Is About

Building AI systems requires precise task requirements, but users typically state their goals without the data-specific constraints, preferences, and edge cases that determine whether the resulting system actually works. This is harder than traditional software requirements because AI behavior depends on both human intent and the statistical properties of the data. The paper builds an interactive environment, a synthetic benchmark, and an automated evaluation pipeline to systematically study how well AI assistants can close this specification gap.

Key Contributions

1. AREAs-Lab environment: A domain-agnostic interactive environment for AI system requirement elicitation that, unlike existing frameworks, prioritizes simultaneous interaction between the user and the underlying data.

2. A synthetic benchmark with information asymmetry: 16 public data sources spanning 10 distinct domains, yielding a final set of 151 (data source, user, task) tuples. Each instance contains a "skeletal" starting requirement (approximately 30 words) plus a hidden full reference requirement and user profile.

3. An automated, scalable evaluation pipeline: An AI-simulated user that discloses hidden information only when appropriately prompted, combined with an atomic-unit matching metric that computes Precision, Recall, and F1 against the reference requirement.

4. Extensive empirical evaluation and human assessment: Comparisons of five elicitation strategies across three backbone models, plus a study with 12 human evaluators and a comparison of simulated versus real user responses.

Main Findings

• All interaction strategies beat no interaction. Across Claude-Sonnet-4.6, GPT-5.4, and Gemini-3.1-Pro, every interactive strategy outperformed the No Interaction baseline on average F1.

• Hybrid Interaction is strongest. Combining adaptive user clarification with direct data inspection achieved the highest average F1 for all three backbones: 40.76 (Claude-Sonnet-4.6), 40.07 (GPT-5.4), and 41.64 (Gemini-3.1-Pro).

• The task remains hard. The best overall F1 was only 41.64, which the authors attribute to the benchmark's deliberate information asymmetry and the strict atomic evaluation metric that may not credit vague, partial, or implicit matches.

• Fixed vs. adaptive questions trade precision for recall. User Interaction (Fixed) consistently achieved higher recall (by 4.33–5.43 points) than the adaptive variant, while adaptive questions generally achieved higher precision.

• Adaptive questioning beats direct data access on data-derived requirements. For data-derived requirements, User Int. (Adaptive) reached an F1 of 40.06 versus 29.87 for User Int. (Fixed) and 27.58 for Data Interaction — suggesting assistants can reason about likely data characteristics and ask about them even without inspecting the data.

• Data Interaction plateaus. Over ten interaction rounds on ArXivSum tasks, average F1 for Data Interaction remained largely stagnant, because it often extracts features irrelevant to the user's core needs. Strategies involving user feedback (User Interaction and Hybrid Interaction) showed consistent average improvement, though individual tasks fluctuated and prolonged interaction could introduce semantic noise, requirement drift, or conflicting constraints.

• More collaborative users yield better requirements. Under Hybrid Interaction, performance improved monotonically from Passive to Normal to Active simulated users, evaluated on a subset of four datasets (MultiNews, UK-Legislation, Asclepius, and ESConv) due to resource constraints.

• Human evaluators rated the assistant well on coverage and quality. Mean scores of 4.36 for Coverage, 4.36 for Requirement Quality, and 4.00 for Experience. Inspiration lagged at 3.73 with wider variance.

• Human-written requirements were often less complex than synthesized ones, frequently lacking detailed data constraints and precise specifications, and evaluators often raised unexpected elements during interaction that they had not initially considered.

• Simulated users are a useful but imperfect proxy. Combined "Similar" and "Partially Similar" classifications were 63% (Active), 59% (Normal), and 68% (Passive). The Passive style had the lowest "Different" rate at 32%, with simulated response length closely matching humans (24 vs. 17 words). Still, 32%–41% of simulated responses were classified as Different.

• Variance increases with interaction. All interaction-based strategies showed higher F1 variance than No Interaction, reflecting the introduction of irrelevant details, redundant specifications, or conflicting constraints.

Methodology in Plain English

The researchers did not use the 16 public datasets directly as benchmark tasks. Instead, they used them as grounding material in a four-stage synthesis pipeline:

1. Extract data features. An LLM pulled schema features (domain context, data types, column roles) from dataset metadata and instance features from 100 randomly sampled instances per dataset.
2. Create user personas. Five realistic personas were generated per data source, each with a skill profile, a primary business objective, and domain challenges.
3. Write task requirements. For each persona, two tasks were generated at different difficulty levels — Medium (interpretive reasoning) and High (resolving conflicting constraints or ambiguity). Each full requirement specified execution steps, data-weighting instructions, and an evaluation rubric. It was then condensed into a roughly 30-word informal variant used as the agent's starting instruction.
4. Validate quality. Two annotators manually audited random samples of 30 instances per pilot run, and two high-capability LLMs independently scored persona–task triplets on six dimensions (including Role-Schema Alignment, Reasoning Depth, and Data Grounding) on a 1–5 scale. Instances scoring below 4.0 on average overall, or below 3.0 on any critical dimension, were discarded, leaving 151 tuples.

Evaluation works like this: a simulated user opens the conversation with the simplified requirement. The AREAs assistant may ask questions and/or inspect data, depending on the strategy. Afterward, it produces a final requirement. Both the generated and reference requirements are decomposed into atomic units, and an LLM matching function aligns them semantically to compute Precision, Recall, and F1.

The five compared strategies are No Interaction, Data Interaction (one round), User Interaction (Fixed, ten predefined questions, ten rounds), User Interaction (Adaptive, ten rounds), and Hybrid Interaction (adaptive user interaction plus data inspection, ten rounds). Simulated users adopt one of three collaboration styles — Passive, Normal, or Active — and are forbidden from volunteering requirement details unless asked a relevant question.

Why This Matters

Impact on research: The paper reframes requirement elicitation for AI systems as a distinct problem from traditional software requirements, where specifications are stochastic and coupled to data. It provides an open-sourced, reproducible testbed and surfaces a concrete capability gap — even frontier models top out near 41.64 F1 — that future work can target.

Real-world applications:

• Building automated audit or traceability tools over regulated document collections such as medical literature or legislation.
• Drafting specifications for summarization, classification, or extraction pipelines where column schemas and data artifacts (for example, tokenization anomalies) determine success.
• Supporting non-technical domain experts — clinicians, financial analysts, legal reviewers — who can state a goal but not the data constraints behind it.
• Internal enterprise AI intake processes, where an assistant could interview stakeholders and propose a first-draft specification for review.

Industry relevance: The authors are affiliated with Capital One's AI Foundations group alongside academic collaborators, and the paper's risk section addresses enterprise concerns directly — PII exposure during data inspection, requirement drift, hallucinated compliance constraints, automation bias, and the need for human approval and audit logs before specifications pass to implementation agents.

Future Directions

• Closing the simulated-to-human gap. Larger, more diverse randomized human studies are needed, since substantial behavioral differences remain and LLM proxies cannot reproduce shifting attention, emotional friction, inconsistent preferences, or conversational fatigue.
• Multimodal elicitation. The benchmark is limited to text-based and structured data, while real elicitation often involves visual mockups, UI diagrams, and complex database schemas.
• Reducing LLM coupling. Both the elicitation agent and the simulated user depend on context window limits and instruction-following stability; in ultra-long dialogues the simulator may suffer information decay or hallucination, adding variance.
• Governance-aware evaluation. AREAs-Lab does not evaluate role-based data access, PII redaction, organizational approvals, or compliance review, and synthetic instances do not reproduce the legal, organizational, privacy, and security constraints of high-stakes deployments.

Target Audience

Researchers and practitioners working on LLM agents, human-AI interaction, conversational requirement elicitation, and AI system specification. It is also relevant to product and engineering teams in regulated enterprises who need to understand where an AI elicitation assistant helps, where it still fails, and what governance guardrails are required before deploying one. Readers needing deep technical background in model training will not find it here; the paper is primarily about environment design, benchmark construction, and evaluation methodology.

Authors’ abstract

Building effective AI systems increasingly depends on writing high-quality task requirements, yet users often struggle to articulate the constraints, preferences, and edge cases that determine success. This problem is especially acute in AI development, where behavior is shaped not only by human expectations but also by data characteristics. We present AREAs-Lab, an interactive environment for AI-driven Requirement Elicitation for AI systems. In AREAs-Lab, an assistant iteratively refines an initially incomplete requirement by analyzing the underlying dataset and asking targeted clarification questions to uncover the user's latent intent. To study this setting systematically, we construct a synthetic benchmark grounded in 16 public datasets spanning diverse domains and task types. Each benchmark instance includes a user profile, a complete reference requirement, and an intentionally underspecified version that serves as the assistant's starting point. We further introduce an automated evaluation pipeline based on an AI-simulated user that reveals hidden information only when appropriately prompted, enabling scalable and reproducible assessment of interactive elicitation quality. AREAs-Lab provides a controlled testbed for studying how AI assistants can transform vague user goals into actionable requirements for AI systems.