Don't Guess, Ask Wisely: Benchmarking and Training Proactive Elicitation in Open-Domain LLM Conversations

AI summary

Overview

Research area: Natural Language Processing — conversational LLM behavior, clarification/question-asking, and reinforcement learning for dialogue.

Technical level: Intermediate. The paper is readable without deep math, but the training method assumes familiarity with reinforcement learning, specifically GRPO-style group-relative advantage estimation and reward shaping.

Scope (1 sentence): The paper builds a benchmark (VagueBench) for measuring whether LLMs ask clarifying questions when user queries are underspecified, and proposes an offline RL training method (DialogGRPO) that teaches models both when to ask and what to ask.

What This Paper Is About

Large language models default to answering, even when a user's query is missing critical information, so they quietly guess at unstated intent and constraints instead of asking. The authors quantify this across seven models from 8B parameters to frontier scale and find that fewer than 11% of underspecified real-world queries trigger a follow-up question. Their goal is twofold: build a benchmark with precisely known "hidden information" so asking behavior can be measured fairly, and train models to ask targeted questions without over-asking when a query is already complete.

Key Contributions

1. VagueBench benchmark. A benchmark built from real open-domain queries from a commercial AI assistant, where each seed is annotated with structured intent and detail units and then systematically degraded into an intent-vague and a fully-vague variant. Because degradation is controlled, the hidden information set for every instance is exactly known.

2. A three-metric evaluation protocol. The benchmark scores decision appropriateness (Ask%), elicitation efficiency (IAR, information units recovered per asking turn), and answer coverage (Cov. F1 against the gold information set), using a simulated user that releases a hidden unit only when the model asks a semantically matching question.

3. DialogGRPO training method. An offline reinforcement learning method that organizes multi-turn elicitation as a dialogue-state directed acyclic graph (DAG). Candidate questions at each node form a group for relative comparison, and each edge gets a dense turn-level reward combining immediate information gain with lookahead value — enabling per-step credit assignment with no runtime user simulation.

4. Empirical findings and transfer. A demonstration that elicitation suppression persists across model families and scales, that DialogGRPO beats prompting, test-time scaling (UoT), and SFT baselines, and that the learned behavior transfers zero-shot to MultiWOZ 2.4 and AmbigQA.

Main Findings

• Elicitation is suppressed everywhere. On the seven evaluated models (Qwen3-8B, Llama-3.1-8B-Instruct, Qwen3.5-27B, GPT-oss-120B, GPT-5.4, Claude Sonnet 4.6, Gemini 3.1 Pro), Ask% on intent-vague queries stays at or below 9% and on fully-vague queries never exceeds 11%. Open-source models scored 0.8–2.0 on intent-vague; closed-source models ranged from 1.6 to 8.8, with no consistent advantage for larger models.

• Humans disagree sharply with the models. In a human study of 100 sampled queries, annotators chose to ask on 96–98% of intent-vague queries and 100% of fully-vague queries — indicating the gap is behavioral suppression, not genuine ambiguity in the data.

• Prompting unlocks latent ability but plateaus. A single elicitation-encouraging prompt lifted Qwen3.5-27B's Ask% from 1.6% to 38.4% (intent-vague) and from 2.0% to 47.2% (fully-vague), but IAR stayed low at 0.22 and 0.38, showing prompting shifts the ask/answer boundary without improving per-turn efficiency.

• DialogGRPO asks better, not just more. It reached 88.4% Ask% / 0.64 Cov. F1 / 0.76 IAR on intent-vague and 91.2% / 0.67 / 0.87 on fully-vague, versus SFT's 82.0 / 0.60 / 0.64 and 85.6 / 0.63 / 0.80. Against the untrained Qwen3.5-27B baseline (0.49 and 0.38 Cov. F1), the coverage gains reflect answering on recovered rather than assumed information.

• No over-asking on complete queries. On information-complete seeds, DialogGRPO asked on only 5.2% of samples (versus the base model's 1.6%), while lifting Cov. F1 from 0.58 to 0.74. Prompt and UoT pushed Ask% to 42.8% and 56.4% respectively, interrupting roughly half of already well-specified queries.

• UoT's bottleneck is triggering, not efficiency. Its per-turn recovery was reasonable when it did ask (IAR 0.45 and 0.41), but its uncertainty heuristic triggered elicitation on only about 60% of vague queries.

• The two reward terms do different jobs. Removing the lookahead term dropped Cov. F1 by 0.06–0.07 while IAR stayed nearly intact; removing the immediate term left Cov. F1 largely unchanged but cut IAR by up to 0.08.

• Zero-shot transfer holds. On AmbigQA, Qwen3.5-27B went from 1.1% Ask% / 2.1% Ask F1 to 16.3% / 26.3% with DialogGRPO; Gemma3-27B went from 1.1% / 2.2% to 34.4% / 44.7%. On MultiWOZ 2.4, success went from 11.4% to 24.7% (Qwen) and from 15.1% to 17.8% (Gemma). For reference, GPT-5.4 scored 39.9% success on MultiWOZ and 5.1% Ask% / 9.6% Ask F1 on AmbigQA; GLM-5 scored 23.8% and 0.9% / 1.9%.

• The recipe transfers across base models. Re-running the full pipeline on Gemma3-27B with identical hyperparameters produced comparable Ask%, Cov. F1, and IAR lifts. Before training, both 27B models showed the same suppression pattern (Ask% below 3% on both splits).

• General capability is preserved. On six benchmarks spanning coding, math, commonsense reasoning, and general knowledge, DialogGRPO matched or slightly exceeded the untrained Qwen3.5-27B (Appendix E).

Methodology in Plain English

Building the benchmark. The authors took over 100,000 real queries from a commercial AI assistant and filtered them down to 4,930 information-complete seeds (about 5% of the raw pool). Filtering removed unsafe or malformed queries (48.7%), greetings and context-dependent utterances (39.3%), queries too detail-sparse to degrade (6.1%), and annotation failures (0.9%). Each seed averages 1.36 intents and 4.63 details. Annotators — two frontier LLMs (GLM-5) with GPT-5.4 as adjudicator, validated against humans at κ ≥ 0.81 — marked "clarity anchors": linguistic cues split into an intent layer (predicate verbs, speech-act frames) and a detail layer (output-type and scope markers). Degrading a query then becomes a principled removal operation: strip only intent anchors to get an intent-vague variant (which still preserves 77.0% of information units on average), or strip intent anchors plus detail anchors down to a 5–10 token topic phrase for a fully-vague variant (18.7% coverage). The test set is 250 complete queries plus both variants (750 instances); the remaining 4,680 seeds and their DAG expansions form the training set.

Measuring asking behavior. A simulated user holds the complete query and reveals a hidden unit only when the model asks something semantically matching it, deflects irrelevant or repeated questions, and the dialogue ends when the model answers, all units are recovered, or 10 turns elapse. This makes evaluation reproducible.

Training. For each training query, an external strong LLM (GLM-5.1, distinct from the policy) proposes candidate questions at each dialogue state, expanding a tree that is then merged into a DAG wherever different asking orders reach the same acquired-information set. Each edge gets a reward: an immediate term rewarding questions that recover more units (capped so one high-gain question can't dominate), plus a lookahead term estimated by sampling paths through the pre-built graph and averaging their end-of-trajectory returns. Questions that recover nothing get a penalty relative to the parent's mean lookahead. All candidates at a node form a comparison group, and normalized within-group advantages weight a simple log-likelihood regression objective — no importance ratios or clipping needed, since candidates are pre-generated and fixed. The policy never sees the gold units; it learns to discriminate good questions through the relative advantage signal rather than by imitating the external model.

Why This Matters

Impact on research. The paper reframes clarification from disambiguation (picking among multiple valid readings) to elicitation (recovering information that was never expressed at all), and it isolates a capability that appears present in pretrained weights but suppressed by post-training pipelines that reward direct answers. It also offers a cheaper alternative to on-policy RL with user simulators: precomputing a dialogue-state graph turns trajectory-level feedback into dense per-turn credit assignment.

Real-world applications:

• Customer support and assistants — asking one targeted question about budget or use case instead of guessing when a user says "recommend a laptop."
• High-stakes advisory domains — the paper cites medical diagnostic accuracy falling from 94.9% to 34.5% when symptoms are underspecified, and task accuracy dropping 39% when instructions are fragmented.
• Task-oriented dialogue systems — slot-filling agents that must decide when a slot genuinely needs to be asked for.
• Ambiguous question answering — search and QA systems that surface a clarifying question before committing to an interpretation.

Industry relevance. Deployed assistants already receive underspecified traffic at scale (over half of the 5,000 analyzed queries lacked critical information), yet scaling from 8B to over 120B parameters did not fix the behavior. The method is offline, avoids runtime simulation cost, and the calibration result — asking on only 5.2% of complete queries while asking on over 88% of vague ones — is precisely the trade-off product teams need. The authors note asking frequency is controlled by reward hyperparameters and should be tuned per application.

Future Directions

• Broaden the data foundation. VagueBench comes from a single commercial assistant's traffic and is English-only, so coverage of conversational scenarios, languages, and modalities is limited; extending it is an open task.
• Model more realistic users. The simulated user cooperatively releases information on matching questions, abstracting away partial answers, topic switches, noise, and refusals — so reported gains likely represent an upper bound for cooperative interlocutors.
• Densify the dialogue graph. Open-domain dialogue spaces are unbounded, so the DAG covers only a sparse set of trajectories; interactions outside the training distribution remain a limitation despite local contrastive coverage and state merging.
• Handle dynamic preferences. VagueBench fixes the hidden preference set for reproducibility, but in domains like shopping, preferences can emerge or change mid-conversation — behavior the current setup does not capture.
• Constrain what may be solicited. Because proactive elicitation is dual-use, the authors recommend deployment guardrails: limiting what the model can ask for, auditing questions touching sensitive attributes, and giving users a clear way to decline.

Target Audience

Researchers and engineers working on conversational LLM behavior, clarification and mixed-initiative dialogue, and reinforcement learning for multi-turn agents. It is most useful to those building production assistants who need a measurable definition of "when should the model ask," and to benchmark designers interested in controlled-degradation methodology for constructing test sets with known hidden information. Readers wanting to reproduce or extend the training recipe will need working knowledge of GRPO-style policy optimization; readers interested only in the evaluation findings can read the benchmark and results sections alone.

Authors’ abstract

When user queries are incomplete, large language models should ask targeted questions before committing to an answer, a capability we call proactive elicitation, yet evaluation on real user-assistant queries across seven LLMs (8B to frontier) shows that fewer than 11% of underspecified queries trigger a follow-up question. To benchmark and train this capability, we introduce VagueBench, where each seed query is annotated with structured intent and detail units then degraded into intent-vague and fully-vague variants via controlled removal of linguistic cues, yielding known hidden information sets for evaluating both when and what to ask. We propose DialogGRPO, an offline reinforcement learning method that organizes multi-turn elicitation as a dialogue-state directed acyclic graph, where candidate questions at each node receive dense turn-level rewards combining immediate information gain with lookahead value through group-relative comparison, enabling fine-grained credit assignment without runtime simulation. DialogGRPO elicits missing information on over 88% of vague queries with substantial gains in both recovery efficiency and answer coverage, without over-asking on complete queries, outperforms prompting and test-time-scaling baselines, and generalizes zero-shot to MultiWOZ 2.4 and AmbigQA.