Clarification, Uncertainty, and Abstention

More than half of the questions real people type into a search engine support more than one defensible answer. That is a count, not an impression. It comes from AmbigNQ, a dataset published in 2020. Min and colleagues took 14,042 questions from the NQ-open benchmark — questions users had actually typed into a search engine — and checked how many admit a single reading. The abstract states the finding flatly: “We find that over half of the questions in NQ-open are ambiguous, with diverse sources of ambiguity such as event and entity references.”

Read that as a statement about production traffic, not about a benchmark. The ambiguity is not exotic phrasing. It is unresolved event and entity references: which event, which entity, which of the several things a name can point at. An agent that resolves those silently is not handling an occasional edge case. On traffic shaped like NQ-open it is applying a guessing policy to the majority of what it receives. And in the log, the guesses it got wrong look exactly like the ones it got right.

The design question this lesson answers is not whether an agent should ever ask. It is which readings are worth stopping for, what the stop costs, and what the runtime does with the answer once it has it.

Agents encounter ambiguity about identity, intent, constraints, authority, and environment state. A clarification policy names which missing facts would change what the agent does. It then asks the smallest useful question. Read that opening line as five things the agent may not know: who is asking, what they actually want, what limits apply, who is allowed to approve, and what the system looks like right now.

Abstention is a valid terminal or intermediate state. It is also a capability in its own right, not a by-product of being accurate, and it has a measured price. SQuAD 2.0 added 53,775 unanswerable questions to the existing SQuAD set, written adversarially by crowdworkers to look like answerable ones. The 2018 paper sets the bar in its abstract: “To do well on SQuAD 2.0, systems must not only answer questions when possible, but also determine when no answer is supported by the paragraph and abstain from answering.”

The price shows up immediately. The best system tested, DocQA + ELMo, scored 85.8 F1 on SQuAD and 66.3 F1 on the new test set. That is 19.5 points. Human accuracy is 89.5 F1. The abstract rounds it: “a strong neural system that gets 86% F1 on SQuAD 1.1 achieves only 66% F1 on SQuAD 2.0”. Nothing about the answerable questions got harder. What was added was the requirement to notice when the evidence supports no answer at all. That alone took a fifth of the score away. A system that cannot carry the cost does not abstain quietly. It converts low confidence into a guess and reports the guess as an answer.

Five moments decide whether an ambiguous goal is resolved or quietly guessed: Detect ambiguity, Estimate impact, Choose interaction, Record the answer, and Resume or stop. Record the answer and Resume or stop should have separate owners, and separate tests. Estimate impact is the step teams skip. It asks whether the two readings of the request would lead to different actions, and only then is it worth stopping to ask. If both readings book the same meeting, the agent should just book it. That step is also the one that has been measured.

Left alone, a language model almost never asks. The 2022 CLAM paper put that default and its obvious correction side by side. Its abstract states the default: “We show that current language models rarely ask users to clarify ambiguous questions and instead provide incorrect answers.” The correction is a baseline that always asks, and it is worse than it looks. It takes a penalty on the unambiguous questions, where the answer would have been right and the question was pure cost. Only selective clarification — the policy that gates on whether the readings diverge — improves adjusted accuracy. Both fixed policies lose. The gate is the whole design.

The capability was never the bottleneck. Prompted to produce a clarifying question, the model produced the correct one for 84%, 99% and 95% of items in Ambiguous TriviaQA, CLAQUA I and CLAQUA II respectively, judged on 100 manually labelled pairs per dataset. Models can write the question. What they do not do is decide to write it, unless Estimate impact exists as a real step with its own threshold.

A material ambiguity leaves three moves: Clarify, Reversible default, and Abstain or escalate. Clarify costs the user a moment. A reversible default is a guess the agent writes down and can undo. Abstaining is stopping without one, and handing the task to someone who can decide. What the three cost is not a matter of taste. SimpleQA is where it has been tabulated.

SimpleQA is 4,326 short fact-seeking questions, published by OpenAI in 2024. Every answer is graded as one of three things: correct, incorrect, or not attempted. That makes abstention a first-class measured outcome rather than a missing row. The abstract sets the standard: “A model with ideal behavior would get as many questions correct as possible while not attempting the questions for which it is not confident it knows the correct answer.”

Two systems in its Table 3 sit at opposite ends of exactly that trade-off. GPT-4o declined to attempt 1.0% of the questions and was outright wrong on 60.8% of them. Claude-3.5-Sonnet declined 35.0% and was wrong on 36.1%. Their F-scores are near-identical, 38.4 against 35.0, reached by opposite policies. One system converts nearly all of its uncertainty into confident wrong answers. The other converts a third of it into an explicit non-answer. The headline number cannot tell them apart.

Which one you want is not a question about model quality. It is a question about what a wrong action costs in your workflow. The paper offers a way to price it: a penalty metric where correct = 1, not attempted = 0 and incorrect = −p. At p = 9, none of the models it evaluates would score positive.

That is the real content of the three options. Clarify and Abstain buy down the 60.8% column, and pay for it in latency and coverage. A reversible default is only honest when the undo is real. An irreversible action taken on a guess is scored at −p, whatever the log says.

Pick a workflow where the agent already guesses when the request is vague. Then write its clarification policy. It lists the fields that must be certain before the agent acts, the wording it uses when one of them is not, and what it does while it waits. Written well, the policy makes the pressure visible. It shows where the agent has been rewarded for guessing quietly, and it leaves a record of the cases where it stopped and asked before committing.

List ambiguous fields and Set materiality thresholds are the cheap steps. Design concise questions is the one with a measured payoff. Qulac, built in 2019 out of TREC Web Track data from 2009–2012, is over 10,000 question-answer pairs covering 198 TREC topics and 762 facets. Its abstract reports what one question is worth: “Our experiments on an oracle model demonstrate that asking only one good question leads to over 170% retrieval performance improvement in terms of P@1, which clearly demonstrates the potential impact of the task.”

That is the number the policy trades against. One bounded, well-chosen question moved precision at rank 1 by more than 170% on an oracle model. No amount of additional inference over the same ambiguous request can reach that. The missing information is not in the request. Support reversible progress keeps the latency from being dead time: stage the draft, gather the neutral evidence, and leave the committing action behind the answer. Measure behavior then has something to count on both sides — unnecessary questions, missed clarifications, and harmful guesses — instead of only the completions.

A strong agent is not one that always acts. It is one that chooses action, question, escalation, or abstention according to evidence and consequence. A team that takes this seriously tracks both sides: how often the agent asked when the request was genuinely unclear, and whether anything in its scoring quietly rewards it for not asking at all.

This is not a research posture. A regulator has already authorised an autonomous AI whose defined output set includes an explicit refusal to decide. The US FDA granted IDx-DR De Novo classification on 11 April 2018. In its pivotal trial, published later that year in npj Digital Medicine, 900 participants were enrolled between January and July 2017 at 10 primary-care sites. For each patient the system returned one of three states: mtmDR detected, mtmDR not detected, or insufficient image quality. It reached sensitivity 87.2% (95% CI 81.8–91.2%), specificity 90.7% (95% CI 88.3–92.7%) and an imageability rate of 96.1% (819/852).

The third state is the one to look at. “Of the 852 participants that received a completed FPRC grading, 33 participants (4%) received an insufficient image quality output from the AI system after completion of the AI system protocol.” Those 33 were not the easy cases. Disease prevalence among them was 10/33 (30%), against the 23.8% the paper reports for the fully analysable set. The abstentions fell on patients as sick as everyone else. That is precisely why converting them into a guess would have been the expensive move.

Two widely used frameworks build the same pause into the runtime itself. LangGraph's interrupt function stops the graph where it is called and saves the state through a checkpointer: “When an interrupt is triggered, LangGraph saves the graph state using its persistence layer and waits indefinitely until you resume execution”. The prerequisite is stated just as plainly: “To use interrupt, you need: 1. A checkpointer to persist the graph state”. Work resumes by “invoking it again with a Command that contains the resume value”. Microsoft's AutoGen is a separate implementation by a different organisation, and it ships the same primitive: “When UserProxyAgent is called during a run, it blocks the execution of the team until the user provides feedback or errors out.” A question is a runtime state here, not a conversational flourish. The graph is stopped, the work so far is saved, and nothing moves until a person answers.