A Contextual Measure of Ambiguity in Prescriptive Text

AI summary

Overview

Research area: Natural Language Processing, with ties to computational social science, LLM simulation, and legal/policy analysis. Technical level: Intermediate — the core measure is a short, self-contained formula, but evaluating it involves LLM pipelines, human-subject experiments, and deliberation datasets. Scope: The paper defines and validates DELTA, a method that scores the ambiguity of prescriptive text (laws, rules, policies) by measuring how much a diverse pool of raters disagrees about whether the policy permits or prohibits a set of concrete cases, including an LLM-simulated version called LLM-DELTA.

What This Paper Is About

Ambiguity in rules and policies can be useful (it lets rules adapt to new circumstances) or harmful (it invites arbitrary enforcement and disputes over meaning). Existing ways of measuring ambiguity judge how a text reads on its own, not how people actually interpret it when applied to real situations. The authors propose measuring ambiguity as disagreement among stakeholders about what a policy means for specific cases, and test whether this can be approximated at scale with language models.

Key Contributions

1. The DELTA framework and score. A formal definition of policy ambiguity as expected disagreement among raters about a policy's consequences, computed via a per-case score d_ij = 4 x̄_ij (1 - x̄_ij) (twice the Gini impurity of pooled verdicts) averaged across cases, bounded in [0,1]. The score decomposes exactly, by the law of total variance, into an intra-rater indeterminacy term and an inter-rater disagreement term.
2. A scalable LLM approximation (LLM-DELTA). A pipeline in which GPT-5 generates challenging cases (by sampling policy pairs) and stakeholder personas (in opposed pairs), expands each persona into a short reflection, and Gemini-2.5-Flash at temperature 1 issues judgments under both positive and negative framings. A revealed judgment x_ijk = (q_ijk + (1 - q'_ijk))/2 captures cases that are neither clearly permitted nor clearly prohibited.
3. Validation across four datasets. A human-subject benchmark (20 university speech policies, 20 real FIRE campus controversies), an adversarial dataset (20 policies, 32 cases, 30 personas, modeled on the 1999 Australian republic referendum), four Pol.is deliberation datasets, and the Habermas Machine experiment (eight topics, ~70 collective statements each).
4. A substantive test of the "ambiguity as compromise" intuition, plus a convergence diagnostic and a public Python package with all data on GitHub.

Main Findings

• LLM-DELTA tracks human disagreement, moderately. On the 20 university free-speech policies, LLM-DELTA correlates positively with human-DELTA, with Spearman ρ = 0.48 reported in the limitations section. Confidence intervals in the main comparison figure come from a stratified rater bootstrap (2,000 iterations), resampling Democrats, Independents, and Republicans separately per policy.
• Lexical and direct-judge baselines do worse. A dictionary-based concreteness measure (Brysbaert et al.'s ~40,000-entry list, 1–5 scale, words covering above 85% of the text, negated so higher means more ambiguous) shows little relationship with human-DELTA. A direct LLM-as-a-judge ambiguity rating fares better than the dictionary but remains weak. A second LLM-as-a-judge prompt that explicitly describes the DELTA mechanism performs poorly, indicating that abstract reasoning about disagreement does not substitute for concrete cases and diverse simulated judgments.
• DELTA survives an adversarial test that fools the baselines. In the constructed setup, DELTA ranks the ten consensus-inducing policies as the ten least ambiguous. Both LLM-as-a-judge variants rank the policies in nearly the reverse order of DELTA.
• Ambiguity does not produce bridging. Across all four Pol.is datasets, LLM-DELTA and a conservative bridging score min(p_A, p_B) show null or negative Pearson correlation at the 20% engagement threshold, and no dataset shows a stable positive association across the thresholds examined.
• Ambiguity softens enthusiasm rather than converting opponents. In the Habermas data, as DELTA rises, the share of "strongly agree" responses drops significantly while "slightly agree" and neutral responses gain a comparable combined amount; the disagree side changes only modestly.
• Scores converge quickly. Using the Pol.is Uber dataset with a reference design of 60 cases and 30 raters, subsampling 100 times per cell shows Spearman correlations with full-design rankings rising rapidly well before the full design size. The authors adopt roughly 30 raters and at most 50–60 cases as a practical stopping rule.

Methodology in Plain English

The authors start from a reframing: applying a policy to a situation is really a yes/no classification task — is this situation permitted or not? Ambiguity, then, is what happens when that classification is contested. For each policy they assemble a set of realistic, debatable scenarios and a diverse set of raters, ask each rater whether the policy permits the scenario and whether it prohibits it, and average the two answers into a single judgment per rater-scenario pair. Disagreement across raters is measured with a normalized version of the probability that two randomly drawn raters would conflict, which peaks when the pool splits evenly.

To scale this up, they replace human labor at three points. GPT-5 writes evaluation cases by being shown pairs of policies and asked for scenarios that comply with one while violating the other, which forces the cases onto the precise points of interpretive friction. GPT-5 also generates stakeholder personas as opposing pairs, and each persona is first expanded into a paragraph about that stakeholder's values on the topic before judging, to avoid the narrow, uniform responses the authors saw when asking models to judge directly. A lighter model, Gemini-2.5-Flash, then issues the actual permit/prohibit verdicts. To check validity, they ran a paid human version on Prolific with a politically balanced sample (30 raters per policy, $6 each, median 18 minutes), built a deliberately adversarial policy set where surface wording and real disagreement point in opposite directions, and applied the measure to two existing deliberation corpora to see whether ambiguous statements attract broader support.

Why This Matters

Impact on research: The paper offers a general, principled operationalization of ambiguity that does not require a hand-built rubric per domain — a contrast with prior work such as Hall and Paschal's clarity rubric, which predicted disputed prediction-market contracts with out-of-sample AUC of roughly 0.75 but had to be engineered domain by domain. Making the cases and rater population explicit also makes ambiguity claims auditable and comparable.

Real-world applications:

• Drafting and stress-testing legislation, contracts, and agency rules against edge cases before enactment.
• Predicting which policy provisions are likely to generate disputes, appeals, or inconsistent enforcement.
• Evaluating whether LLM-drafted constitutions or rule sets are interpretable consistently — related work found different executors of the same constitution agree only 73% of the time.
• Screening citizen-generated proposals in participatory processes for statements that are too indeterminate to guide action.

Industry relevance: Any organization deploying LLMs to write or interpret policy — compliance, HR, platform moderation, AI governance — needs a way to detect rules that will be applied inconsistently. DELTA is a measurement tool, not a generator, and the authors position LLM-DELTA as decision support for early-stage drafting rather than a replacement for real stakeholder consultation.

Future Directions

• Improve rater simulation and case generation, and test whether the human–LLM correlation holds beyond a single English-language domain with 20 policies, particularly for domains underrepresented in pretraining data.
• Extend the human benchmark to the Pol.is and Habermas settings, where validity is currently supported only indirectly.
• Investigate whether adversarial constructions designed specifically against DELTA can break it, since the paper's robustness result only shows baselines failing on a setup built to expose their weaknesses.
• Test whether DELTA scores predict downstream implementation conflict in real policy settings, moving beyond the correlational bridging analysis, which cannot rule out confounds such as statement length or selection effects.

Target Audience

Researchers and practitioners working on LLM evaluation, computational social science, deliberative democracy tools, and AI governance or policy drafting. It suits readers comfortable with basic statistics (variance decompositions, rank correlation) who want a concrete, reusable metric rather than a new model, and it is directly useful to anyone who needs to justify why a rule's wording will or will not cause interpretive conflict.

Authors’ abstract

Ambiguity in prescriptive text such as laws, contracts, rules, policies, or recommendations can be both desirable (e.g., if it enables a law to adapt to changing context, such as new technologies) and undesirable (e.g., if it leads to arbitrary discretion or conflict over the law's interpretation). To help navigate these trade-offs, it would be valuable to have methods for quantifying ambiguity that are predictive of such outcomes, but existing measures capture how text is perceived in isolation rather than how it is interpreted in context. In this paper, we propose quantifying ambiguity as the degree of disagreement among a set of stakeholders ("raters") on how alternative prescriptive texts ("policies") apply in each of a set of relevant scenarios ("cases"). Using two policy domains, we present preliminary analysis suggesting that, even when using simulated raters, this approach (i) correlates with real human disagreement, and (ii) is robust to spurious text features. We then apply the measure to existing deliberation datasets to describe the association between policy ambiguity and support for those policies.