Enabling Fine-Grained Operating Points for Black-Box LLMs

AI summary

Overview

Research area: Black-box LLM decision making, confidence elicitation, black-box uncertainty estimation, and operating-point analysis using precision-recall (PR) and Receiver Operating Characteristic (ROC) curves.

Technical level: Intermediate. The paper assumes familiarity with binary classification, confusion matrices, precision/recall, ROC/PR curves, and calibration, but its arguments are presented without heavy mathematical prerequisite beyond the formal definition of "operational granularity."

One-sentence scope: The paper diagnoses why black-box LLMs produce only a handful of distinct verbalized probability values, and proposes three lightweight methods — one unsupervised and two supervised — that expand those scores into thousands of distinct values so practitioners can pick fine-grained operating points without losing predictive performance.

What This Paper Is About

Black-box LLMs served behind APIs can act as classifiers, but when asked to output a probability for their decision they tend to return only a small set of distinct values (16–23 unique scores across 11 datasets in this paper). Because the number of available thresholds depends on the number of unique scores, this leaves users with only coarse control over operational metrics such as precision, recall, and false positive rate — problematic when an application needs, for example, precision ≥ 95%. The goal is to increase the number and diversity of available operating points from black-box LLM outputs without degrading classification performance or inflating inference cost.

Key Contributions

1. Demonstrating the prevalence and impact of low cardinality in LLM verbalized scores through experiments on 11 binary classification datasets and 3 LLMs, showing that verbalized probabilities produce orders-of-magnitude coarser operational curves and lower-cardinality distributions than token probabilities.
2. Providing a hypothesis for this behavior with supporting experiments: LLMs exhibit a "rounding bias" toward rounded but informative verbalized probabilities (numbers ending in "0" and "5"), consistent with human cognitive bias toward round numbers.
3. Offering simple, efficient solutions — Ours-Unsup, Ours-Sup-1call, and Ours-Sup-2call — that outperform state-of-the-art confidence elicitation and uncertainty estimation techniques on the task of increasing cardinality (operational granularity) while maintaining comparable or better predictive performance.
4. Showing that prompting alone is not sufficient: increased prompt specificity does raise cardinality and granularity but has an unpredictable impact on performance, and the authors state that model fine-tuning may be necessary to fundamentally modify LLM scores (which is outside their scope).

Main Findings

• Verbalized probabilities have low cardinality. Across the 11 datasets combined, Claude (verbalized) had 16 unique predictions, Nova 23, and Qwen 21, while Qwen's token probabilities yielded 41,671 unique values.
• Coarser operational granularity from verbalized scores. Measured with the paper's granularity definition, verbalized scores gave g_pre of 0.080 (Claude), 0.085 (Nova), and 0.128 (Qwen), versus 0.001 for Qwen's token probabilities. Corresponding g_rec values were 0.450, 0.424, 0.219, and 0.028, and g_fpr values were 0.191, 0.270, 0.194, and 0.028.
• Higher cardinality does not always mean finer granularity. The authors note that many unique predictions may densely populate one subregion and leave the rest of the space sparse.
• Rounding bias is the proposed explanation. All three LLMs most often ended their verbalized probabilities with "0" and "5" tokens regardless of task, consistent with prior observation that verbalized confidences usually fall in the 80–100 range and are expressed in multiples of 5. In the third character position, distributions spread across all digit tokens with slight bias toward extremes.
• Increased prompt specificity raises cardinality but can hurt performance. For Qwen, low-specificity prompting gave 28 unique values (PRAUC 0.775, AUROC 0.792); medium gave 89 (PRAUC 0.713, AUROC 0.748); high gave 179 (PRAUC 0.633, AUROC 0.663). Granularity improved alongside cardinality, but AUCs dropped.
• Prompting variants mostly fail to help. Minimized cardinality changes were observed when prompting for scores in [0, 100], combining coarse and fine scores, generating 20 scores in one output and aggregating, or supplying 20 example scores in context. Two-stage prompting approaches from prior work (2 LLM calls per instance, with and without chain-of-thought) also stayed at 15–16 unique values.
• Proposed methods expand cardinality by nearly three orders of magnitude. On the combined dataset, Ours-Unsup reached 18,673 unique values in 1 call, Ours-Sup-1call 20,596 in 1 call, and Ours-Sup-2call 20,607 in 2 calls, versus 15 for Prompt-Naive, 97 for Sample-Class (20 calls), and 2,291 for Sample-Prob (20 calls).
• Granularity improves along all three axes. Ours-Sup-1call and Ours-Sup-2call achieved g_pre 0.016, g_rec 0.001, and g_fpr 0.004–0.005, compared with Prompt-Naive at 0.078, 0.451, and 0.207.
• Supervised methods shine on the combined, diverse dataset. On the combined dataset, Ours-Sup-2call improved mean PRAUC by 8 percentage points over Prompt-Naive and 2.6 percentage points over the 20-sample Sample-Prob approach, while needing only 2 samples per instance at inference (10× more efficient than Sample-Prob).
• On individual datasets, no method dominates. Aggregated results across the 11 datasets (5 runs, 2σ error bars) show that while naive prompting is usually worse, no method consistently outperforms the rest; the supervised methods did not provide significant boosts because most datasets are small (five have as few as 250 samples) and their PR curves are mostly monotonic.
• Ours-Unsup is competitive despite its simplicity. It performed similarly to approximated token probabilities on Nova and Qwen while achieving comparable or higher density in the operational curves on individual datasets.
• Joint noise-and-calibration learning is better. Ablation studies (Appendix A.7) show that jointly learning to introduce noise and correct miscalibrated behavior outperforms conducting these steps separately.

Methodology in Plain English

The authors start by treating a black-box LLM as a classifier: the model is prompted to output a probability between 0 and 1 for the positive class, no weights, logits, or token probabilities are accessible, and the model is frozen (no fine-tuning). Because only the distinct output values can serve as decision thresholds, the number of distinct values determines how finely precision, recall, and false positive rate can be adjusted — what the paper formalizes as operational granularity, the cell size of the narrowest uniform grid that contains at least one operating point in every cell.

They first measure this problem across 11 combined binary classification datasets using three models: the black-box Anthropic Claude 3.0 Sonnet, the black-box Amazon Nova Pro 1.0, and the white-box DeepSeek R1 Distill-Qwen-32B. For Qwen they can also read the token probability of the decision token, giving a high-cardinality reference point. They plot PR and ROC curves, count unique predictions, and inspect character-by-character distributions of the generated digits to check the rounding hypothesis. They also test prompt-specificity variants (low, medium, high).

Their fix is to add controlled noise to the verbalized scores rather than replace them. Ours-Unsup adds uniform noise z ~ U(0,1) scaled by a weight w, clips the result to [0, 1], and picks the largest w that does not change the relative ranking of predictions — because ranking is what determines PR and ROC curves, this interpolates the existing curve without altering performance. Ours-Sup-1call turns this into a supervised problem: a small ReLU network (2 hidden layers of width 8) takes only the verbalized probability and Gaussian noise and outputs a corrected score through a sigmoid, with a λ-weighted term that pushes entropy up by shrinking w. It never sees input features, so as w grows it reduces to a calibrator. Ours-Sup-2call accepts two verbalized probabilities (temperatures 0 and 1), is trained on 20 samples per instance, and needs only 2 samples at inference. Baselines include naive prompting, two-stage and chain-of-thought confidence elicitation, sampling the most frequent class 20 times (Sample-Class), and averaging 20 verbalized samples (Sample-Prob). Hyperparameters were tuned by grid search: learning rates in [0.01, 0.05, 0.1] and λ from LogUniform(1e-4, 1e-1).

Why This Matters

Research impact. The paper reframes an under-examined property of LLM outputs — the cardinality of verbalized probabilities — as a first-class operational problem, separate from calibration or raw accuracy. It gives the field a formal metric (operational granularity) and shows that popular black-box confidence elicitation and uncertainty estimation techniques do not solve it cheaply, since sampling-based methods need tens to hundreds of LLM calls per sample to converge. It also provides evidence for a rounding-bias explanation of LLM numerical output, connecting it to the human cognitive bias documented in prior work.

Real-world applications (as described in the paper):

• Fraud detection, where a model that declines banking activities may need low false positive rate but high recall.
• Medical diagnosis, where precision control in 0.25 increments may be sufficient for sentiment classification but not for clinical use.
• Product classification and other commercial decision tasks deployed with minimal data annotation or ML expertise.
• Human-in-the-loop or escalation systems, where decisions are delegated to a larger model or a human when the probability of an incorrect decision is high.

Industry relevance. Cloud-hosted LLM APIs are a common business model and typically hide weights, activations, logits, and token probabilities, which means the standard remedy — calibrating on a high-cardinality score distribution — is unavailable. The proposed methods require only 1–2 calls at inference and no access to model internals, and Ours-Sup-2call is reported as 10× more efficient than a 20-sample sampling baseline while outperforming it on the combined dataset, which matters for cost-sensitive production systems that need fine control over operating points.

Future Directions

• Fine-tuning to remove rounding bias at the source. The authors explicitly state that model fine-tuning might be necessary to fundamentally modify LLM scores, and that this is outside the scope of their work, leaving it as the natural follow-up.
• Extending beyond binary classification. The paper limits discussion to binary classification while noting the analyses and approaches can be extended to multiclass via 1-vs-rest or 1-vs-1 approaches — an extension that is proposed but not empirically tested here.
• Making high-specificity prompting reliably performance-preserving. Prompt specificity increased granularity and cardinality but reduced PRAUC and AUROC in the reported experiment; the authors note that further trial and error might maintain performance but that the impact is not always predictable.
• Improving behavior on small, monotonic datasets. Most of the 11 datasets are small (five with as few as 250 samples) with largely monotonic PR curves, which limited the gains of the supervised methods over the unsupervised one — a regime where better methods could still help.

Target Audience

Practitioners and applied researchers who deploy black-box LLM APIs for decision-making under operational constraints (precision floors, recall targets, human-escalation thresholds), especially those building risk-, compliance-, or safety-sensitive classification pipelines. It is also relevant to researchers working on LLM confidence elicitation, black-box uncertainty estimation, and model calibration, and to ML engineers who need to reason about PR/ROC trade-offs from models whose internals they cannot inspect. Readers should be comfortable with binary classification metrics and confusion matrices; the granularity formalism is introduced from scratch.

Authors’ abstract

Black-box Large Language Models (LLMs) provide practical and accessible alternatives to other machine learning methods, as they require minimal labeled data and machine learning expertise to develop solutions for various decision making problems. However, for applications that need operating with constraints on specific metrics (e.g., precision $\geq$ 95%), decision making with black-box LLMs remains unfavorable, due to their low numerical output cardinalities. This results in limited control over their operating points, preventing fine-grained adjustment of their decision making behavior. In this paper, we study using black-box LLMs as classifiers, focusing on efficiently improving their operational granularity without performance loss. Specifically, we first investigate the reasons behind their low-cardinality numerical outputs and show that they are biased towards generating rounded but informative verbalized probabilities. Then, we experiment with standard prompt engineering, uncertainty estimation and confidence elicitation techniques, and observe that they do not effectively improve operational granularity without sacrificing performance or increasing inference cost. Finally, we propose efficient approaches to significantly increase the number and diversity of available operating points. Our proposed approaches provide finer-grained operating points and achieve comparable to or better performance than the benchmark methods across 11 datasets and 3 LLMs.