Text Classification and Intent Detection

A good contract makes borderline and out-of-scope cases visible. It also fixes how finely the label space cuts a single domain. That granularity has a measured price.

BANKING77 is one domain cut 77 ways. Casanueva and colleagues released it in 2020 and describe it in a sentence: “The new BANKING77 dataset comprises 13,083 customer service queries labeled with 77 intents.” Their Table 3 sets that single-domain taxonomy beside CLINC150, whose 150 intents are spread over 10 general domains. With the full training data, fine-tuned BERT reaches 93.66 on BANKING77 against 96.93 on CLINC150. Cut supervision to 10 examples per intent and the gap widens sharply: 83.42 against 91.93.

Read that as a contract cost, not a model result. Both label sets are defensible. The finer one packs more neighbouring boundaries into one domain, and neighbouring boundaries are exactly what a thin annotation budget cannot hold apart. Deciding the label dimension decides how many examples per intent you will have to buy.

The strongest choice depends on data, latency, control, and label stability. On the prompted option, one of those four — consistency — has been measured directly, and it is worse than it looks.

Few-shot prompting is not a stable classifier. Zhao and colleagues put the finding in the opening lines of their 2021 paper: “We show that this type of few-shot learning can be unstable: the choice of prompt format, training examples, and even the order of the examples can cause accuracy to vary from near chance to near state-of-the-art.” Their contextual calibration recovers up to 30.0% absolute.

Ordering on its own is enough to produce that swing. Lu and colleagues isolated it in 2022: “We demonstrate that the order in which the samples are provided can make the difference between near state-of-the-art and random guess performance: essentially some permutations are 'fantastic' and some not”. The result holds “across model sizes (even for the largest current models)”. Selecting permutations with an entropy statistic bought them a 13% relative improvement for GPT-family models across eleven different established text classification tasks.

Nothing in the label set changes between the best and the worst of those runs. The variance sits in material a designer would call incidental — which examples went in the prompt, and in what order. That is why the prompted column carries schema validation and evals rather than trust. It is also why a sparse lexical baseline stays in the experiment, as the number the flexible option has to beat.

The model’s highest score is only one input to the product decision, and step 2 is not a formality.

A modern classifier’s confidence number does not mean what it appears to mean. Guo and colleagues opened their 2017 paper with the finding: “We discover that modern neural networks, unlike those from a decade ago, are poorly calibrated.” Measuring expected calibration error with M = 15 bins across vision and NLP classifiers, they found miscalibration “typically between 4 to 10%”. On the 20 Newsgroups text-classification task, a 3-layer deep averaging network carried 8.02% ECE uncalibrated and 4.11% after temperature scaling. An SST binary TreeLSTM went from 6.63% to 1.84%.

Temperature scaling is one scalar fitted on held-out data. It reorders nothing and improves no prediction. It changes only what the confidence number means. A policy threshold applied before that step is a threshold applied to a quantity that does not say what it appears to say, and the review queue sized from it will be the wrong size in one direction or the other.

Treating every error as a model-capacity problem wastes data and hides product defects. The shortcut case in particular has been measured rather than merely suspected.

Delete the premise from a natural language inference pair and half the input the task is defined on is gone. Gururangan and colleagues did that in 2018 and trained on the hypothesis alone: “Specifically, we show that a simple text categorization model can correctly classify the hypothesis alone in about 67% of SNLI (Bowman et. al, 2015) and 53% of MultiNLI (Williams et. al, 2017).” An ordinary text categorizer recovered two thirds of the labels on SNLI with nothing to compare the hypothesis against. Their conclusion is that “the success of natural language inference models to date has been overestimated”. A headline accuracy on a corpus built that way is partly a measurement of annotation habits. No confusion matrix over the intended labels will say so.

Accuracy can hide a rare high-consequence class. Macro averages give each class equal weight. Micro or traffic-weighted measures emphasize frequent decisions. Precision and recall trade off through thresholds.

Create an error ledger listing false-positive and false-negative consequences, review capacity, prevalence, and delayed outcomes for every label; report confusion and slice behavior, not one aggregate number.

The same dataset shows how far apart two numbers from one classifier can sit. On the full data, with out-of-scope supplied as an extra training class, BERT reached 96.9 in-scope accuracy. Its out-of-scope recall on the same run was 40.3. A single headline figure would have hidden the second one. The authors state the pattern outright: “while the classifiers perform well on in-scope intent classification, they struggle to identify out-of-scope queries”. It is the second column that decides whether a router hands an unsupported request to a person.

The aggregate also hides the slice the classifier will actually meet. Desai and Durrett built that test into their 2020 design: “For each task, we consider in-domain as well as challenging out-of-domain settings, where models face more examples they should be uncertain about.” Out of the box, averaged over 5 fine-tuning runs, BERT’s expected calibration error was 2.54 in-domain on SNLI and 7.03 out-of-domain on MNLI. On the other pair it moved from 2.49 on SWAG to 12.62 on HellaSWAG, while accuracy on that same pair fell from 79.40 to 34.48. An in-domain ECE of 2.49 reads as a well-behaved model. One shift later, the same model is wrong two thirds of the time and five times less honest about it. Calibration is a property of a distribution, not of a checkpoint. It belongs in the slice table beside accuracy.

A small set of targeted checks can distinguish taxonomy, data, representation, and policy problems. Step 2 is the one teams skip, and it is the one with the largest documented payoff.

Hate-speech classifiers carry a bias out of their training corpora and into their predictions. Sap and colleagues measured it in 2019: “Then, we show that models trained on these corpora acquire and propagate these biases, such that AAE tweets and tweets by self-identified African Americans are up to two times more likely to be labelled as offensive compared to others.” Then they diagnosed it. They re-annotated, changing nothing about the data or the model and only the instruction given to the annotators: priming annotators with a tweet’s dialect made them significantly less likely to label it offensive.

That locates the defect in the annotation guidelines rather than in model capacity. No amount of retraining on the same labels could have drawn the distinction. Every retrained model would have reproduced the labels faithfully and scored well doing it. Step 3 is the same move applied to the input instead of the guideline — mask the template, the names, the source field, and see how much accuracy survives.

Pick a real routing problem. Define the label dimension, unknown policy, annotation examples, group and time splits, baselines, metrics, class thresholds, review queue, and retraining trigger.

Include one scenario where classification is the wrong abstraction and extraction, retrieval, or a rule would work better.

Cost the unknown policy before writing it. The CLINC OOS+ condition is one in which “there are 250 out-of-scope training examples, rather than 100”. That one change moves BERT’s out-of-scope recall from 40.3 to 59.2, while in-scope accuracy barely shifts, 96.9 to 96.7. The authors draw the general lesson from it: “out-of-scope performance can be increased by increasing the relative number of out-of-scope training queries”. They are equally clear that it “still remains low relative to in-scope accuracy”. Budget the unknown class like a class, because it behaves like one.