Label-Wording Augmentation in a Typed-Decision Encoder: Gains on Unfamiliar Words, Not on Abbreviations or Codes

AI summary

Overview

Research area: Natural Language Processing — specifically label-wording robustness for typed-decision models (classifiers that receive their candidate options at request time rather than at training time), with an empirical study on Russian-language routing.

Technical level: Intermediate. The paper is readable with a general ML background; the key concepts (cross-encoders, softmax normalization, temperature scaling, risk–coverage curves, McNemar tests) are introduced or used in standard ways, though the statistics sections assume some familiarity with paired testing and bootstrap intervals.

Scope (one sentence): A controlled comparison of two releases of a 307M-parameter Russian cross-encoder — one with and one without multi-surface-form label augmentation — evaluated on ten label vocabularies at two candidate-set sizes, plus out-of-distribution and cross-model checks and a list of four evaluation hazards.

What This Paper Is About

Some classifiers do not fix their label set at training time: the caller supplies the candidate options in their own words at request time, so accuracy depends on strings the model developer never controlled. The paper asks which kinds of unfamiliar label wording a model handles better when training has shown each label in several surface forms (paraphrases, transliterations, clipped forms). The author compares two releases of the same model, Ru-Jev, holding the questions fixed and varying only how 18 routing labels are worded.

Key Contributions

1. A controlled wording measurement. Ten label vocabularies (produced by three separate processes and audited for overlap against training data), two candidate-set sizes (five and eighteen), and 1,000 fixed questions, with exact paired McNemar tests, label-level bootstrap intervals, and a Holm correction over 23 cells.
2. The headline empirical finding. Label-wording augmentation improves accuracy on held-out vocabularies made of ordinary or borrowed words (English-derived jargon, office slang, department names, descriptive sentences) but yields no usable gain on abbreviations or opaque codes — and a probe at a single candidate-set size can misstate the effect.
3. Out-of-distribution and cross-model results. Two Russian datasets neither release saw (ruMTEB news headline topics, CEDR emotions) plus a comparison against Laya, a 322M model on the same mmBERT backbone that reads all options in one pass.
4. Four documented evaluation hazards from release-to-release comparison, each with its measured effect on numbers the author had prepared for publication (shared random stream leakage, matching rows on text alone, mixed denominators, integrity checks against a stale reference).

Main Findings

• Word-shaped labels improve, in every cell. On all fourteen word-vocabulary cells (seven vocabularies × two candidate-set sizes) the augmented release v1.3 is more accurate. Largest gains are on English-derived jargon: +0.235 with five candidates and +0.142 with eighteen on the strict slice (11 labels, 625 questions); on all 18 jargon labels, +0.222 and +0.209, both marked *** after Holm adjustment. On jargon the label-level interval excludes zero in both settings for all labels.
• Eleven of fourteen gains survive Holm correction. The exceptions noted are office slang with eighteen candidates and descriptive sentences with five candidates. Slang gains +0.166 with five candidates (interval above zero) and +0.050 with eighteen; the recorded reading rule returned "transfer" (raw p = 0.016) but the gain does not survive correction and its interval spans zero.
• Smaller but consistent moves elsewhere. Department names gain +0.034 and +0.101; descriptive sentences +0.019 and +0.030. Control vocabularies (English key, Russian word — present in both releases' training data) move by +0.005 / +0.006 and +0.004 / +0.009. The Russian phrase, which is a training style for v1.3 only, gains +0.076 and +0.182.
• No usable gain on abbreviations or opaque codes. On three-letter Cyrillic abbreviations v1.3 is −0.024 (not significant) on all labels and −0.060 on the clean slice, a difference created by one excluded label on which v1.3 gains 0.829. On opaque codes at eighteen candidates on the strict slice (9 labels, 512 questions) v1.1 scores 0.018 (below chance of 0.056) and v1.3 0.061 (at chance). Code identifiers such as ALARM_SCN contain the trained key and are easy for both releases, so the author does not treat them as a robustness test.
• Single-size probes can mislead. The size of the effect depends on candidate-set size differently per vocabulary; on the strict slang slice at eighteen candidates the result is described as "a reshuffle as much as a lift," with 5 labels improving and 6 getting worse (e.g. справочное бюро 0.000 to 0.585; перечни 0.534 to 0.086).
• Confidence tells a mixed story. In most cells where accuracy rises, the risk–coverage area (AURC) falls — strict jargon 0.235 to 0.078 at five candidates and 0.452 to 0.418 at eighteen; strict slang 0.223 to 0.121 at five, essentially unchanged (0.476 against 0.470) at eighteen. The only non-control exception is slang on all labels with eighteen candidates (0.354 to 0.395). But confident errors (wrong answers above 0.8 confidence per 1,000) fall on strict jargon at five candidates (110 to 62) while rising at eighteen for jargon (79 to 109) and strict slang (151 to 191), and rising on abbreviations (77 to 118 on the clean slice).
• Out of distribution, the two releases trade places. On 1,000 news-headline questions (k = 6) v1.1 scores 0.652 and v1.3 0.568 — a drop of 8.4 points, Holm p < 10⁻⁶, with five of six labels worse and none better. On 1,000 CEDR emotion questions (k = 5) v1.1 scores 0.455 and v1.3 0.511 — a gain of 5.6 points. Laya scores 0.610 and 0.605 respectively; no model is best on both.
• In distribution, the releases are indistinguishable. On the six RussianSuperGLUE tasks with identical rows and option wording, both answer exactly 8,120 of 10,157 decisions correctly and disagree on 1,680 (840 in each direction; McNemar p = 1.0). Overall accuracy is 0.7994 for both.
• Calibration differs but residual error does not. v1.3 takes a lower fitted temperature in every one of 25 half-splits (1.50–1.60 against 1.94–2.10), but residual expected calibration error after each model receives its own temperature is essentially the same (0.0190 against 0.0191, with the sign changing across splits).
• Typed-decision scoring costs more as options grow. Median latency on one Apple M4 CPU using all ten cores: Ru-Jev v1.3 takes 18.5 / 32.8 / 73.8 ms at k = 2 / 5 / 18, Laya 15.5 / 17.8 / 30.5 ms — Ru-Jev is 1.2× slower at two options and 2.4× slower at eighteen.
• Four hazards, each of which had produced a false result. A shared random stream caused 90.3% of the new acceptability validation rows and 89.4% of reasoning rows to have been in v1.1's training data (turning an apparent regression of 0.909 to 0.647 into 0.650 on held-out rows). Matching rows on input text alone turned a labelling difference into a phantom toxicity gain of 0.722 to 0.954 (v1.1 scores 0.961 with the wording it trained on). A summary row on 5,078 decisions (0.7948 against 0.7920) was printed beside a table counting 10,157. Integrity checks against a stale local copy trained seed runs without the transliteration styles, yielding jargon accuracy of 0.588 and 0.611.

Methodology in Plain English

The model. Ru-Jev is a pointwise cross-encoder: it scores each (question, option) pair independently with a shared encoder and a linear head, then normalizes the scores across options to get a probability distribution. A single fitted temperature calibrates the outputs. Two consequences matter for the experiment: the output does not depend on option order, and the distribution over any subset of options can be obtained exactly by renormalizing — so the same 1,000 questions can be scored at five candidates and at eighteen from one pass.

The comparison. The base encoder is mmBERT-base (307M parameters, input truncated to 256 tokens), trained on thirteen Russian tasks drawn from RussianSuperGLUE, RuCoLA, MERA, ruMTEB (including the MASSIVE intent and scenario tasks) and a toxicity set — 143,991 training and 26,746 validation decisions. Release v1.1 shows each MASSIVE scenario label as its English key and an ordinary Russian word. Release v1.3 stores several surface forms per label and draws one uniformly per training example (for scenarios: English key, Russian word, Russian phrase; for the 60 intents also a Cyrillic transliteration and a rule-based clipped form).

The probe design. All probes use the same 1,000 questions from the MASSIVE scenario validation split, each with one gold scenario among 18. Only the label wording changes. Distractors at five candidates are drawn deterministically and are identical across all vocabularies and models, so chance is 0.20; at eighteen candidates every label is shown, so chance is 0.056. Vocabularies were written by three separate processes — six by the model's own development team ("card bench"), one generated by rule by the AI coding agent that trained the model, and two by a separate AI assistant session that had not trained the model and had seen no scores.

The overlap audit. Every probe label was compared with every option string in both rebuilt training mixes, by exact equality and by prefix in both directions. This yields "clean" and "strict" slices that drop labels with a relation to training data. Two card-bench vocabularies turned out to be training styles for both releases and serve only as controls.

The statistics. Each cell gets an exact McNemar test on paired answers and a 95% interval from a two-stage bootstrap that resamples labels first, then questions within labels — treating the label, not the question, as the unit that generalizes. Because cells have 9 to 18 labels, the author states plainly that these intervals are wide and approximate. Holm correction runs over 23 cells.

Why This Matters

Impact on research. Most benchmarks score models on the label names the dataset authors wrote, while deployments use names the organization already has — department names, internal codes, abbreviations, borrowings from English. This paper shows that a model can improve substantially on unfamiliar words while gaining nothing on codes, and that measuring at one candidate-set size can misstate the effect in either direction. It also documents four concrete ways a release-to-release comparison produced a publishable-looking result that was not real — including leakage rates above 89% and a withdrawn figure that had already reached a model card.

Real-world applications:

• Citizen-appeal and service routing, where an operator supplies a department list in the organization's own vocabulary and needs to know whether unfamiliar internal names will work.
• Internal ticket triage using office slang and jargon rather than curated label taxonomies.
• Deployment planning for on-premise or CPU-only settings, where the latency table (73.8 ms at eighteen options versus 30.5 ms for a one-pass model) directly informs whether a pointwise design is affordable.
• Human-in-the-loop review workflows, where the coverage-versus-accuracy table (answering 50% of cases at 0.967 accuracy, or 90% at 0.859) sets staffing levels.

Industry relevance. The practical recommendations are directly actionable for teams shipping request-time label sets: choose augmentation styles that resemble the wording the deployment will use; refit the temperature on that wording rather than reusing a released one; and evaluate robustness at more than one candidate-set size. The protocol in Section 9 is a checklist any team comparing two model releases could apply.

Future Directions

• Isolate the augmentation. The releases differ in data partitions, toxicity wording and learning-rate schedule as well as label augmentation, and each was trained once; the author explicitly frames the results as differences between two trained models consistent with the augmentation as cause, not a controlled ablation. A properly built single-factor comparison is the obvious next step.
• Seed variance at more settings. Seed variance was measured only for the v1.1 recipe, on jargon, with five candidates (a range of 0.036 on all labels and 0.046 on the clean slice). No estimate exists for eighteen candidates or for the slang and code probes, and several of the smaller differences in Table 2 are described as not reproducible.
• Augmentation aimed at codes. Since abbreviations and opaque codes carry little of their meaning in the string, whether any augmentation strategy (or an explicit none-of-these option, which the model currently cannot express) could help there is left open.
• Broader scope. Wording probes cover one Russian routing task and vocabularies written within the author's own project or by AI agents in it; whether the word-versus-code split holds for other languages, other tasks, and vocabularies collected from real deployed systems is untested.

Target Audience

Practitioners deploying classifiers with request-time label sets; researchers working on zero-shot classification, verbalizer choice and prompt/label sensitivity; ML engineers responsible for model release comparisons and evaluation hygiene; and anyone building Russian-language NLP systems. Readers focused on statistical methodology will find the label-level resampling and the four-hazard protocol section as useful as the accuracy results.

Authors’ abstract

Typed-decision models score options that the caller supplies at request time, so their accuracy depends on label wording that the model's developer does not control. We compare two releases of a 307M-parameter multilingual cross-encoder, Ru-Jev, trained on thirteen Russian-language tasks; the later release adds an augmentation that presents labels in several surface forms. Holding questions fixed and varying only how eighteen routing labels are worded, we evaluate ten label vocabularies, produced by three separate processes and checked for overlap with the training data, with five candidates per question and, for eight of them, also with eighteen. On held-out vocabularies made of ordinary or borrowed words (English-derived jargon, office slang, department names), the augmented release is more accurate in every setting, by up to 0.24. Eleven of these fourteen gains survive a Holm correction (slang with eighteen candidates and descriptive sentences with five do not), and for jargon on all labels the label-level interval excludes zero in both settings. On abbreviations and opaque codes it gives no usable gain: on abbreviations it is not better, and worse on the clean slice, and on opaque codes both releases stay at or below chance once labels related to training options are removed. In most cells where accuracy rises, the risk-coverage area falls, but with eighteen candidates confident errors rise on jargon and slang. On two datasets neither release saw, accuracy moves by -8.4 and +5.6 points. We also report four evaluation hazards from comparing the two releases.