LLM2Jev: LLMs Are Already Jev-Style Decision Models -- When and How to Fine-Tune Them

AI summary

Overview

Research area: Natural Language Processing — applying and adapting large language models as categorical decision models (the "Jev" paradigm), spanning evaluation, fine-tuning methodology, calibration, and multimodal decision-making.

Technical level: Intermediate. The results are largely empirical and easy to interpret, but the fine-tuning objective involves tree-factorized listwise loss, local renormalization over trie nodes, and KL-divergence anchoring that assume some familiarity with language-model training.

Scope: This paper asks how much categorical decision-making ability a general-purpose LLM already has without training, when fine-tuning actually helps, and how that fine-tuning should be done — evaluated with the LLM-as-Jev framework on Qwen3.5-4B and Qwen3-0.6B.

Note on naming: the arXiv listing title reads "LLM2Jev," while the paper text itself consistently titles the work "LLM-as-Jev."

What This Paper Is About

Software systems constantly have to pick one option from a fixed set — routing a support ticket, identifying a user's intent, or checking whether evidence satisfies a rule. Standard chat models can do this, but their free-form output has to be parsed into a structured value, which adds latency and risks malformed answers. The Jev paradigm instead returns a probability distribution directly over candidate options, and this paper investigates whether ordinary off-the-shelf LLMs already behave like Jev-style decision models, and under what conditions fine-tuning them is actually worth the cost.

Key Contributions

1. LLM-as-Jev, an architecture-preserving framework. Candidate options are numbered [1] … [K], the assistant turn is prefilled with Best answer: [, and each option corresponds to a candidate suffix k]. The design needs no new heads, reserved tokens, tokenizer changes, or vocabulary expansion, and applies to any causal LLM.
2. A training-free inference recipe. Candidate suffixes are scored jointly in one prompt by summing their token log-probabilities and normalizing with a softmax. Because the suffixes terminate in a closing bracket, they are prefix-free, so the interface supports arbitrarily large candidate sets (unlike letter-logit readouts, which are bounded to 16 or 26 options). Suffix branches can be scored in parallel via cached KV activations and teacher forcing.
3. A fine-tuning recipe with KL anchors. A tree-factorized listwise loss (negative log-likelihood over the tree-factorized categorical distribution) trains only the choice among "legal" continuation tokens at each trie node, and is computed solely along the gold path. Three KL-divergence anchors against the frozen reference model — L_mass, L_out, L_pos — keep legal-token mass, the distribution over illegal tokens, and next-token distributions at non-decision positions close to the base model. The combined objective is L = L_tree + λ(L_mass + L_out + L_pos).
4. An empirical map of when fine-tuning helps and how to configure it. Six training configurations, anchor-weight sweeps (λ ∈ {1, 0.1, 0.01, 0}), and full-parameter versus LoRA comparisons on two model scales, evaluated across JevBench, seven external benchmarks, general capabilities, multimodal perception, and conversational behavior.

Main Findings

• A pre-trained 4B LLM is already an effective decision model. Without any training, frozen Qwen3.5-4B resolves 188 of the 231 public JevBench items (81.4%), matching or exceeding community Jev-style models built on the same backbone: SemIf (186 correct via single-letter logits; 15 wins vs. 13 losses, p = 0.85) and fine-tuned reflex 4B (184 correct). Only the larger Winnow-12B (86.6%) and proprietary GPT-5.6 Sol (94.4%) scored higher among evaluated baselines.

• Joint scoring beats letter-logit readouts. Across External benchmarks the training-free recipe surpasses SemIf by 3.2 percentage points in macro accuracy, winning 313 instances and losing 154 (p < 10⁻¹²), with advantages on MMLU (+123/−56), MMLU-Pro (+96/−41), and ARC-Challenge (+28/−8).

• High-cardinality candidates work in a single pass. The framework handles all 77 Banking77 intents, reaching 69.0% training-free — a task letter-logit readouts cannot address without changing their prompt formulation. With sufficient batch capacity, a 151-way task needs one prefill and at most three short forward passes under digit-level tokenizers, or a single pass under multi-digit tokenizers.

• Calibration is strong without post-hoc scaling. The training-free 4B ECE is 0.057, better than SemIf (0.061) and reflex 4B (0.070, even with its official calibration mapping).

• Scoring agrees with autoregressive generation. Under greedy decoding, the 4B model achieves the identical 81.4% accuracy with zero parse failures and selects the same candidate as parallel scoring on all 231 instances.

• Multimodal decisions emerge zero-shot. Prepending image tokens, Qwen3.5-4B reaches 82.4% on MMBench under CircularEval and 64.6% on MMStar with no multimodal tuning; withholding images collapses performance to 15.2% and 28.8%.

• The 0.6B model shows where training is genuinely needed. Its 56.7% JevBench accuracy exceeds the raw-logit leaderboard baseline (48.1%), but calibration is poor (ECE 0.278), high-cardinality performance collapses (22.2% on Banking77), and its 151-class CLINC150 validation cross-entropy is 6.9 nats, worse than uniform guessing (ln 151 ≈ 5.0). It redundantly emits descriptive option text after the identifier in 219 of 231 queries.

• Gains track supervised task coverage. Improvements concentrate on domains present in the training mixture where the base model is weak — every mixture includes intent classification, so Banking77 improves consistently; knowledge-intensive benchmarks absent from training (MMLU, MMLU-Pro, TriviaQA) show negligible gains despite high lexical similarity to the reasoning sets used.

• Narrow intent training induces heuristic shortcuts. After one epoch on an intent mixture, the 4B model shows confirmation bias toward "yes" on binary queries, suppression of "none of the above" fallbacks, reliance on surface lexical matches, and overconfidence. JevBench accuracy falls from 81.4% zero-shot to 77.5% at 100 steps and 74.9% after a full epoch, even as in-distribution validation loss keeps improving.

• Format diversity fixes the shortcuts. Replacing narrow commonsense data with six diverse reasoning benchmarks (Reason-9k) removes the affirmative bias, restores calibration, and matches zero-shot JevBench accuracy at 81.4% — the strongest full-parameter 4B result.

• The short-context trap. Scaling short-context reasoning data to 21k and 30k instances degrades JevBench accuracy (p ≤ 0.03), concentrated on long-context queries, because the reasoning datasets contain prompts under 570 words. Adding three long-context corpora (Reason+Long-12k) halts the regression at 4B and doubles long-context accuracy on 0.6B (p = 0.02).

• Model scale governs adaptation utility. Full-parameter tuning gives the 0.6B model broad structural gains, while the 4B model over heterogeneous tasks stays in the 74.9–81.4% range versus an 81.4% baseline. Only LoRA improves the 4B baseline on JevBench (+2.6 points).

• KL anchors are essential even though benchmarks hide it. Without anchors, the 4B model compulsively continues generating text after the identifier on 69 of 231 JevBench queries, and the 0.6B model suffers severe termination failures in chat. Adding λ ≥ 0.01 removes virtually all format-breaking generations. Benchmark accuracy and language-modeling metrics do not reveal this.

• λ = 1 is the recommended default. At 4B it gives peak JevBench accuracy and lowest calibration error under both full-parameter and LoRA training. At 0.6B, relaxing to λ ≤ 0.1 slightly lifts Banking77 accuracy but triggers significantly more non-terminating chat replies (p = 0.007).

• LoRA is preferred for capable backbones. At 4B, LoRA with λ = 1 reaches 84.0% on JevBench with the best calibration (ECE 0.050), significantly beating full-parameter tuning (p = 0.01). At 0.6B the two strategies are comparable. Parameter efficiency does not substitute for anchors: unregularized LoRA still produces syntax termination failures, and unconstrained LoRA on 0.6B drifts more than full fine-tuning.

• General capability is preserved. No statistically significant decline in the General average at either scale, and the 4B models fully retain zero-shot visual decisions despite text-only training, actually improving on MMBench.

Methodology in Plain English

The researchers avoid building anything new into the model and instead change how they ask. Options are listed on separate lines with bracketed numbers, and the assistant reply is started for the model as "Best answer: [", so choosing option k means producing the short continuation "k]". Because every such continuation ends with a bracket, no one option's text can be the beginning of another's — a prefix-free property that removes the ceiling on how many options can be offered.

For the no-training recipe, the model is run once to cache the prompt's internal state, then every candidate's continuation is scored by summing its token log-probabilities and normalizing across candidates. Since all candidates are known ahead of time, their branches can be scored in parallel rather than decoded one at a time. Because candidates are scored after conditioning on the full prompt, each option is judged relative to the whole slate rather than in isolation.

For training, the candidate continuations form a small prefix tree (a "trie"). At each node some vocabulary tokens are legal continuations toward a valid answer and the rest are illegal. The listwise loss renormalizes probabilities over legal continuations at each step and takes the negative log-likelihood of the correct answer, which means gradients only need to be computed along the path spelling out the gold answer — tractable even for 151 candidates. To stop the model from drifting, three KL penalties are added against the frozen original model: one keeping total probability on legal tokens stable, one keeping the relative distribution over illegal tokens stable, and one keeping next-token distributions at ordinary positions (the reply start and the position after the closing bracket) unchanged. A tail approximation retains the reference model's top-64 tokens for each anchored distribution, which captures over 94.5% of the anchored distribution; sampling three off-path trie nodes matches gold-path regularization at about 2.2 times the computational overhead. Training uses AdamW under FSDP, a conservative learning rate with linear warmup and cosine decay, shuffled candidate order, and a single epoch, with a validation check that halts training if legal-token mass drops substantially. Training data comes only from public human-annotated datasets — no teacher-generated synthetic labels — with filtering against any 13-word n-gram or identical sentence of six or more words shared with an evaluation benchmark. The 231 public JevBench items are treated as an out-of-distribution probe, and the paper reports diagnostic accuracies on those public items rather than official leaderboard standings, since held-out and sealed suites remain private.

Why This Matters

Impact on research. The paper challenges an implicit assumption behind much of the Jev-style model ecosystem — that decision capability must be added through custom heads, reserved tokens, or synthetic teacher-generated data. It shows that a frozen general-purpose LLM scores all options jointly and matches or exceeds fine-tuned systems on the same backbone, and it separates two things that benchmarks usually conflate: task accuracy and behavioral integrity. Its demonstration that KL anchors matter for generation stability while being nearly invisible in benchmark scores is a methodological caution for anyone evaluating fine-tuned models. The finding that more short-context training data can actively harm long-context transfer, and that data composition rather than training duration governs transfer fidelity, is a concrete design principle.

Real-world applications.

• Routing support tickets or classifying user intent across many categories, including 77-way intent routing as tested on Banking77, without a bespoke classifier per task.
• Binary verification tasks — checking whether evidence satisfies a rule — where calibrated "yes"/"no" probabilities let downstream logic escalate low-confidence cases.
• Multimodal visual decisions such as image classification and visual question answering through the same interface, with no multimodal fine-tuning.
• Deploying a new task by supplying only an instruction prompt and a candidate list, which the paper positions as the cost advantage of the Jev paradigm over dedicated classifiers.

Industry relevance. Deploying a new task becomes a prompt-and-option-list change rather than a training project, and the interface produces structured values natively, removing parsing latency and malformed-output risk. The paper's practical guidance — default to the training-free baseline, keep λ = 1, prefer LoRA on capable backbones — gives engineering teams a starting configuration without exhaustive tuning. However, the paper reports that GPT-5.6 Sol was evaluated in standard text generation mode, and Jev's own weights and training methodologies remain proprietary, so direct comparisons to the original Jev system are not available here.

Future Directions

• Inference-time permutation averaging. The paper notes permutation averaging could reduce residual option-order bias but omits it to prioritize efficiency; quantifying its accuracy-versus-cost trade-off is an open question.
• Data coverage for heterogeneous decision suites. Because public data covers only part of the competencies probed by JevBench, extending training mixtures to cover more answer topologies and input-length distributions is the identified route to broader transfer.
• Testing on held-out and sealed JevBench suites. The paper evaluates only the 231 public items and reports diagnostic accuracies; whether these conclusions hold on the private suites is untested.
• Extending the architecture-preserving claims to more backbones. The framework is argued to apply out of the box to any causal LLM, but the empirical work covers Qwen3.5-4B and Qwen3-0.6B, and the paper's own comparison table spans many community systems built on different bases.

Target Audience

Researchers and engineers working on LLM-based classification, routing, and structured decision-making, especially those weighing whether to fine-tune or use a frozen model; practitioners in applied NLP who need calibrated, machine-consumable outputs from LLMs; and anyone maintaining or benchmarking Jev-style models who wants to understand how much of the reported performance comes from training versus from the base model and the prompting interface. Readers get the most value if they are comfortable with log-probability readouts, KL-divergence regularization, and parameter-efficient fine-tuning.

Authors’ abstract

Jev-style decision models return categorical probability distributions over predefined options without generating free-form text, enabling software systems to act on their outputs directly. In this work, we investigate the extent to which general-purpose LLMs already possess this capability out of the box, and when fine-tuning is actually necessary. We present LLM2Jev, an architecture-preserving framework that extracts calibrated decisions directly from next-token probabilities over bracketed numeric identifiers. LLM2Jev provides both a training-free inference recipe and a fine-tuning objective that optimizes candidate selection via a tree-factorized listwise loss while anchoring auxiliary predictions to the base model using KL divergence penalties. Evaluating on Qwen3.5-4B and Qwen3-0.6B, we find that modern LLMs are inherently effective decision models: without training, the 4B model matches community Jev-style models built on the same backbone, outperforms letter-logit readouts, supports arbitrary option counts, and natively handles multimodal decisions over images. Fine-tuning provides targeted rather than universal benefits -- substantially improving weaker models and specific tasks (such as many-option intent routing), but offering diminishing returns for strong backbones. Crucially, our KL anchors prevent behavioral degradation in conversational text generation, with LoRA delivering the strongest performance on capable models.