Cross-Primitive Consistency in Jev

AI summary

Overview

Research area: Evaluation and calibration of large-language-model decision systems, specifically "Jev"-style structured decision models that answer through typed query primitives (binary Noul queries and multiple-choice Choice queries) rather than free-form text. The paper is listed under Computer Vision, but its subject matter is LLM reliability, consistency, and calibration.

Technical level: Advanced. The paper assumes comfort with logits, probability simplexes, total variation distance, ROC-AUC, PCA projections, and residual/parameter-efficient fine-tuning.

Scope (one sentence): The paper introduces a four-axis benchmark (PrimaCons) that measures whether three Jev models give the same answer to the same semantic question when the query primitive changes, and proposes a lightweight residual calibrator (Cprc) that reduces the measured inconsistencies.

What This Paper Is About

Jev models answer decision tasks by returning structured predictions directly from typed query primitives — for example, a binary Noul question ("is this hallucinated?") or a Choice question with several candidate options. The same underlying decision can therefore be asked in several logically equivalent ways, and prior work has evaluated these models only under a single primitive at a time. This paper asks whether the same semantic state produces consistent decisions across primitives, builds a benchmark to measure that, and shows the inconsistencies are systematic enough to predict decision failure.

Key Contributions

1. A four-axis benchmark (PrimaCons). A controlled benchmark that varies the query primitive or its presentation while holding the decision semantics fixed, spanning Complement Consistency, Label Hierarchy Consistency, Abstention Alignment Consistency, and Label-Description Consistency. It is built from HaluEval, PKU-SafeRLHF, AbstentionBench, and AG News, with separate calibration and test splits.

2. An empirical study across three model sizes. The authors evaluate JevK5-2B, JevK5-4B, and JevK5-9B and report consistency violations on all four axes, then characterize the violations as systematic rather than random.

3. A diagnostic finding. Cross-primitive probability discrepancies rank decision failures with a ROC-AUC of 0.91, meaning disagreement between equivalent views is itself a usable error signal.

4. A mitigation method (Cprc). The Cross-Primitive Residual Calibrator attaches a lightweight MLP residual network to a frozen backbone and trains it with a joint cross-entropy plus consistency-penalty objective, improving both consistency and accuracy, including on a held-out domain under a leave-one-domain-out protocol.

Main Findings

• Binary isomorphism is violated (Axis I). Under perfect complementarity, a positive Noul and a reversed Noul should give probabilities summing to 1. Measured probability gaps between the positive and reversed Noul are 0.1858 (JevK5-2B), 0.1360 (JevK5-4B), and 0.0892 (JevK5-9B). The Choice primitive aligns with neither Noul view — for JevK5-4B, G(n+, c) = 0.1132 and G(n−, c) = 0.0689.

• Disagreement predicts failure. Treating the residual |p(n+) − p(n−)| as a ranking signal for decision error yields a ROC-AUC of 0.91, as shown in the inset of Figure 3. Sample-wise predictions cluster in the top-left and bottom-right corners rather than on the diagonal y = x.

• Finer label granularity amplifies inconsistency, with probability-mass inflation (Axis II). For JevK5-4B, total variation between Choice and Noul is 0.0870 at the coarse (2-class) level but rises to 0.5329 at the medium (6-class) level and 0.6323 at the fine (12-class) level. Average residuals between medium-category probability and the sum of its two fine probabilities fall below 0 for all medium categories, indicating the model inflates rather than partitions a fixed probability mass.

• Abstention depends on the elicitation procedure (Axis III). For JevK5-4B, the abstention probability gap between Choice and Noul is 0.2614, the hard-label conflict rate is 26.69%, and the accuracy gap between primitives is 10.83%. Quadrant analysis shows 24.74% of samples fall where Noul leans toward abstention but Choice does not, versus 4.45% in the symmetric quadrant. Conflict rates are 36.54% (2B), 26.69% (4B), and 17.18% (9B).

• Descriptive text and primitive switching both shift the distribution (Axis IV). On JevK5-4B, adding neutral category descriptions to Choice labels shifts the distribution by TV(pʳ, pᵈ) = 0.1013, while switching from Choice to normalized per-category Noul queries gives TV(pʳ, pⁿ) = 0.2405, and combining both gives TV(pᵈ, pⁿ) = 0.2694. The scatter plots show an asymmetric crossing pattern, and mean markers for World, Sports, and Sci&Tech lie farther from the diagonal for described labels, meaning description augmentation exacerbates rather than anchors inconsistency.

• In-domain calibration works. On JevK5-2B (Axis I), Cprc reduces G(n+, n−) from 0.1832 to 0.0869 (−52.56%) while Choice accuracy rises from 65.03% to 79.00% (+13.97%). On JevK5-2B (Axis II), D_M drops from 0.481 to 0.130 (−73.03%) while accuracy rises from 77.67% to 98.48% (+20.81%). On JevK5-4B (Axis II), D_F drops from 0.633 to 0.355 (−43.95%) while accuracy rises from 95.20% to 99.65% (+4.45%).

• Gains transfer to an unseen domain. Under leave-one-domain-out with Axis IV held out, JevK5-2B reduces TV(pʳ, pⁿ) from 0.439 to 0.182 (−58.57%) while raw-label accuracy rises from 84.90% to 87.42%. JevK5-9B reduces TV(pʳ, pⁿ) from 0.225 to 0.094 (−58.42%) with accuracy rising from 88.02% to 89.75% (+1.96%). JevK5-2B also reduces TV(pᵈ, pⁿ) from 0.474 to 0.237 (−49.98%).

• Axis III improvements are mixed for the smallest model. In Table 5, JevK5-2B's abstention gap G_A rises from 0.1884 to 0.2260 even as its conflict rate K falls from 36.15% to 18.37% and accuracy rises from 77.81% to 87.96%. The 4B and 9B models reduce both G_A and K.

• Overhead is small. Averaged over 6,000 test instances, Cprc adds only +0.75% latency.

• Note on reported figures. The abstract states a 58.57% inconsistency reduction and a +2.97% accuracy gain, while Section 1 states −58.6% and +2.52%. Table 7 supports a 84.90 → 87.42 accuracy change, which corresponds to the +2.52 figure; the paper does not reconcile the two.

Methodology in Plain English

The authors start from the observation that a Jev model takes a fixed context plus a query primitive as input. That separation lets them build a controlled test: keep the meaning of the question identical, change only the interface, and see whether the model's probability distribution stays put.

The four axes. Each axis corresponds to a logical rule a reliable model should obey.

• Complement Consistency asks the same yes/no question both ways, plus a binary Choice version, and checks whether the answers sum to one (binary isomorphism).
• Label Hierarchy Consistency exploits PKU-SafeRLHF's existing label taxonomy of 12 fine safety categories, which pair into 6 medium parent categories that project to a single coarse safe/unsafe label. Predictions at each level should add up consistently (taxonomic coherence).
• Abstention Alignment Consistency compares a three-option Choice (correct answer, distractor, "unable to answer") against a binary Noul that asks whether the question is answerable. The judgment should not depend on how it was elicited (procedure invariance).
• Label-Description Consistency compares Choice with raw AG News labels against Choice with textual category descriptions, and against a distribution built by normalizing four category-specific Noul queries. Neither rephrasing labels nor swapping primitives should move the distribution (description invariance and projection invariance).

Datasets and splits. HaluEval (6,000 calibration / 6,000 test), PKU-SafeRLHF (6,000 / 4,000, unsafe samples only, 12 fine categories), AbstentionBench (6,000 / 6,000, balanced answerable and unanswerable), and AG News (6,000 / 6,000, four categories).

Metrics. Probability gaps between paired primitives (Equation 1), sample-averaged total variation across hierarchy levels (Equation 2), an abstention probability gap and a hard-label conflict rate (Equations 3–4), and total variation among the raw, described, and normalized-Noul distributions (Equations 5–6).

The calibrator. Cprc keeps the JevK5 backbone completely frozen and adds a small MLP. Its inputs are a PCA-projected hidden state from the (L−1)-th layer of the L-layer backbone, the model's raw output probabilities, and view-level metadata such as predictive entropy. Separate binary and multiclass heads predict a logit correction, squashed with tanh and added directly to the raw logits. Training minimizes cross-entropy plus a weighted consistency penalty (α) drawn from the axis-specific measures above, so accuracy and cross-primitive agreement are optimized together.

Protocol. PCA dimensionality and hyperparameters are selected only on the calibration split to avoid leakage. For generalization, the leave-one-domain-out protocol trains on three axes' calibration sets and tests on the fourth; Axis IV is chosen as the held-out target because it involves more diverse comparisons.

Why This Matters

Impact on research. Consistency research on LLMs has mostly varied prompts, weights, or reasoning traces. This paper isolates a different axis: the querying interface itself, which Jev-style architectures deliberately decouple from the context. Framing consistency as four separately testable logical invariants — binary isomorphism, taxonomic coherence, procedure invariance, description invariance, projection invariance — gives the evaluation community concrete, falsifiable criteria rather than a generic "robustness" score. The ROC-AUC of 0.91 also suggests cross-primitive disagreement could serve as a cheap, label-free uncertainty signal.

Real-world applications.

• Clinical decision support, where the paper's related work cites Jev methods applied to medical and clinical evaluation; a model that changes its answer depending on whether it is asked positively or negatively is unsafe to deploy there.
• Safety moderation, where a model judging toxic versus non-toxic content should not flip when the label set is described in words rather than as bare category names.
• Hallucination detection, the setting of Axis I, where binary factual judgments are the product being sold.
• Abstention-aware question answering, where the decision to say "unable to answer" must not depend on whether abstention was offered as a menu option or asked as a yes/no question.

Industry relevance. The calibrator is deliberately cheap: a frozen backbone, an MLP on a PCA-projected hidden state, and a +0.75% latency overhead measured over 6,000 instances. That makes it practical as a post-hoc add-on for already-served models rather than requiring retraining. Because the gains survive a held-out domain under LODO, the approach is positioned as learning general corrections rather than memorizing one dataset's statistics — the property that matters if a vendor wants one calibrator across product surfaces.

Future Directions

• Reconciling the headline numbers. The abstract and introduction report different reductions (−58.57% versus −58.6%) and different accuracy gains (+2.97% versus +2.52%), and the paper does not explain the discrepancy. A cleaner accounting of which setting each figure describes would help.

• Covering more primitives and more axes. PrimaCons tests three primitives (positive Noul, reversed Noul, Choice) and four axes. Whether the same failure patterns hold for other typed querying interfaces, or for chains of primitives, is left open.

• Understanding why probability mass inflates with granularity. The paper documents that medium-category probabilities systematically exceed the sum of their fine-category probabilities, but does not identify the mechanism. Whether this is a softmax normalization artifact, a training-data property, or a tokenization effect is unresolved.

• Pushing the calibrator beyond one held-out domain. Only Axis IV serves as the unseen target in the LODO protocol. Whether Cprc generalizes when the held-out domain differs more sharply — or when the backbone is a different model family entirely — is untested.

• Fixing Axis III for smaller models. On JevK5-2B the abstention probability gap actually increases after calibration even though the conflict rate halves, which points to a residual weakness in how abstention confidence is aligned.

Target Audience

Researchers and engineers working on LLM reliability, structured/constrained decoding, and parameter-efficient calibration will get the most from this paper. It is directly relevant to teams deploying Jev-style decision heads in safety, moderation, or clinical settings who need an audit method that does not require ground-truth labels. Evaluation and benchmarking researchers will find the four-axis framing reusable even outside Jev. Readers without a background in probability distances and logit-level calibration should expect a steep climb, since the paper presents its results primarily through TV distance, probability gaps, and residual logit corrections.

Authors’ abstract

Jev enables efficient decision-making with large language models (LLMs) through typed querying primitives, allowing the same semantic state to be expressed in different formats, such as binary and multiple-choice queries. However, the consistency of decisions across primitives has not been systematically evaluated. This work introduces PrimaCons, a four-axis benchmark for assessing cross-primitive consistency while preserving the semantics of the query. Evaluations across three Jev sizes reveal consistency violations across all four axes. Our analysis shows that these violations follow systematic patterns and that cross-primitive discrepancies predict decision failure with a ROC-AUC of 0.91. To mitigate these inconsistencies, we propose the Cross-Primitive Residual Calibrator (CPRC), which attaches a lightweight residual network to a frozen backbone and jointly optimizes predictive accuracy and cross-primitive consistency. Under a leave-one-domain-out protocol, CPRC reduces cross-primitive inconsistency by 58.57% and improves accuracy by +2.97%., demonstrating its effectiveness in both in-domain and out-of-domain settings. Our findings establish cross-primitive consistency as a key dimension for evaluating Jev and provide a practical approach to improving the reliability of its decisions. Code and benchmark is available at https://github.com/67L1/PrimaCons-CRPR.