Chinese-Jev: Bringing System One Model to Chinese-Language Tasks

AI summary

Overview

Research area: Natural Language Processing — efficient, structured decision-making models (so-called "System One" models) applied to Chinese-language tasks instead of open-ended text generation.

Technical level: Intermediate. The paper assumes familiarity with encoder architectures, softmax candidate scoring, cross-entropy loss, and calibration metrics such as expected calibration error, but the motivation and the experimental comparisons are accessible.

Scope: The paper describes Chinese-Jev, a compact encoder-only model trained on a unified pipeline of Chinese decision data, together with a new benchmark (CJ-Bench) covering general, medical, legal, and financial decisions, and reports accuracy, calibration, latency, and on-device deployment results.

What This Paper Is About

Many applications need a model to pick an answer, judge whether a statement holds, or assign a rating — not to write prose. Existing "System One" decision models such as Jev handle these three decision types but perform poorly on Chinese-language inputs. This paper builds Chinese-Jev, which adapts a lightweight encoder to Chinese decisions through a shared probability-over-candidates training formulation, first on a broad 10-million-example corpus and then separately for medicine, law, and finance.

Key Contributions

1. Chinese-Jev models. A System One model family for Chinese-language decision-making, produced by general Chinese-language pre-training followed by independent fine-tuning in medicine, law, and finance. All models share one decision interface supporting option selection (choice), proposition judgment (noul), and ordinal rating (score) in a single forward pass.

2. A unified data construction pipeline. A reusable pipeline that converts heterogeneous Chinese QA annotations into probability targets over candidate options, covering single-answer questions, Yes/No annotations, ratings, multiple-answer questions, and synthetic rule-based tasks. It yields a pre-training corpus of 10 million examples spanning eight task categories, plus domain corpora of 3,000,000 medical decisions and 24,000 decisions each for law and finance.

3. CJ-Bench. A benchmark of 307,900 held-out decisions built from the test partitions of the general, medical, legal, and financial corpora (100,000, 200,000, 4,300, and 3,600 decisions respectively) for evaluating accuracy, calibration, and latency under a common protocol.

4. Empirical evaluation and on-device deployment. Chinese-Jev surpasses the closed-source Jev model on general-task accuracy and on medicine while retaining 92% of Jev's average accuracy across the three specialized domains, with measured speedups of 20.3x and 17x relative to the hosted Jev API. An INT8 browser deployment enables local smartphone inference at approximately 1 second per decision.

Main Findings

• General-task accuracy. After first-stage pre-training, Chinese-Jev General reaches 69.20% accuracy on the General component of CJ-Bench, versus 68.35% for Jev — a gain of 0.85 percentage points, or a 1.24% relative improvement. It is the highest accuracy among the evaluated models.

• Calibration on general tasks. Chinese-Jev General reduces expected calibration error from 11.45% (Jev) to 3.78%, a 67.0% reduction. The comparison with its initialization point (Laya Multilingual, 40.22% accuracy and 23.04% ECE) shows a 28.98 percentage-point accuracy gain and a drop in ECE to 3.78%, with no change in architecture and no measured latency cost (both 14 ms median per decision).

• General-task latency. Chinese-Jev General measures a median of 14 ms per decision, 20x faster than Qwen3.5-2B under the same local timing protocol and 20.3x lower than the hosted Jev API. The paper notes that the Jev figure includes network round-trip time, so it reflects end-to-end response latency rather than a direct comparison of model inference speed.

• Domain fine-tuning gains. Relative to Chinese-Jev General, the specialists improve accuracy by 47.88, 21.34, and 29.33 percentage points on Medical, Legal, and Finance, raising average accuracy from 37.97% to 70.82%. On Finance, Chinese-Jev General reaches only 35.39%, below the multilingual initialization's 38.31%, showing that general pre-training helps unevenly across domains.

• Medicine. The medical specialist reaches 87.65% accuracy, exceeding Jev by 3.36 percentage points, or 4.0% in relative terms, and outperforming all four open baselines on all three domains.

• Legal and finance gaps. The legal and finance specialists reach 60.10% and 64.72% accuracy after fine-tuning on 24,000 decisions each, but both remain below Jev (68.35% and 78.39%). Across the three specialized domains the specialists achieve 92.0% of Jev's average accuracy.

• Specialist latency. Median latency is 15 ms per decision in each domain, corresponding to 17.5x to 29.2x speedups over Qwen3.5-2B locally and roughly one-seventeenth of the hosted Jev API's measured latency.

• Calibration after specialization is uneven. ECE falls from 17.33% to 7.25% on Medical and from 14.43% to 11.55% on Finance, but rises from 15.37% to 17.45% on Legal despite the accuracy gain. The medical specialist has the highest accuracy while Open-Jev obtains the lowest ECE; the specialists' average ECE (12.08%) remains higher than Jev's (5.57%).

• On-device inference. An INT8-quantized model running in a mobile browser on an iPhone 15 Pro with iOS 26.7 achieves approximately 1 second of inference latency per decision, with tokenization and inference executed locally and downloaded model files cached for reuse.

• Training data composition. The 10 million general decisions are split almost evenly across decision types (3,333,334 choice, 3,333,333 noul, 3,333,333 score) and span eight task categories, with semantic matching (24.067% of training) and review rating (19.438%) the largest. Twenty public sources plus 10,700 programmatic rule decisions contribute to the corpus.

Methodology in Plain English

The authors start from the observation that many tasks need a bounded answer rather than generated text. They therefore frame every task as scoring a set of candidate answers and producing a probability distribution over them.

Data. They search Chinese datasets (papers, author repositories, Hugging Face) for annotations that can be expressed as candidate decisions. Each source annotation is rewritten into a common format: a context, an instruction, a decision type, a candidate list, and a target distribution. Single-answer questions and category labels become choice decisions with all probability mass on the annotated answer; true/false or proposition annotations become noul decisions with the target ordered as (false, true); ratings become score decisions on the original ordered scale, and continuous ratings are spread across adjacent integer levels per rater and then averaged so the soft target keeps the mean and inter-rater variation; multiple-answer questions are decomposed into one noul proposition per option. The team also generates 10,700 rule-based decisions for condition checking, counting, and grading. Duplicate and conflicting supervision is removed using a hash fingerprint over the task identifier, decision type, normalized context, instruction, and canonicalized candidates — sorting candidates for choice so that reordering independent alternatives does not create a false conflict, while keeping score order and noul false/true semantics fixed. Decisions derived from the same source material are kept in the same partition (training, development, calibration, or test) to reduce leakage. Decision-type budgets aim for roughly one third each of choice, noul, and score in the general, legal, and financial corpora; the medical corpus contains a larger noul share because of its matching and multiple-label tasks.

Model and training. The model serializes decision type, instruction, all candidates, and context into a single sequence with a marker before each candidate. A bidirectional mmBERT encoder contextualizes the sequence, a learned type embedding is added at every position, a decision Transformer with two layers processes the result, and a shared MLP scores all candidate markers in one forward pass, with a softmax producing the decision probabilities. Readouts differ by type: the highest-probability candidate for choice, the probability of true for noul, and an expected zero-based level index (or the expected value on the source rating scale) for score. Training combines cross-entropy against the target distribution with an RLCD policy-gradient term over perturbed candidate logits, where the reward uses log and spherical scores and adds a ranked probability score for ordered score decisions.

Experimental setup. The encoder and head are initialized from Laya Multilingual, with an mmBERT-base backbone of 22 layers and hidden size 768 plus the head, totaling approximately 322 million parameters. Training uses full-parameter fine-tuning on eight NVIDIA H200 GPUs: one epoch on the general corpus, then four epochs per domain starting independently from that checkpoint. Medical specialization uses 3,000,000 decisions; legal and financial specialization use 24,000 each. RLCD samples four logit perturbations per decision in both stages. Evaluation uses accuracy, expected calibration error

Authors’ abstract

System One models such as Jev offer an efficient alternative to generative language models for tasks that require decisions rather than open-ended responses. However, existing Jev models exhibit limited Chinese-language decision accuracy, restricting their utility in both general and specialized settings. In this paper, we introduce Chinese-Jev, a System One model that addresses this gap through a unified data processing and training pipeline. Our data processing protocol converts heterogeneous Chinese-language annotations into probability targets over candidate options, enabling a shared training formulation across domains and question formats. To enable efficient inference, Chinese-Jev adopts a lightweight encoder-only backbone for text encoding and learns to score candidate answers through decision-oriented training. To address the misalignment between the pre-training distribution and downstream Chinese-language scenarios, we first train the model on a general-purpose corpus of 10 million examples, then fine-tune it separately for the medical, legal, and financial domains. To evaluate decision accuracy and calibration in both general and domain-specific Chinese-language settings, we introduce Chinese-Jev Bench (CJ-Bench). After first-stage pre-training, Chinese-Jev exceeds the accuracy of the closed-source Jev model by 1.24% on general-domain tasks while achieving a 20.3x speedup. Subsequent domain-specific fine-tuning yields a 4.0% accuracy improvement over Jev in medicine and achieves 92% of Jev's average accuracy across specialized domains, with a 17x speedup and an average latency of only 15 ms per example. We further demonstrate on-device deployment of an INT8-quantized model on mobile devices, achieving an inference latency of approximately 1.0 second per decision. The project is available at https://gulucaptain.github.io/Chinese-Jev/.