From Retrieval to Typed Decisions: Calibrated System One Models from Biomedical Sentence Encoders

AI summary

Overview

Research area: Natural language processing — biomedical text mining, sentence-encoder retrieval models, and calibrated “System One” typed-decision models.

Technical level: Intermediate. The paper assumes familiarity with Sentence-Transformers, cross-encoders, cross-entropy training, temperature scaling and proper scoring rules, but the core question (does retrieval training help downstream decisions?) is stated plainly.

One-sentence scope: The paper converts eleven biomedical and general-domain sentence encoders into typed-decision models using four conversion heads and six training objectives, evaluates them on a new biomedical decision suite plus 243k training decisions derived from NLM indexing, and finds that retrieval initialisation helps mainly when the conversion keeps the retrieval prior, while the released RLCD objective trails plain cross-entropy.

What This Paper Is About

Biomedical software often needs small, schema-constrained answers — is this abstract a randomised trial, did the study include human participants, how strong is the evidence — returned as probabilities that code can threshold to act, defer or escalate. Existing System One models that do this are general-domain and start from masked-language-model or decoder checkpoints, while many biomedical encoders have been trained contrastively for retrieval. This paper asks whether those retrieval-trained sentence encoders are a good starting point for typed-decision models, and how much that depends on the conversion head and the training objective.

Key Contributions

1. sbert2s1, a framework that converts Sentence-Transformers encoders into typed decision models with four conversions: Z (frozen encoder, per-type scale only), B (bi-encoder), C (a cross head that replicates Laya’s mask-slot head) and PFR (a prior-fused residual head that adds the cross score to the retrieval prior).
2. BioDecide, a biomedical typed-decision suite with public grounded and knowledge tracks plus credentialed clinical tracks, and MEDLINE-S1, 242,986 training decisions over 40k abstracts labelled from NLM MEDLINE indexing rather than an LLM teacher (test: 5k abstracts, 30,522 decisions).
3. A controlled study over eleven encoders, six parent-retriever pairs and a matched grid of two heads and five training objectives (plus PG w/o CE for PFR), with three seeds per cell.
4. An analysis showing that RLCD optimises a noise-smoothed proper score, that the released estimator is biased and rescaled, and that a leave-one-out baseline (PG-LOO) removes both problems and recovers most of the accuracy gap.

Main Findings

• Retrieval training improves zero-shot content matching. Untrained contrastive checkpoints reach 15–24 chance-normalised points on questions whose options differ in content (disease areas, study designs, specialties, rubrics, exam answers), against 0–7 for their MLM parents and 20.3 for Laya-large. On options that differ only by a label word (yes/no, true/false), cosine gaps are of order 10⁻³ and no encoder is reliably above chance.
• After fine-tuning, the benefit depends on the head. With full data and the cross head the contrastive stage is roughly neutral: S-PubMedBERT 67.7 and PubMedBERT 67.3, both MPNet checkpoints 66.4, BioClinical-ModernBERT 72.9 and 73.1, with the only Holm-significant difference being negative (GTE-ModernBERT −2.5). With PFR the retriever is better in three of five pairs (all-MPNet +4.5, Nomic +2.7, GTE +2.9), worse for S-PubMedBERT (−1.6), and ModernBERT embedders gain +10.9 on held-out tasks.
• Scarce labels widen the PFR advantage. For PFR, retrievers are ahead in four of five pairs at both 10% and 2% (significant in three and four; Nomic +5.3 and +9.2, GTE +4.7 and +7.4, up to +16.3 on held-out tasks). For C, ModernBERT-family retrievers are 2.5–3.7 points behind their parents at 10%, BioClinical-ModernBERT’s embedder is 8.5 points behind at 2%, and the only gain is S-PubMedBERT at 10% (+1.9).
• Overall tally. Across all 15 pair-and-data-size comparisons, retrieval training is significantly better in ten and worse in one with PFR, and better in one and worse in five with C.
• The cross head is more accurate. In the matched grid, C exceeds PFR by 1.3–2.4 seen-task points for all five shared objectives (Holm p ≤ .011 for CE, Proper and PG; .08 and .14 for PG-LOO and Reparam) and by 3.8–10.2 points on held-out tasks.
• Prior fusion reduces option-order sensitivity. A random permutation changes 0.3% of S-PubMedBERT PFR answers versus 15.7% for C, whose flips concentrate on unseen inputs (held-out 17%, exams 28%, seen tasks 3%). PFR also has lower ordinal error on Druglib for every objective.
• The released RLCD recipe trails cross-entropy. With either head, CE, Proper and Reparam are within one point of each other, while PG trails CE by 3.0 (PFR) and 2.5 (C) points (Holm p = .005), with a fourfold larger seed sd for PFR (1.5 vs. 0.4). Adding the spherical and ranked-probability terms does not significantly improve accuracy (Proper − CE +0.2 and +0.5, n.s.), and removing CE from PG has little effect (+0.2).
• The deficit comes from reward normalisation. PG-LOO gains 2.5 (PFR) and 1.5 (C) points over PG (Holm p = .005) and is within 0.8 points of Reparam (n.s.). A matched probe shows the released estimator points in the right direction (cosine 0.98 with the true smoothed gradient) but is 3.6–3.8, 5.7–6.1 and 14–15 times too large at σ = 0.4, 0.25 and 0.1, so the score term outweighs CE by 5–20× instead of 1.3×. The pathwise estimator is 100–1,500 times less noisy at matched scale, yet PG-LOO trains nearly as well, so estimator variance is secondary.
• No clear calibration benefit after scaling. PG has the lowest raw ECE (7.0 vs. 9.3 for CE with PFR), but after temperature scaling all objectives lie between 3.6 and 4.4, and PG has the worst NLL.
• Fine-tuning degrades retrieval. SciFact nDCG@10 of the pooled embedding falls from 67.6 (Z) to 43.3 (C), 35.6 (PFR), 34.3 (B) and 24.6 for C trained with CE; only a frozen prior keeps the retriever intact.
• Encoders versus LLMs. Fine-tuned encoders are competitive on seen tasks: S-PubMedBERT C trained with CE (70.2) and Laya-large (75.0) exceed zero-shot Gemma-4-31B (65.7) and Qwen3.8-27B (57.4), with better calibration after scaling (ECE 3.8–4.3 vs. 8.5–9.1). LLMs win on held-out tasks (Gemma 52.6 vs. 31.0), the clinical track (81.6–83.4 vs. at most 38.4 zero-shot, 55.2 with clinical training) and exams. Routing the 20% least confident questions to Gemma raises macro accuracy from 59.5 to 68.4 for S-PubMedBERT C (CE) and from 64.3 to 69.1 for Laya-large, against 62.1 for Gemma alone.
• Data integrity. An exact-input audit found nine DDI and seven Druglib training duplicates in calibration splits and two SciFact and 13 Druglib records in test splits; removing all affected test inputs lowers seen-task scores by 0.3–0.4 points for every model and changes no ranking.

Methodology in Plain English

A typed decision pairs a state (a text or record) with typed questions — choice (one option from a schema), score (a distribution over an ordinal rubric) and noul (the probability that a statement holds). The model returns a distribution over each question’s options in one forward pass, and downstream code thresholds the confidence.

The authors take existing sentence encoders and bolt on one of four conversion heads. Z freezes the encoder and fits only a per-type scale on cosine similarity. B trains the encoder and that scale. C is a cross-encoder head that encodes the instruction and mask-slot options together with the state and scores each mask slot, replicating Laya’s head so the two can load each other’s weights. PFR adds the cross score as a residual on top of the retrieval prior, with the scorer’s last layer zero-initialised so an untrained PFR model equals Z.

Training objectives are compared on identical data and schedule: cross-entropy (CE); Proper, which maximises a combined proper scoring rule without noise; PG, the released RLCD recipe; PG without CE; PG-LOO, which replaces PG’s same-sample mean baseline and reward normalisation with a leave-one-out baseline; and Reparam, the reparameterisation trick. After training, one temperature per question type and option-count bucket (K = 2, 3–5, 6–10, ≥11) is fitted on a calibration split carved from the training data before training, so every metric is reported before and after scaling.

Evaluation data comes from converting existing biomedical datasets into typed decisions (BioDecide: 371,425 train and 70,753 test decisions), plus MEDLINE-S1 derived from NLM indexing, plus credentialed clinical data processed only on institutional hardware. Statistically, the headline metric is chance-normalised accuracy averaged over question groups within a task and then over tasks, with paired bootstrap intervals (10,000 draws) over document clusters and Holm correction within each research question. Long clinical notes are handled by truncation, sliding windows, or retrieve-then-decide, where the model’s own embedding picks the m = 3 most similar chunks.

Why This Matters

Impact on research. The paper turns a plausible but untested assumption — that retrieval-trained encoders are a good starting point for decisions — into a measurable, head-dependent result, and it shows that a widely used open objective (RLCD) is misspecified in a way that a simple estimator change largely fixes. It also releases a biomedical typed-decision suite and a training-set source (NLM indexing) that avoids LLM-teacher labels.

Real-world applications.

• Triage of biomedical literature: flagging study design, human-versus-animal subjects, adverse effects or drug therapy in PubMed citations.
• Evidence and claim checking: deciding whether an abstract supports, refutes or neither supports a claim.
• Patient-facing review analysis: scoring drug-review effectiveness, side-effect severity and overall rating.
• Routing and escalation: acting on confident cases and deferring the rest to a larger LLM or a human.

Industry relevance. A base-size encoder with a cross head reaches 73.1 (BioClinical-ModernBERT) or 70.2 (S-PubMedBERT with CE), close to fine-tuned Laya-large (75.0) at about a third of its size, and a ten-question request takes 6.6 ms for C, 15.2 ms for PFR and 22.7 ms for Laya-large on one MI300X. That makes calibrated System One models a practical first stage in a cascade.

Future Directions

• Tune objectives separately and run the head × objective grid on more than one encoder, since results cover base-size encoders, one fixed training budget and hyperparameters inherited from Laya.
• Test conversions on larger and decoder-based backbones, given the LLMs’ advantage on held-out, clinical and exam tasks.
• Build independent replications of the retrieval-initialisation comparison rather than six parent-retriever pairs that share parents and differ in retrieval corpora.
• Validate calibration and thresholds under distribution shift and on de-duplicated splits, since low ECE on seen tasks does not carry over, option-order sensitivity of cross heads grows under shift, and temperatures were not refitted on de-duplicated data.
• Improve long clinical-note handling, where strategies remain close to chance for base encoders.

Target Audience

Researchers and engineers working on biomedical NLP, dense retrieval and calibrated decision models; practitioners building low-latency, schema-constrained classifiers or LLM cascades; and anyone implementing or evaluating RLCD-style objectives, who will find the analysis of the reward-normalisation bias and the leave-one-out fix directly applicable.

Authors’ abstract

Typed decision models answer schema-constrained questions about a text in one forward pass and return probabilities meant to be thresholded. We ask whether biomedical sentence encoders trained for retrieval are good starting points for such models. We present SBERT2S1, which converts Sentence-Transformers encoders into bi-encoder, cross-head (C) and prior-fused residual (PFR) decision models, together with BIODECIDE, a biomedical typed-decision suite, and MEDLINE-S1, 243k training decisions derived from NLM indexing. Across six parent-retriever pairs, retrieval training improves zero-shot matching of content-bearing options. After fine-tuning, its effect depends on the head: across five pairs and three training-set sizes, retrieval training significantly helps PFR, which keeps the retrieval prior, in 10 of 15 comparisons, but helps C in one and hurts it in five. A matched grid of two heads and five training objectives shows that C outperforms PFR under every objective, and that the released RLCD recipe of open System One models trails cross-entropy by 2.5-3.0 points. The deficit stems mainly from its reward normalisation, which inflates the noisy score-function term 3.6-15-fold; an unbiased leave-one-out estimator recovers most of the gap. After temperature scaling, no objective is clearly better calibrated than cross-entropy. We release the code, the MEDLINE-S1 labels and a model.