NOWJ @BioCreative IX ToxHabits: An Ensemble Deep Learning Approach for Detecting Substance Use and Contextual Information in Clinical Texts

AI summary

Overview

Research area: Clinical Natural Language Processing (NLP) / Named Entity Recognition (NER) in a low-resource, domain-specific language setting.

Technical level: Intermediate. Readers should know roughly what BERT is, what token classification means, and what precision/recall/F1 measure. The paper explains CRF decoding explicitly, so deep prior knowledge of sequence modeling is not required.

Scope: A description of the NOWJ team's winning-style submission to the ToxHabits Shared Task at BioCreative IX (IJCAI 2025), which detects toxic substance mentions and their contextual attributes in Spanish clinical case reports using an ensemble of multi-output BERT-CRF models rather than large language models.

What This Paper Is About

Unstructured electronic health records contain valuable information about patient substance use — what drug, how much, how often, by what route — but extracting that information reliably from free text is hard, especially in Spanish clinical writing where annotated data is scarce. The ToxHabits Shared Task posed exactly this problem, asking teams to identify substance "Trigger" spans (Tobacco, Cannabis, Alcohol, Drug) and their "Argument" spans (Type, Method, Amount, Frequency, Duration, History) in 1,499 Spanish clinical case reports. This paper presents the NOWJ team's system for that task: a deliberately non-LLM, ensemble-based approach that jointly handles both subtasks in a single architecture.

Key Contributions

1. A multi-output BERT-CRF architecture that shares a single BETO encoder but branches into two task-specific label heads (one for Triggers, one for Arguments), predicting both subtask outputs simultaneously from one model rather than training two separate systems.

2. A sentence filtering stage — a lightweight, independently trained binary BETO classifier that discards sentences containing no triggers or arguments before they reach the main model, reducing computation and, in most runs, raising precision.

3. A structured ensemble strategy combining resampled training subsets (k-fold splits of both full and partial datasets) with three class-imbalance mitigation techniques: label-weighted loss, data oversampling, and weighted random sampling.

4. A documented argument for conventional encoder models over LLMs in clinical settings, citing concerns about trustworthiness, controllability, and inference efficiency — and backing this with competitive official results.

Main Findings

• Strong trigger detection: The best Subtask 1 (ToxNER) run — "BERT Ensemble + Sentence Filtering + Tuning (full train)" — achieved 0.94 F1 and 0.97 precision, the highest precision of any configuration tested.

• Solid argument detection: The best Subtask 2 (ToxUse) result was 0.91 F1 from the plain full-train ensemble, with the tuned and filtered variant reaching 0.95 precision but dropping to 0.90 F1 due to lower recall.

• Sentence filtering helps precision, sometimes hurts recall: Filtering raised Subtask 1 F1 from 0.93 to 0.94 on the full dataset, but for Subtask 2 it pushed F1 down marginally (0.91 to 0.90) by removing some true positives along with the noise.

• Full training data dominates partial data: Models trained on the full dataset consistently outperformed partial-data models by a wide margin — roughly 0.10 F1 on Triggers (0.94 vs. 0.84) and 0.14 on Arguments (0.91 vs. 0.77).

• Tuning produces a precision-recall trade-off: The fine-tuned configuration delivered the highest precision on both subtasks while sacrificing recall, useful when false positives are more costly than misses.

• Ensemble size varies by regime: The full-dataset configuration used 6 models; the partial-dataset configuration used 19 models with more diverse sampling strategies.

Methodology in Plain English

The researchers treat both subtasks as a single token-labeling problem. Each clinical document is first split into sentences. A small binary classifier — a fine-tuned BETO model — then looks at each sentence and asks: does this contain anything worth extracting? Sentences that don't are thrown away, which saves computation on the many clinical sentences that mention no substances at all.

The surviving sentences are tokenized and tagged with BIO labels (Beginning, Inside, Outside), marking where each entity span starts and continues. These tags are fed into the core model: a shared BETO encoder that produces contextual word representations, followed by two parallel branches — one projecting to trigger labels, one to argument labels. Each branch applies a Conditional Random Field (CRF) on top of its output scores. The CRF is important because it models which label transitions are legal (for instance, an "Inside" tag should generally follow a "Beginning" tag, not appear out of nowhere), and decodes the globally best label sequence using the Viterbi algorithm.

Because the dataset is small and imbalanced, the team trained many copies of this model on different random subsets of the data, using three tricks to stop the model from ignoring rare classes: giving underrepresented labels more weight in the loss, duplicating sentences that contain entities, and sampling informative sentences more often during training. At prediction time, all models vote on each token, and the majority answer wins. Subword predictions are then stitched back into whole-word spans and normalized.

The loss function is simply the sum of the two CRF losses, weighted equally (both weights set to 1). Training used BETO-base, AdamW, a learning rate of 5e-5, batch size 8, dropout 0.1, and 5 epochs — about one hour of training per model on two NVIDIA Tesla T4 GPUs. The separate sentence filter took about 2.5 hours to train.

Why This Matters

Impact on research: This work demonstrates that carefully engineered encoder-based systems with ensembling and domain-appropriate pretraining can still match or beat more fashionable LLM approaches on low-resource clinical extraction, at a fraction of the computational cost. It also provides a reproducible recipe — sentence filtering plus multi-task CRF ensembling — that other teams tackling similar Spanish or low-resource clinical NER tasks can adapt directly.

Real-world applications:

• Clinical documentation and chart review: Automatically populating structured substance-use fields from narrative case reports, cutting manual abstraction time.
• Public health surveillance: Aggregating population-level patterns of alcohol, tobacco, cannabis, and drug use from hospital records at scale.
• Patient risk stratification: Flagging patients with heavy or polysubstance use patterns for targeted intervention or follow-up.
• Clinical research cohort building: Identifying eligible patients for studies on substance use disorders by querying extracted entity spans instead of raw text.

Industry relevance: For electronic health record vendors, clinical NLP startups, and hospital informatics teams, this paper offers a practical template that runs on modest hardware (two T4 GPUs), avoids the governance and privacy complications of sending patient text to external LLM APIs, and provides the precision/recall control needed to tune a system for either high-sensitivity screening or high-precision documentation workflows.

Future Directions

1. Adaptive loss weighting: The team fixed both task weights at 1 for simplicity and note that when trigger and argument frequencies differ substantially, tuning these weights could improve the joint optimization.

2. Integrating LLMs selectively: The authors explicitly left LLM integration for future work, suggesting hybrid systems where a conventional encoder handles span detection and an LLM contributes contextual reasoning or generalization.

3. Recovering the recall lost to filtering: The sentence filter improved precision but cost recall on argument detection; a softer filtering scheme, a threshold-based approach, or a filter trained on arguments specifically could recover this.

4. Cross-lingual and cross-domain transfer: Since the approach relies on BETO (Spanish-specific) and a small annotated corpus, testing how well it transfers to other Romance-language clinical text or other clinical NLP tasks would clarify how generalizable the recipe is.

Target Audience

Clinical NLP researchers and shared-task participants will get the most value from this paper, particularly those working on Spanish or other low-resource clinical text, named entity recognition, or multi-task sequence labeling. Practitioners building production healthcare information-extraction systems — including hospital informatics teams and EHR vendors — will find the architecture and the precision/recall tuning results directly actionable. Machine learning engineers interested in ensembling and CRF-based decoding on small datasets will also find the training-strategy discussion useful. Readers looking for a deep dive into LLM prompting for clinical extraction will not find it here; this is deliberately a non-LLM paper.

Authors’ abstract

Extracting drug use information from unstructured Electronic Health Records remains a major challenge in clinical Natural Language Processing. While Large Language Models demonstrate advancements, their use in clinical NLP is limited by concerns over trust, control, and efficiency. To address this, we present NOWJ submission to the ToxHabits Shared Task at BioCreative IX. This task targets the detection of toxic substance use and contextual attributes in Spanish clinical texts, a domain-specific, low-resource setting. We propose a multi-output ensemble system tackling both Subtask 1 - ToxNER and Subtask 2 - ToxUse. Our system integrates BETO with a CRF layer for sequence labeling, employs diverse training strategies, and uses sentence filtering to boost precision. Our top run achieved 0.94 F1 and 0.97 precision for Trigger Detection, and 0.91 F1 for Argument Detection.