HalleluBERT: Let Every Token That Has Meaning Bear Its Weight

AI summary

Overview

Research area: Natural Language Processing — monolingual (Hebrew) transformer encoder pre-training, language resources, and benchmarking.

Technical level: Intermediate. The paper assumes familiarity with BERT/RoBERTa-style masked language modeling, subword tokenization, and standard fine-tuning evaluation, but it is written as an empirical resource paper rather than a new-architecture paper.

Scope: The paper describes the training, release, and benchmark evaluation of HalleluBERT, a RoBERTa-based encoder family for Modern Hebrew released in base and large variants.

What This Paper Is About

Hebrew has BERT-style models, but the available RoBERTa-style Hebrew encoder (HeRo) was trained with comparatively limited compute, does not fully report key training details such as effective batch size or total update steps, and no large-scale Hebrew model existed. The goal of this work is to train Hebrew base and large RoBERTa encoders from scratch on a large cleaned Hebrew corpus and to test them against existing Hebrew-specific and multilingual models on native Hebrew named entity recognition and sentiment classification benchmarks.

Key Contributions

1. The authors trained and released HalleluBERT base and HalleluBERT large, RoBERTa-style encoders for Hebrew, trained from scratch on 49.1 GB of Hebrew text (47.5 GB HeDC4 plus 1.6 GB Hebrew Wikipedia) with a Hebrew-specific 52k byte-level BPE vocabulary.
2. They benchmarked HalleluBERT on named entity recognition (BMC, NEMO) and sentiment classification (SMCD), producing a combined performance analysis against previous monolingual and multilingual models.
3. They released model weights and the tokenizer under the MIT license to support reproducible Hebrew NLP research, along with reported hyperparameters, computation times, and sequence-length statistics for reproducibility.
4. They provide an explicit discussion of why they could not reproduce HeRo's reported state-of-the-art results under their standardized fine-tuning pipeline.

Main Findings

• Best NER results from the base model: HalleluBERT base achieved the highest micro-F1 among base models on both BMC (93.33) and NEMO (87.06), giving the strongest NER average (90.20).
• Base slightly beats large on BMC: HalleluBERT base marginally outperformed HalleluBERT large on BMC (93.33 vs. 93.23), which the authors suggest may reflect BMC's limited size or domain coverage. Large models retained a small but consistent advantage on NEMO and on the NER average column.
• Sentiment classification favored other models among base models: AlephBERT obtained the highest F1 among base models on classification (83.66), slightly ahead of mmBERT base; HalleluBERT base was competitive but did not lead this task.
• Largest model led classification: HalleluBERT large achieved the highest classification score (84.91), surpassing XLM-RoBERTa large by over one point.
• Best combined average: On the overall unweighted mean across BMC, NEMO, and SMCD, HalleluBERT large attained the highest score (88.95), with HalleluBERT base second (87.83). XLM-RoBERTa large scored 87.49.
• Monolingual beat multilingual: HalleluBERT outperformed both multilingual models (XLM-RoBERTa and mmBERT) across all tasks, which the authors interpret as support for monolingual pre-training for Hebrew.
• Multilingual comparison within family: XLM-RoBERTa base outperformed mmBERT base despite having fewer parameters.
• HeRo not reproduced: The authors were unable to reproduce HeRo as the reported state of the art under their standardized fine-tuning pipeline and hyperparameter search; AlephBERT-Gimmel base ranked second among base models and outperformed HeRo in both NER and overall averages.
• Convergence behavior: Base models typically stabilized after 20k–30k pre-training steps, while large models converged by around 40k steps, mirrored by validation perplexity evaluated once per epoch.
• Compute for downstream experiments: All Hebrew downstream experiments together required roughly 65 hours of GPU time (65:47, approximately 2.74 days): BMC 15:43, NEMO 27:54, SMCD 22:10.

Methodology in Plain English

The author combined the deduplicated HeDC4 Hebrew web corpus (47.5 GB) with a Hebrew Wikipedia dump (1.6 GB) for about 49.1 GB of pre-training text, shuffling documents to avoid ordering effects from source-specific crawls. Rather than reuse the English-optimized GPT-2 tokenizer, the author trained a Hebrew-specific byte-level BPE vocabulary of 52k subwords on 20 GB of shuffled Hebrew text drawn from the same sources, following the strategy used for GottBERT.

Pre-training used the fairseq framework on a 128-core TPUv4 pod in full precision (no mixed precision). Both models followed the standard RoBERTa schedule: 100k update steps, a global batch size of 8k, a 10k-step warmup, and polynomial learning rate decay, with peak learning rates of 0.0004 for base and 0.00015 for large. Base training took about 30.2 hours; large took about 6.0 days. The dataset permitted roughly 61 epochs.

Evaluation followed the HeRo benchmark suite but was restricted to NER (BMC, NEMO) and sentiment classification (SMCD), using an NNI- and Hugging Face transformers-based grid-search pipeline. For each model and task, a small grid search covered batch sizes {16, 32} and learning rates {5e-6, 7e-6, 1e-5, 2e-5, 5e-5}, with the best configuration chosen by validation performance. Training was capped at 30 epochs with early stopping at patience three, a linear learning rate schedule with 10% warmup, a single fine-tuning run per configuration, and a fixed random seed of 42. All experiments ran on two NVIDIA RTX 3090 GPUs. NER results are reported as micro-F1; SMCD results as macro-F1 (on the deduplicated 8,465-sample version of the OmiLab sentiment dataset, split into roughly 72% train, 8% validation, 20% test).

Why This Matters

Impact on research. The work fills a stated gap: a RoBERTa-style Hebrew encoder trained at scale and released in both base and large variants, with weights, tokenizer, hyperparameters, and compute figures disclosed. It also provides a concrete data point that monolingual pre-training can still beat multilingual encoders on native Hebrew benchmarks, and it documents a reproduction gap with a prior Hebrew model.

Real-world applications

• Hebrew named entity recognition pipelines, such as extracting people, organizations, and locations from Hebrew news, social media, or documents.
• Hebrew sentiment analysis of user-generated social media comments, as modeled by the SMCD benchmark.
• Hebrew text classification and tagging backends in products that currently rely on multilingual models, where a Hebrew-specific encoder may give better accuracy.
• Research and teaching infrastructure: an MIT-licensed base/large Hebrew encoder that others can fine-tune without training from scratch.

Industry relevance. The release of both base (126M parameters) and large (357M parameters) variants under MIT gives practitioners a choice between a compact model comparable in size to AlephBERT and HeRo, and a larger model comparable in scale to mid-sized multilingual encoders. Public reporting of pre-training and fine-tuning compute (about 65 hours of RTX 3090 time for the downstream experiments) helps teams estimate cost before adopting or reproducing the setup.

Future Directions

1. Experiment with Whole Word Masking (WWM), which was not applied in this work.
2. Run ablations on byte-level BPE vocabulary size (for example, 32k vs. 52k vs. 128k) to clarify the performance–efficiency trade-off in sequence length and throughput, and conduct broader ablations and throughput analyses.
3. Extend evaluation to additional native Hebrew benchmarks and to tasks the paper omitted, such as QA, NLI, or paraphrase-style tasks, and to long-context settings via variants such as LongHeRo.
4. Perform a detailed error analysis of model outputs, for example common NER boundary errors and sentiment misclassifications, and explore longer training schedules or larger configurations that mixed-precision training would make more feasible.

Target Audience

Researchers and engineers working on Hebrew or other low- and mid-resource language NLP; practitioners who need an open, MIT-licensed Hebrew encoder or a RoBERTa-style baseline for NER and sentiment classification; and benchmarking or reproducibility researchers interested in how pre-training budget, tokenizer choice, and fine-tuning pipeline interact. The paper is not reported to include a detailed error analysis or multi-seed results, so readers seeking those should look elsewhere.

Authors’ abstract

Transformer-based models have advanced NLP, yet Hebrew still lacks a RoBERTa encoder that is trained at scale and released in both base and large variants. We present HalleluBERT, a RoBERTa-based encoder family trained from scratch on 49.1~GB of deduplicated Hebrew web text and Wikipedia using a Hebrew-specific byte-level BPE vocabulary. On native Hebrew benchmarks for named entity recognition (BMC, NEMO) and sentiment classification (SMCD), HalleluBERT outperforms monolingual and multilingual baselines, and yields the highest unweighted mean score across the three benchmarks. We release model weights and tokenizer under the MIT license to support reproducible Hebrew NLP research.