Forging GEMs: Advancing Greek NLP through Quality-Based Corpus Curation

AI summary

Overview

• Research area: Natural Language Processing — monolingual and domain-specific transformer language models for Modern Greek, with a focus on the legal domain.
• Technical level: Intermediate. The paper assumes familiarity with transformer encoder architectures (BERT, RoBERTa, ELECTRA, ConvBERT, Longformer, ModernBERT) and with standard NLU benchmarks, but the core arguments about corpus curation and evaluation are accessible.
• Scope: The paper introduces the Greek Embedding Models (GEMs) family — seven encoder models pre-trained from scratch on curated general, legal, and bilingual Greek corpora — and benchmarks them against existing Greek state-of-the-art models on named entity recognition, legal topic classification, and natural language inference.

What This Paper Is About

Greek, like many morphologically rich and moderately-resourced languages, lacks large, well-curated training corpora and has largely been confined to early transformer architectures such as BERT and RoBERTa, usually with a 512-token context limit. The authors address both problems at once: they build a set of deliberately cleaned and configured Greek corpora (general, legal, and Greek–English legal), then pre-train a family of modern encoder architectures on them to test whether better data curation and broader architectural coverage translate into measurable gains on Greek NLU tasks.

Key Contributions

1. A new model family (GEMs). The authors develop seven models covering both established architectures (RoBERTa, Longformer) and architectures not previously applied to Greek (ELECTRA, ConvBERT, ModernBERT), plus two bilingual models, GEM*-RoBERTa and GEM*-XLM-RoBERTa. This is described as the first systematic training and evaluation of a suite of Greek models with an extended 1024-token context window.
2. New curated corpora. They construct and release a large general-domain corpus alongside multiple specialized legal corpora, including a 26GB bilingual Greek–English legal corpus combining proprietary Greek legal data with an English subset of the Pile of Law. This is the first bilingual Greek–English embedding model effort tailored to the legal domain.
3. Data-centric training strategies. They show that targeted repetition of high-quality legal sub-corpora (repetition factors of up to 4x, expanding a 16.75GB legal corpus to an effective 21.12GB) is an effective route to domain adaptation.
4. A comparative and statistical benchmark. They evaluate all models on NER, multi-class legal topic classification, and NLI, and apply Friedman Aligned-Ranks with Finner post-hoc tests to establish that the gains are statistically significant.

Main Findings

• GEM-RoBERTa and GEM-ConvBERT lead overall. These two models deliver the best overall performance across all three benchmarks, substantially outperforming Greek-BERT, Greek-Legal BERT, and Greek-Legal RoBERTa. The abstract reports accuracy gains of up to 3.6% over established state-of-the-art models, and a Friedman Aligned-Ranks / Finner post-hoc analysis confirms statistical superiority across multiple metrics.
• The largest single reported gain appears on NLI. GEM-RoBERTa trained on the General-Domain corpus reaches 75.7 accuracy and 92.0 AUC, against 72.1 accuracy and 91.4 AUC for Greek-BERT — a 3.6-point accuracy difference.
• Modern architectures are competitive. GEM-ELECTRA and GEM-ModernBERT report competitive and sometimes superior results relative to the traditional Greek embedding models, which the authors treat as validation of these architectural innovations for Greek.
• Longformer variants are the weakest but still improved. Longformer-based models generally show the weakest performance across the evaluated architectures, yet GEM-Longformer consistently beats the Greek-Media-Longformer baseline (for example, on NER with the HQ Repeated corpus, GEM-Longformer scores 96.7 accuracy / 98.1 AUC / 38.9 GM versus 95.9 / 97.3 / 22.8 for Greek-Media LongFormer).
• Corpus choice interacts with task. On NER, every GEM variant does best with the HQ-Repeated corpus except GEM-RoBERTa, which prefers the Legal corpus. On legal topic classification, HQ-Repeated is best for all variants except GEM-ConvBERT, which prefers the Legal corpus. On NLI, all GEM models do best with the General-Domain corpus.
• Best headline numbers per benchmark. NER best accuracy is 97.5 (GEM-RoBERTa on Legal and on General-Domain, tying Greek-BERT's 97.4 and exceeding it); the best NER GM is 54.7 for GEM-RoBERTa on the Legal corpus versus 54.1 for Greek-Legal RoBERTa. On legal topic classification (Volume level), GEM-ConvBERT on HQ Repeated reaches 90.6 accuracy / 99.1 AUC / 88.9 GM and GEM-RoBERTa on HQ Repeated reaches 90.5 / 99.1 / 88.9, both above Greek-Legal BERT (89.4) and Greek-Legal RoBERTa (89.2). On the Subject level, GEM-RoBERTa on HQ Repeated achieves 84.9 accuracy / 99.4 AUC / 57.6 GM, ahead of Greek-Legal RoBERTa (83.7 / 99.4 / 47.2).
• Bilingual models are competitive. GEM*-RoBERTa and GEM*-XLM-RoBERTa, both trained on the Bilingual Legal Corpus, perform competitively across all benchmarks and occasionally achieve the best overall results, with GEM*-RoBERTa slightly ahead of GEM*-XLM-RoBERTa.
• Tokenizer study favors WordPiece on legal text. Evaluated on a held-out Greek dataset of roughly 150 million tokens, WordPiece trained on the Legal Corpus achieved the best balance among the authors' own tokenizers (fertility 1.334, bytes/token 9.02). BPE variants were competitive (fertility 1.340 on the General-Domain corpus, 1.355 on the Legal corpus). A Unigram tokenizer with a 128,000-token vocabulary showed substantially higher fertility (1.601) and lower bytes/token (7.51), suggesting over-segmentation of Greek words despite the large vocabulary.
• DeBERTa was dropped. DeBERTa was evaluated but performed poorly relative to all other models, so it was not included in the final family. The authors suggest it may need different hyperparameters or longer pre-training, and note that DeBERTa uses the Unigram tokenizer, which performed poorly on Greek morphology in their ablations.
• The NLI results table is truncated in the supplied content. Complete NLI figures are reported for Greek-BERT, Greek-Legal BERT, Greek-Legal RoBERTa, Greek-Media LongFormer, and the three GEM-RoBERTa corpus variants (Legal 70.1 / 89.3 / 70.0; HQ Repeated 69.3 / 88.5 / 69.3; General-Domain 75.7 / 92.0 / 75.7). The remaining rows are not visible in the provided text.

Methodology in Plain English

Step 1 — Gather raw sources. The researchers combined legal and non-legal Greek text. Legal sources included official government publications (ΦEK, 11.00GB), Europarl (0.38GB), EUR-Lex (0.92GB), Greek Parliament proceedings (2.90GB), the Raptarchis legal dictionary (0.35GB), and the Political Reports of the Supreme Court (1.20GB). Non-legal sources were the Greek Wikipedia (1.30GB) and the Greek portion of OSCAR (41.00GB), the largest single dataset. For the bilingual corpus they added an English subset of the Pile of Law (9.20GB), producing a 26GB Greek–English legal corpus that is approximately 60% Greek and 40% English.

Step 2 — Clean aggressively. Because OSCAR is web-crawled and noisy, they applied a dedicated pipeline: dropping records under 1500 characters, dropping records containing any word longer than 60 characters, dropping records containing "lorem ipsum," removing records originating from Wikipedia (already included separately), and stripping emails, URLs, links, hashtags, and emojis. They then deduplicated in two stages — exact deduplication within files using the dataset's precomputed Locality-Sensitive Hashes, and near-deduplication across the whole corpus using MinHashLSH with 5-gram subsets, 128-permutation MinHash signatures, and a Jaccard threshold of 0.8. Finally they ran Monocleaner, which combines rule-based anomaly detection with a 7-gram KenLM language model trained on high-quality Greek text.

Step 3 — Build four training corpora for controlled comparison. The Legal Corpus (~16.75GB) covers only Greek legal and parliamentary sources. The HQ Repeated Corpus upsamples the highest-quality legal subsets (notably the Raptarchis legal dictionary) by up to 4x, expanding the effective size to 21.12GB. The General-Domain Corpus mixes the 17GB legal corpus with cleaned Wikipedia and OSCAR, totaling 59GB. The Bilingual Legal Corpus is the 60/40 Greek–English legal mix.

Step 4 — Choose tokenizers. Three subword algorithms (BPE, WordPiece, Unigram) were each trained from scratch on the corpus corresponding to the model they serve, then compared on fertility and bytes per token.

Step 5 — Pre-train five architectures. RoBERTa, ELECTRA, ConvBERT, Longformer, and ModernBERT were each trained from scratch on the curated corpora. Longformer and ModernBERT were trained at a 1024-token context window rather than their full native capacities. Data was split 90% training / 10% validation with early stopping. Training ran on AWS: p4d.24xlarge instances (8× NVIDIA A100 GPUs, 40GB VRAM each) for standard models and p5.48xlarge instances (8× NVIDIA H100 GPUs, 80GB VRAM each) for ModernBERT and bilingual variants. The AdamW optimizer with a linear learning rate scheduler and bfloat16 mixed precision was used by default, with one RoBERTa variant trained in full fp32 for comparison.

Step 6 — Evaluate. Three benchmarks were used: Greek-Legal NER (254 Government Gazette issues from 2000–2017, seven entity types in IOB format, split 67.5% train / 17.5% validation / 15% test), the Greek-Legal Code dataset for hierarchical topic classification (documents from 1834–2015, 47 volumes, 389 chapters, 2,285 subjects, with categories having fewer than 15 occurrences removed), and NLI (393,000 machine-translated Greek training pairs, with 2,500 XNLI development and 5,000 XNLI test pairs). Metrics were Accuracy, AUC, and Geometric Mean.

Why This Matters

Impact on research. The paper argues that the Greek NLP landscape is fragmented: general-domain gains (long context) and legal-domain gains (specialized data) have not been unified, and modern architectures have been left untried. By releasing curated corpora alongside a matched suite of models and a statistically tested benchmark, it aims to give the Greek NLP community strong new baselines rather than isolated one-off models. It also provides direct evidence on a methodological question broader than Greek: whether repeating high-quality data beats simply adding more data.

Real-world applications.

• Automated classification and routing of Greek legislation, royal and presidential decrees, regulations, and decisions into hierarchical topics.
• Named entity recognition over Government Gazette text, extracting legislative references, organizations, geopolitical entities, and locations from official publications.
• Cross-lingual legal applications between Greek and English, enabled by the bilingual embedding models built on EU legislation, Europarl, and judicial opinion data.
• Long-document processing of Greek legal and parliamentary text at a 1024-token context window, which exceeds the 512-token limit common in earlier Greek encoders.

Industry relevance. Legal-tech, government, and publishing organizations that process Greek-language documents can use these models as drop-in encoders for retrieval, classification, and entity extraction. The released corpora lower the entry cost for building new Greek models. The tokenizer comparison also offers practical guidance for anyone training Greek models: WordPiece on legal text gave the best fertility/bytes-per-token trade-off in this study, while a large-vocabulary Unigram tokenizer over-segmented Greek words.

Future Directions

• Scaling context further. Both Longformer and ModernBERT were trained only at 1024 tokens despite supporting longer native windows — the paper trains "the first phase" of ModernBERT's pre-training curriculum. Extending to the full 4096-token (Longformer) and 8192-token (ModernBERT) ranges is an open step.
• Revisiting DeBERTa. DeBERTa underperformed and was excluded. The authors hypothesize that different hyperparameters, longer pre-training, or a different tokenizer (it uses Unigram) could make it competitive in the Greek legal domain.
• Extending the bilingual line. The first Greek–English legal embedding models are introduced here; how far cross-lingual transfer can be pushed, and whether the 60/40 Greek–English balance is optimal, remains untested.
• Broadening beyond law and beyond encoders. The paper's proving ground is the legal domain, and the family is entirely encoder-based. Whether the corpus-curation and repetition findings transfer to other specialized Greek domains, or to large generative models such as Meltemi-7B, is left open.

Target Audience

Researchers and engineers building NLP systems for Greek or for other moderately-resourced, morphologically rich languages; legal-tech and government practitioners who need Greek document classification, entity recognition, or cross-lingual retrieval; and NLP methodologists interested in the empirical question of whether curated data quality and targeted repetition matter more than raw corpus size. Readers should have basic familiarity with transformer encoders and standard NLU evaluation metrics to get the most from the benchmark tables.

Authors’ abstract

The advancement of natural language processing for morphologically rich and moderately-resourced languages like Modern Greek has been hindered by architectural stagnation, data scarcity, and limited context processing capabilities, particularly in specialized domains such as law. In this work, we propose the Greek Embedding Models (GEMs), a new family of transformer-based language models, specifically developed to address these limitations through architectural diversity and enhanced data curation. The proposed family of models are trained on several large-scale, meticulously curated corpora, encompassing both comprehensive general-domain datasets and specialized legal collections, addressing the persistent data scarcity that has impeded Greek language modeling advancement. The proposed quality-based corpus curation methodology incorporates extensive preprocessing pipelines, sophisticated deduplication strategies and targeted repetition of high-quality legal sub-corpora to enhance domain adaptation. The GEMs family comprises both established architectures (RoBERTa and Longformer) and advanced models not previously applied to Greek (ELECTRA, ConvBERT, and ModernBERT), providing comprehensive coverage of modern transformer designs. Additionally, we introduce the first bilingual Greek-English embedding models tailored for cross-lingual legal applications. Comprehensive evaluation across three core natural language understanding benchmarks demonstrates that the proposed GEM-RoBERTa and GEM-ConvBERT achieve statistically significant performance improvements over established state-of-the-art models, with accuracy gains of up to 3.6\% while conducted statistical analysis using Friedman Aligned-Ranks and Finner post-hoc tests confirms the superiority of our approach across multiple evaluation metrics.