Stoicheia: Character-Level Masked Diffusion for Ancient Greek Textual Restoration, Parsing, and Metrical Scansion

AI summary

Overview

• Research area: Natural Language Processing for historical languages, specifically character-level masked-diffusion language modelling applied to Ancient Greek philology (textual restoration, morphosyntactic tagging and dependency parsing, macronization and metrical scansion).
• Technical level: Advanced. The paper assumes familiarity with diffusion language models, masked-diffusion objectives, biaffine dependency parsing, beam search, and paired non-parametric significance testing, alongside classical-philological concepts such as lacunae, scriptio continua, dichrona, and metrical scansion.
• Scope (one sentence): The paper introduces Stoicheia, a 405M-parameter character-level masked-diffusion encoder for Ancient Greek with five independently maskable input planes, pretrained on an openly released ~361M-word corpus and released as eleven checkpoints, and evaluates it with matched random-initialization controls on three tasks: documentary reconstruction, morphosyntactic tagging and parsing, and macronization with metrical scansion.

What This Paper Is About

Existing neural models for Ancient Greek (Ancient Greek BERT, GreBERTa, PhilBERTa, Logion, and the DeepMind systems Pythia, Ithaca, and Aeneas) generally either keep their pretraining corpora secret, tokenize into subwords that cannot represent a single letter or syllable, treat accented and pre-divided edited text as ground truth, or are trained on essentially the whole surviving corpus so that they cannot be used to interrogate a passage they may have memorized.

The paper's goal is a model that meets four stated desiderata at once: open and documented data, character-level resolution, separable editorial layers (letters, boundaries, diacritics, capitalization, punctuation), and guaranteed ignorance of a given passage so that restoration experiments on held-out text are not circular. Stoicheia is the resulting artifact, and the paper quantifies what character-level diffusion pretraining contributes by comparing each fine-tuned model against a matched random-initialization control trained to convergence.

Key Contributions

1. An open, revision-pinned pretraining corpus of ~361M Ancient Greek words across nine sources, in documented quality tiers: a pristine tier of ~90M words released untouched and a repaired tier of ~271M words reconstructed by an instruction-tuned LLM (Qwen3.6-27B) constrained to correct only well-attested OCR confusions and to discard rather than fabricate. A separate bronze tier of 676K passages (~1.4B characters) was machine-translated into Ancient Greek from the Latin holdings of Corpus Corporum.

2. A 405M-parameter character-level masked-diffusion Transformer (d_model=1024, depth 32, QK-norm) whose input factors into five aligned, independently maskable character-level planes: base letter identity over a 24-symbol minimal alphabet, word/sentence boundaries, diacritics (accent × breathing × iota subscript × diaeresis), capitalization, and punctuation category. Attention is banded: three of every four blocks attend within a 256-character window, the fourth attends globally.

3. Eleven released checkpoints: ten rotated, decontaminated 80/10/10 folds (Stoicheia_fold_0 through Stoicheia_fold_9) such that for any given literary passage at least one released model has never seen its text, plus Stoicheia_doc_clean, which excludes every inscription and papyrus record and every passage flagged by a contamination screen as quoting or closely paraphrasing one.

4. Three fine-tuning experiments, each with a matched random-initialization control trained to convergence, isolating the contribution of pretraining: reconstruction of damaged inscriptions and papyri, morphosyntactic tagging and dependency parsing, and macronization with metrical scansion, with external baselines evaluated in the authors' own harness on identical frozen samples.

Main Findings

• Reconstruction: pretraining is worth 5.64 CER points on inscriptions and 3.42 on papyri. Across the ten-fold rotation, replacing the pretrained backbone with the random-init control and changing nothing else in the v1 recipe costs those amounts, with all ten per-fold differences in the same direction in both domains (p=0.00195, the n=10 floor). The v2 recipe reaches 16.10 CER and 73.33 top-1 on inscriptions and 10.01 CER and 81.53 top-1 on papyri.

• Head-to-head on Ithaca's test split, Stoicheia v2 reduces character error from 24.6 (Ithaca) and 23.5 (Aeneas 2025) to 15.5, and raises top-1 from 63.0 and 64.0 to 74.5. All systems read the same frozen file of 3,000 samples (300 at each gap length L=1–10) at beam width 20. That is a 37% relative CER reduction against Ithaca and 34% against Aeneas. McNemar's exact test on discordant pairs: 439 samples recovered by Stoicheia alone against 93 by Ithaca alone (p=9.7×10⁻⁵⁵); 415 against 100 against Aeneas (p=1.3×10⁻⁴⁶). Aeneas's own gain over Ithaca on this protocol is not significant (197 against 166, p=0.12).

• On documents edited after every system's training cutoff, Stoicheia leads the field. On 4,111 gaps in six recently edited inscriptions, Stoicheia v2 reaches 23.45 CER and 64.41 top-1 against Ithaca's 36.50, Llama 3.1 8B's 34.18, and Aeneas's 33.96. On 1,620 gaps in two Uppsala papyri (Llama's released predictions cover 1,614), v1 reaches 30.93 CER against Llama's 35.65. Relative error reductions are 31–36% against the three inscription baselines and 12.4% against Llama on papyri, with paired McNemar p ≤ 3.9×10⁻⁵⁴ on inscriptions and p=1.9×10⁻⁵ on papyri. The random-init control falls behind every external system on both sets.

• Parsing: pretraining is worth 12.9 LAS. Run to 200 epochs, the pretrained arm reaches 83.89 LAS against the random-init arm's 71.00 (p=0.002, the n=10 floor; d_z=25.3). On the standard schedule the random-init arm reaches 68.41 LAS while still improving at roughly +1.1 LAS per 10 epochs, so stopping it there would have credited pretraining with unfinished optimization (+15.4).

• Stoicheia leads all thirteen configurations over the complete 5-fold × 2-seed matrix (10 runs per encoder) on the Celano (2025) treebank (AGDT, Gorman and Pedalion; 1.26M tokens): 83.81 LAS, ahead of learning-rate-tuned PhilBERTa (82.89) by +0.92 LAS (p=0.00195 at n=10, d_z=6.2), and ahead of tuned GreBERTa by +1.29 and mBERT by +16.85. It exceeds every figure published on this treebank: +6.7 LAS, +6.5 UAS, +1.4 XPOS, and +0.5 UPOS over the best tagger–parser of Celano (2025), and +2.6 lemma over the mean reported for GreTa. Lemmatization is the one metric where subword baselines edge it out in the shared recipe (GreBERTa 93.91 and PhilBERTa 93.84 against 93.78).

• Recipe matters as much as backbone. Celano (2025) report GreBERTa at LAS 53.41 under their own architecture; under the authors' recipe the identical checkpoint reaches 82.52 LAS, a swing of roughly 29 points on the same weights.

• Macronization: pretraining is worth +6.0 points of balanced accuracy. Against the matched random-init control over six seeds per arm, pretraining contributes +6.0 balanced accuracy on macronization and +2.2 on scansion (Table 6, n=2,660 positions). Only the macronization effect is significant, at the n=6 floor (p=0.031, d_z=15.9), with all six differences positive; for scansion p=0.0625 with one negative difference. The joint model macronizes about a point worse than the macron-only arm, the price of a shared backbone.

• Stoicheia beats both prior macronizers on Norma's 1,916 test positions, and never abstains. Rule-based macronization abstains on 40% of positions and reaches 89.46 even when every abstention is scored as short (its most favourable treatment); the 0.87M-parameter character transformer abstains on 6 and reaches 90.66; Stoicheia averages 93.37 over six fine-tuning seeds (range 89.61–94.52; dev-selected seed 94.52), 2.8 points above the dedicated model.

• The macronization/scansion asymmetry is the paper's most informative result. Syllable weight is largely computable from visible orthography, so a randomly initialized model can learn most of scansion from the silver corpus alone (+2.2 gap). Dichronon length is not recoverable from local orthographic evidence, so the +6.0-point macronization gap is the price of not having that lexical and morphological knowledge already present.

• An external metre constraint substitutes for internalized knowledge. Adding a Viterbi decoder over metre automata raises exact-line accuracy on a held-out work split from 77.87 to 78.45, positive on all six seeds; on the random-init arm it is worth +1.27, roughly twice as much.

• Dev bits-per-character converged to comparable levels across the fleet. Fold 0 ≈0.355, fold 9 ≈0.365, doc-clean ≈0.234/0.309 on pristine/repaired reference slices, each after roughly 42 hours of training on 128 NVIDIA GH200 GPUs. Checkpoints differ in length (78K–122K steps) because each is the best held-out bits-per-character step rather than the last. The pristine/repaired gap (0.234 against 0.309) indicates the tiers are not interchangeable in quality, and no manual audit of the LLM correction constraint is reported.

Methodology in Plain English

The authors start by rejecting three habits of prior Ancient Greek models: hidden training data, subword tokenization, and treating a modern edited text as if it were the original physical object.

Data. They assemble an openly licensed corpus of roughly 361M words from nine sources, dominated by Internet Archive volumes (~232M words) and PleIAs Greek-PD (~89M), with born-digital editions contributing the rest under their own licenses (Opera Graeca Adnotata ~34M, papyri.info ~4M, the calfa OCR of the Patrologia Graeca ~3M, and ~0.3M from Corpus Corporum, Project Gutenberg, and the SBL New Testament). Rather than reusing the Internet Archive's existing text layer, they re-OCR every Greek page with a vision–language model fine-tuned for polytonic Greek, roughly quadrupling usable yield, then run normalization and dictionary-based correction against an expanded Ancient Greek Hunspell dictionary. A cleanliness score sorts documents into a pristine tier (~90M words, released untouched) and a repaired tier (~271M words, reconstructed by Qwen3.6-27B under strict constraints).

Model. Stoicheia is a masked-diffusion Transformer with one innovation that drives everything else: rather than swallowing raw character strings, it reads five aligned planes, each of which can be independently set to unknown at inference. So the same backbone can be handed unaccented, unspaced, unpunctuated scriptio continua or a fully edited text, and can be told to predict missing accents while keeping letters fixed, or vice versa.

Training objective. Training sequences are corrupted at a rate drawn afresh each time, and the model predicts the true content at every corrupted position under standard 1/t diffusion loss reweighting. The corruption patterns are shaped like real damage: contiguous spans that ignore word edges, whole words, partial words anchored to beginning, middle or end, scattered single characters, and silent substitution of wrong letters with no mask marker. The key pattern is "elastic" masking, where a span of true length L is replaced by M ≥ L mask slots and the target is the L true characters followed by M−L copies of an explicit empty symbol. This teaches variable-length infilling inside a bidirectional encoder. The boundary, diacritic, and punctuation planes are independently set per position to known, unknown, or patchy.

Decontamination. A 13-stage pipeline clusters records into editions of the same work, assigns every cluster to one of ten zones, and excises from each fold's training data every sentence colliding with its held-out zones, matching exact and reordered duplicates, word 5-grams, and document-level near-duplicates. The doc-clean checkpoint extends this to inscriptions and papyri using exact, bag-of-words, and

Authors’ abstract

We introduce Stoicheia, a 405M-parameter character-level masked-diffusion encoder for Ancient Greek whose input factors into five aligned, independently maskable planes: letters, word and sentence boundaries, diacritics, capitalization, and punctuation. A single backbone can therefore restore lacunae, re-segment, accentuate, and punctuate unspaced text without task-specific retokenization. We pretrain it on an open, revision-pinned corpus of 380M words and release eleven checkpoints: ten rotated, decontaminated folds, guaranteeing that for any given literary passage at least one released model has never seen its text, and one with no exposure to documentary texts. Three experiments - reconstruction of damaged inscriptions and papyri, morphosyntactic tagging and dependency parsing, and macronization with metrical scansion - each carry a matched random-initialization control, isolating what character-level diffusion pretraining contributes: 5.6 CER points on inscription reconstruction, 12.9 LAS on parsing, and 6.0 points of balanced accuracy on macronization. On Ithaca's own test split, with identical frozen samples and strict scoring, Stoicheia reduces character error relative to both prior state-of-the-art systems, from 24.6 (Ithaca) and 23.5 (its 2025 Aeneas-framework successor) to 15.5, and raises top-1 accuracy from 63.0 and 64.0 to 74.5.