Wieszcz-XIX: A 3.1-Billion-Word Corpus of Pre-1918 Polish and Temporally Bounded Language Models Trained From Scratch

AI summary

Overview

Research area: Natural Language Processing — historical corpus construction and from-scratch language model pretraining, with a focus on temporal boundedness (training a model exclusively on text from a bounded historical period).

Technical level: Intermediate. The paper is readable without deep machine-learning background for the corpus-construction and data-audit portions, but the model ladder, tokenizer trade-offs, and bits-per-byte evaluation assume familiarity with language model pretraining concepts.

Scope: The paper presents Wieszcz-XIX, a 6,745,956,943-token (about 3.1 billion word) corpus of Polish published 1800–1918, together with a ladder of three decoder-only language models (47M, 107M, and 349M parameters) trained from scratch on it, and measurements of how well those models respect the period boundary.

What This Paper Is About

Historical Polish is well documented as a language, but machine-readable annotated resources for the period 1800–1918 amount to roughly one million words, far too little to train a from-scratch language model. The rest of the period's text sits behind optical character recognition of variable quality.

The paper's goal is twofold: build a corpus large enough to train a language model that knows only the pre-1918 world, and measure whether such a model actually behaves as temporally bounded when set against modern Polish base models. A model trained only on bounded-period text has no posterior knowledge to suppress — its vocabulary, factual world, and gaps are those of its sources.

Key Contributions

1. A 6,745,956,943-token corpus of Polish from 1800–1918 in 294,369 documents (mostly periodical issues), assembled from Wolne Lektury and the Internet Archive through a released pipeline whose cleaning steps each emit a published report. The corpus is over three orders of magnitude larger than the annotated corpus of the same period.

2. A quantified account of the corpus's defects: OCR corruption measured against a false-positive floor established on clean transcribed text, near-identical duplication measured and removed, post-1918 leakage excluded from the training corpus down to a measured residue of 0.04% to 0.38% of its bytes, and split leakage iterated to zero detectable pairs.

3. A ladder of from-scratch models from 47M to 349M parameters, every rung trained on the same GPU with the same data, steps, and seed, with recipe, cost (about $104 total), telemetry, and epoch-boundary and pre-decay checkpoints at every scale.

4. A measured temporal bound. Scored against two modern Polish base models, one matched in size, the 349M model shows a crossover against both and the 107M against the comparator of its size; the models also keep period spelling where the comparators do so only in part.

Main Findings

• Corpus build: The cleaned build holds 294,369 documents and 6,745,956,943 tokens — 291,655 Internet Archive documents (6,697,876,755 tokens) and 2,714 Wolne Lektury transcriptions (48,080,188 tokens, 0.71% of the corpus by tokens).

• Split: The training split is 291,605 documents and 6,689,593,236 tokens; the held-out split is 2,764 documents and 56,363,707 tokens, 0.84% of the corpus.

• Source concentration: 78.6% of bytes come from the Jagiellonian Digital Library (264,838 documents); 94.2% of Internet Archive documents are periodical issues. Under 1914 borders, 62.0% were published in Austria-Hungary and 22.8% in the Russian Empire.

• OCR corruption: The Internet Archive material shows 1.83% suspicious words against a 0.26% false-positive floor on the clean transcribed source, giving 1.57% true corruption. Per-document rates are unimodal: median 1.59%, ninetieth percentile 3.30%, worst document 20.89%.

• Hand-corrected error rates: On a sample of twenty passages of about 400 characters, eleven could be corrected, with a character error rate of 0.68% (30 edits over 4,434 characters) and a word error rate of 4.71% (31 edits over 658 words). The other nine passages — 45% of the sample — could not be corrected at all because layout collapse removed text.

• Deduplication: MinHash over 5-word shingles found 322 exact-duplicate groups and 411 clusters in total; dropping all but the longest cluster member removed 418 documents, 52,541,412 bytes, or 0.225% of the corpus. That figure is a lower bound.

• Temporal decontamination: 3,733 documents (577,697,456 bytes, 2.47% of the freeze, an estimated 171M tokens) were excluded under overlapping arms. Every strong marker goes to zero after the audit (hitler from 331 documents to 0, copyright from 641 to 0, "digitized by" from 38 to 0).

• Split leakage: Containment testing converged over three iterations, with maximum containment 0.998 → 0.896 → 0.495, zero documents at ≥0.5, and zero candidate pairs in the final state; 180 documents were returned to the training side.

• Tokenizer: A byte-level BPE tokenizer with a vocabulary of 8,000, giving 3.342 bytes per token on held-out text, against 3.689 for Bielik's tokenizer and 4.029 for papuGaPT2's. Larger vocabularies would shorten the stream: 4,000 tokens reach 3.079 bytes per token, 16,000 reach 3.847, and 32,000 reach 4.208 (5.36B tokens instead of 6.75B).

• Model ladder and loss: Held-out cross-entropy is 3.1009 nats per token at 47M, 2.9046 at 107M, and 2.6726 at 349M. In bits per byte, which no vocabulary can move, that is 1.333, 1.249, and 1.149 respectively.

• Temporal crossover: Against two modern Polish base models, one far larger, the 349M shows a crossover, as does the 107M against the comparator of its size. Post-1918 vocabulary costs the historical models about 3.1 bits per byte more than period vocabulary, a gap the comparators do not show, and period vocabulary costs them fewer bits than it costs the comparators.

• Held-out scanning: On held-out text, every rung scores below both comparators on the scanned source.

• Scaling versus repetition: Adding parameters gains about twice as much as a second pass over the data.

• Cost: The ladder trained on one rented RTX PRO 4500 Blackwell (32 GB) at $0.72/h: 18.4 h (about $13) for 47M, 33.7 h (about $24) for 107M, 92.6 h (about $67) for 349M, 144.7 h and about $104 for the ladder.

• Data-constrained position: The 349M model sits at 19.2 unique tokens per parameter, almost exactly the compute-optimal ratio, traversed twice.

• Period spelling: Shown period text, the models keep its spelling and the comparators only partly.

• Content warning: The models reproduce period prejudice, including antisemitic statements.

Methodology in Plain English

The authors assembled the corpus from two deliberately chosen sources with complementary weaknesses. Wolne Lektury supplies transcribed literary works with no OCR noise but modernised orthography; the Internet Archive supplies breadth, mostly periodical issues, as OCR text in original pre-1936 spelling. Two richer sources, Polona and HathiTrust, were rejected because their access is gated and the pipeline could not be re-run by a third party.

Internet Archive material passed four content gates: alphabetic ratio of at least 0.60 (which removed about 10% of the raw pull), stopword rate of at least 0.03, diacritic rate of at least 0.04, and Cyrillic rate below 0.005. Gates are applied retroactively and each rejection is recorded with the gate settings that produced it, so tightened rules can be re-applied.

The crawl used the ISO 639-2 tag pol with a date restriction, returning roughly 358,000 items for 1800–1918, and ran until that query space was exhausted. The freeze of 2026-08-03 holds 298,102 files, 23.35 GB of text, an estimated 6.91 billion tokens at 3.38 bytes per token. Text was cleaned as fetched: boilerplate cut, physical lines dropped when fewer than 55% of words are alphabetic, watermark and URL lines removed, hyphenated line breaks rejoined. Replaying this pass reproduces the cleaned file for 99.9% of documents and removes 16.1% of the bytes.

Near-duplicate removal ran before tokenization using MinHash over 5-word shingles with a Jaccard threshold of 0.8. The temporal audit also ran before tokenization, so flagged documents never enter the training stream — the published corpus is the trained one document for document. Because catalogue dates are unreliable, the audit combined a curated battery of modern-vocabulary markers, publisher's apparatus patterns, and four-digit years inside Polish date phrases, with corroborating signals that never flag alone: a -cja versus -cya spelling share and date-phrase counts. The battery was calibrated against period-legitimate near-anachronisms (telefon appears in 72,719 documents of the raw freeze, aeroplan in 6,391) and hand-read candidate contexts, which removed more terms than they admitted.

A provenance check covered all 3,978 Internet Archive documents outside the Polish digital libraries against the archive's metadata API, excluding 13 lending-collection scans; the metadata date field flagged zero of them. A document-level split of 1% of each source's documents was drawn with a recorded seed, then tested for containment and re-drawn until clean.

OCR quality was measured two ways: four cheap character heuristics run over every document, and a hand-corrected sample of twenty passages drawn with a fixed seed from twelve documents. Period orthography was not counted as error; only misrecognitions were corrected.

Three decoder-only transformer models were then trained from scratch on the same frozen training split for 51,038 steps at 262,144 tokens per step (two epochs), from the same seed, on the same GPU type, so parameter count is the only deliberate variable. Evaluation scores held-out cross-entropy in bits per byte across 8,192 non-overlapping windows, and compares the models against two modern Polish base models — Bielik and papuGaPT2 — on period versus post-1918 vocabulary in matched carriers.

Why This Matters

Impact on research. The paper treats a historical corpus as an artefact with measurable defects rather than a finished product. Its content audit runs before tokenisation instead of after publication, so what is released is what was trained on — a direct response to post-hoc audits that keep finding what corpus builders did not report. The bits-per-byte reporting also lets any model with any vocabulary be set against the ladder's three numbers directly, decoupling model comparison from tokenizer choice.

Real-world applications:

• Historical text generation and completion that stays inside a period's vocabulary and worldview, for use in historical sensemaking, creative writing, or period-voiced drafting.
• Digital humanities and corpus linguistics on nineteenth-century Polish print, where the corpus gives a survey of what was actually printed rather than what a balanced selection committee chose.
• OCR quality triage. Each document carries a released corruption rate keyed to a provenance ledger, so a user can drop documents above 5% (2.1% of Internet Archive bytes) or 3% (13.3%) without the released corpus having pre-filtered them.
• Lookahead-bias-free evaluation. Models trained only on text available before a date can be used where a modern model's knowledge of the future would contaminate the setting.

Industry relevance. The work shows that a temporally bounded model is affordable: the whole three-rung ladder cost about $104 on a single 32 GB consumer-class rented GPU, and the paper reports throughput, micro-batch settings, wall-clock, and cost per rung. That makes the recipe accessible to small teams and independent researchers rather than only large labs. The provenance finding — that four fifths of the corpus comes from a single library — is also a caution for anyone assembling data through an aggregator, since the corpus inherits that institution's acquisition history and digitisation priorities.

Future Directions

• Close the provenance gap. Polona holds the best period press but its OCR endpoint is authentication-gated (HTTP 401), and HathiTrust gates bulk full-text access behind an agreement. Recovering those holdings would broaden a corpus whose periodicals are limited to what the remaining libraries digitised.

• Bound the residue the audit cannot see. Post-1918 text that avoids modern vocabulary and dated self-reference passes the content battery, and two editions of one work — a clean transcription and an OCR'd scan in period orthography — share little token-level overlap and are invisible to literal shingles. The paper flags both as open limits.

• Address the 45% uncorrectable share. Nearly half the hand-sampled passages failed because a page's layout collapsed while each line still passed the filters. Line filtering cannot reach this failure mode, so a layout-aware or page-image-aware stage is an unresolved problem.

• Extend the ladder under the same matched conditions. The corpus's own ceiling on size is roughly 358,000 items for 1800–1918, and the 349M model already sits at the compute-optimal token-to-parameter ratio traversed twice. The paper states that the 349M rung traverses the data twice and that adding parameters gains about twice as much as a second pass, leaving open where the ladder's scaling benefit runs out.

Target Audience

This paper is most useful to researchers building or curating pretraining corpora, especially for historical or low-resource languages, who want a worked example of defect quantification rather than a finished dataset. It also suits NLP practitioners interested in temporally bounded or "time capsule" models, digital humanists working on nineteenth-century Polish print, and engineers who need realistic cost and throughput figures for training small decoder-only models from scratch. Readers should note the content warning: the models reproduce period prejudice, including antisemitic statements.

Authors’ abstract

Historical Polish is well documented as a language but annotated in machine-readable form only to about a million words for the period this paper covers; the rest sits behind optical character recognition of variable quality. We present Wieszcz-XIX, a corpus of 6.75 billion tokens (about 3.1 billion words) in 294,369 documents, most of them periodical issues, of Polish published from 1800 to 1918, assembled from Wolne Lektury and the Internet Archive by a pipeline that filters, deduplicates, audits for post-1918 leakage and splits at the document level. It is over three orders of magnitude larger than the annotated corpus of the same period, and we quantify its defects: recognition corruption against a false-positive floor, near-identical duplication, which is removed, and post-1918 leakage, which is excluded from the training corpus itself down to a known residue of 0.04 to 0.38% of its bytes, found in the transcribed source, so the published corpus is the trained one document for document. On a hand-corrected sample the character error rate is 0.68% where the text is legible, and 45% of the sampled passages cannot be corrected. On it we train a ladder of decoder-only models from 47M to 349M parameters from scratch, and measure their temporal boundedness. Against two modern Polish base models, one far larger, the 349M shows a crossover, as does the 107M against the comparator of its size: post-1918 vocabulary costs them about 3.1 bits per byte more than period vocabulary, a gap the comparators do not show, and period vocabulary costs them fewer bits than it costs the comparators. Shown period text, the models keep its spelling and the comparators only partly. Adding parameters gains about twice as much as a second pass over the data. We release the corpus, code and weights. Content warning: the models reproduce period prejudice, including antisemitic statements.