TINY_SCHILLER: A Drop-In German Drama Corpus for Small Language Models

AI summary

Overview

• Research area: Natural Language Processing — specifically small-language-model (small-LM) data resources, corpus construction, tokenization, and supervised fine-tuning for German literary text.
• Technical level: Beginner-Friendly. The paper's explicit goal is to remove engineering friction so that a practitioner can reach German literary text in one line of code, and its reference training script fits "on a screen."
• Scope: The paper introduces tiny_schiller, a 2.07 MB single-file German drama corpus of eleven public-domain Schiller plays, together with tokenization splits, fine-tuning Parquets, and a documented reference fine-tune on one consumer GPU.

What This Paper Is About

English has tiny_shakespeare, a single 1.1 MB file that lets anyone train a small transformer in minutes on a laptop. German has larger and richer resources — the Deutsches Textarchiv and DraCor — but none of them arrive as a single cleaned file, so practitioners face parser engineering (TEI/XML parsers, namespace handling, edition normalisation, speaker-tag reconciliation, encoding fixes) before a single token reaches a model. The paper names this aggregate burden "single-file friction" and builds tiny_schiller to eliminate it for German literary text.

Key Contributions

1. A cleaned, single-file release. A 2.07 MB UTF-8 file (no BOM) produced by a deterministic pipeline (scripts/parse.py), drop-in for small-LM trainers from scratch (including nanoGPT) and for standard fine-tuning pipelines.
2. Tokenization splits matching standard small-LM workflows. Precomputed character-level, GPT-2 BPE, and cl100k_base token streams, each following a deterministic 90/10 train/validation partition and written as train.bin / val.bin in the nanoGPT convention.
3. Fine-tuning Parquets. A whole-work split, an instruction-formatted dialogue-completion split (instruct.parquet), and 89 per-character persona parquets, all derived deterministically by scripts/build_instruct.py.
4. A reference fine-tune and an agent-ready data card. An end-to-end two-stage SFT run on a single consumer GPU documented with concrete numbers, plus DATA_CARD.md, a Markdown summary structured as a data statement to enable robust corpus ingestion by agentic tooling.

Main Findings

• The corpus itself: tiny_schiller.parsed.txt contains 2,019,857 characters in 2,067,041 bytes of UTF-8 (no BOM), with LF line endings, 88 unique codepoints, 11 drama works, and the canonical speaker-tag format SPEAKER: \n. It is 1.88× tiny_shakespeare in bytes, but fewer tokens under BPE.
• The eleven works: Die Räuber, Die Verschwörung des Fiesco zu Genua, Kabale und Liebe, Don Carlos, Wallensteins Lager, Die Piccolomini, Wallensteins Tod, Maria Stuart, Die Jungfrau von Orleans, Die Braut von Messina (test split), and Wilhelm Tell (test split), spanning Schiller's dramatic career from 1781 to 1804.
• Persona counts per work: Die Räuber 20, Die Verschwörung des Fiesco zu Genua 25, Kabale und Liebe 10, Don Carlos 34, Wallensteins Lager 15, Die Piccolomini 23, Wallensteins Tod 24, Maria Stuart 23, Die Jungfrau von Orleans 39, Die Braut von Messina 11, Wilhelm Tell 51 — giving 89 per-character persona splits in total.
• Tokenizer comparison on the file: schiller_char has vocabulary 88 and yields 1.00 chars/tok; schiller_bpe (GPT-2 BPE) has vocabulary 50,257 and yields 2.36 chars/tok; schiller_cl100k has vocabulary approximately 100k and yields 3.14 chars/tok. The paper uses schiller_bpe as its default for token-based reporting.
• Cross-lingual tokenizer effect quantified: The cl100k_base split is included specifically to quantify context-budget effects, reflecting the known inefficiency of English-trained tokenizers on German documented at scale elsewhere.
• Instruction split size: instruct.parquet comprises 7,454 training rows and 153 evaluation rows (a 2% holdout).
• Persona split size: char_MOOR.parquet comprises 127 training rows and 3 evaluation rows.
• Reference fine-tune results: On Qwen/Qwen2.5-0.5B-Instruct, Stage 1 (full instruct.parquet) reached final eval loss 0.182 and final token accuracy 0.965; Stage 2 (persona specialisation on char_MOOR.parquet) reached final eval loss 0.044 and final token accuracy 0.989. Stage 1 took 3 h 34 min and Stage 2 took 3 min — 3.6 hours total on an NVIDIA RTX 3060 (12 GB VRAM, bf16 precision), with early stopping (patience 2, eval every 100 steps) not triggered.
• Explicit limits: The corpus is too small to pretrain a competitive general-purpose language model and does not constitute a benchmark in the sense of a held-out evaluation suite. Speaker-tag detection is regex-based rather than grammar-based, so edge-case typography — such as stage directions formatted as speaker turns — survives into the released file.

Methodology in Plain English

The author started from DraCor's GerDraCor plain-text export (CC0) rather than from raw TEI/XML, and ran a deterministic parsing script over the concatenated downloads. That script does two things. First, mechanical normalisation: convert CRLF to LF, collapse runs of three or more blank lines to two, strip the narrow no-break space (U+202F) found in some upstream editions, and normalise en-dashes. Second, speaker-tag unification: upstream editions mix three styles (inline Speaker. text, standalone Speaker., and standalone Speaker:), and all are rewritten into the SPEAKER: \n text form used by tiny_shakespeare and nanoGPT.

The single cleaned file is then the parent of every other artifact. A character-level tokenizer and two subword tokenizers (GPT-2 BPE via tiktoken, and cl100k_base) are applied and written out as train.bin / val.bin on a 90/10 partition. A second script builds the fine-tuning data by sliding a window over the speaker-tagged file: the prompt fills one of six non-persona dialogue-continuation templates with the three preceding speaker turns from one work, and the completion is the next speaker's turn formatted as NAME: \n text. Per-character files use the same window with four character-specific persona templates. Wilhelm Tell and Die Braut von Messina are held out as the test partition. Finally, the author demonstrates usability by running a two-stage SFT: first on the full instruction split to teach the dialogue-continuation register, then on a copy specialised to the Karl Moor persona.

Why This Matters

Impact on research. The paper argues that the barrier to German small-LM work is not data scarcity but friction — the attention cost of upstream preprocessing that competes with the modelling task itself. By providing a clean, deterministic, single-file artifact with a data statement, it lowers the entry cost for small-scale language modelling studies on a non-English literary register and connects German work to the same style of fixed small corpus that the BabyLM Challenge established as a first-class research unit. The BPE and cl100k_base splits also serve as a small testbed for the documented cross-lingual inefficiency of English-trained tokenizers.

Real-world applications:

• Rapid prototyping of architectures and tokenization choices in low-compute settings, where the corpus drops in via a single HuggingFace call.
• Stylistic fine-tuning of pretrained German or multilingual models on a homogeneous literary register.
• Per-character persona fine-tuning using the 89 prebuilt per-character splits — the paper's example completes in three minutes.
• Education and research, letting a course session begin with a model instead of a parser.

Industry relevance. The contribution is operational rather than evaluative: it targets the practical cost of getting data into a training loop, and the release includes an agent-ready data card so that automated tooling can ingest the corpus reliably. The reference fine-tune demonstrates that the whole workflow runs on a single consumer GPU (an RTX 3060 with 12 GB VRAM), which is the relevant hardware envelope for small teams, classrooms, and prototyping.

Future Directions

• Beyond Schiller. The same toolchain applies to other public-domain drama with explicit speaker tags, such as Goethe, Lessing, and Kleist, and to German translations of international authors, enabling future tiny_<author> releases for cross-author comparison at the same scale. Minor adaptations extend it to non-dramatic text and to additional languages. The paper also names Goethe and Lessing as reasonable alternative single-author choices.
• Improving speaker-tag handling. Because detection is regex-based rather than grammar-based, edge-case typography such as stage directions formatted as speaker turns survives into the released file — a known limitation left open.
• Formal evaluation. The paper makes no claim about generation quality and states that the corpus does not constitute a benchmark in the sense of a held-out evaluation suite; an evaluation harness would be a natural extension.
• Scale limits. The corpus is explicitly too small to pretrain a competitive general-purpose language model, leaving open what scale would be needed for German single-file corpora to support broader pretraining.

Target Audience

The paper is most useful to practitioners who want to run small language models on German literary text without building a preprocessing pipeline: educators teaching language modelling, students and researchers doing small-scale non-English LM studies, engineers prototyping architectures or tokenizers on limited compute, and anyone doing stylistic or persona-level supervised fine-tuning of small open base models. Researchers working on multilingual tokenization will also find the fertility comparison relevant.

Authors’ abstract

tiny_schiller closes the small-language-model prototyping, fine-tuning, education, and research gap for German literary text, providing a single-file, drop-in counterpart to Karpathy's tiny_shakespeare. The available German literary corpora are larger and richer, but require parser engineering before a single line of training or fine-tuning code can run. tiny_schiller is a 2.07-megabyte single file of eleven public-domain Schiller dramas, sourced from DraCor's GerDraCor export (CC0) and processed by deterministic parser engineering. Character-level, GPT-2 byte-pair encoding, and cl100k_base tokenization splits, an instruction-formatted dialogue-completion split, and 89 per-character persona splits load from a single HuggingFace call. A small language model literally reaches German literary text in one line of code.