Qwerty AI: Explainable Automated Age Rating and Content Safety Assessment for Russian-Language Screenplays

AI summary

Overview

Research area: Natural Language Processing, specifically document-level content moderation and legal compliance classification for Russian-language narrative text.

Technical level: Intermediate. The paper assumes familiarity with transformer language models, parameter-efficient fine-tuning (LoRA), and quantization, but the system design and evaluation are described in accessible terms.

Scope: An end-to-end, locally-deployed system that segments Russian screenplays, detects five categories of content violations under Russian Federal Law No. 436-FZ, assigns an age rating (0+, 6+, 12+, 16+, 18+), and produces human-readable justifications.

What This Paper Is About

Russian media producers must assign an age rating to every screenplay under Federal Law No. 436-FZ, based on the presence of violence, sexual content, profanity, substance use, and frightening elements. Manual expert review is slow, subjective, and expensive — the paper cites 4-8 hours for a 120-page script. Qwerty AI automates this by combining a fine-tuned compact language model with deterministic legal rule-matching, producing not just a rating but an explanation and editing recommendations for each flagged scene.

Key Contributions

1. Domain adaptation of a compact LLM. The first application of an instruction-tuned ~3.8B-parameter model (Phi-3-mini) to screenplay age-rating with categories tied to statutory text, fine-tuned on 4,200 labeled scenes.

2. A hybrid explainability framework. Neural predictions are paired with deterministic "legal anchors" — regex lexicon matches mapped to specific articles of 436-FZ — so every flag cites both a model judgement and a statutory pattern. This combination raised accuracy from 76.7% to 80.0% over the neural-only variant.

3. Efficiency analysis under hard production constraints. The paper documents how 4-bit NormalFloat quantization, LoRA adapters, dynamic batching, and a contrastive consistency loss let a 700-page, ~350,000-token script be processed in 116 seconds on a 2×A100 40GB node, with no measured accuracy loss from quantization (80.0% with and without).

4. An open evaluation and baseline set for Russian content safety. Segmentation metrics, per-category F1 scores, a document-level confusion matrix, bootstrap confidence intervals, five baselines, and a human evaluation of explanation quality by five professional script editors.

Main Findings

• Rating accuracy: 80% exact match (12/15 scripts), 93.3% within ±1 rating level (14/15), and a mean absolute error of 0.27 rating levels. Bootstrap resampling (10,000 iterations) gives a 95% confidence interval of [66.7%, 93.3%] for accuracy and [0.13, 0.47] for MAE.

• All 3 misclassifications were one level off: a 12+ rated 6+ (metaphorical violence missed), a 16+ rated 12+ (historical alcohol reference), and an 18+ rated 16+ (sexual content read as implied rather than explicit).

• Segmentation is format-dependent: F1 80.0% on general .txt files (precision 79.8%, recall 80.3%) versus F1 94.9% on formatted .docx (precision 94.7%, recall 95.1%), measured against 237 manually annotated scenes in 5 test scripts.

• Per-category detection (F1): violence 0.849, frightening content 0.849, profanity 0.821, sexual content 0.789, drugs/alcohol 0.773; macro average 0.816. Profanity recall is lowest because lexicon anchors miss euphemisms and metaphor.

• Fine-tuning is the dominant factor: the base Phi-3-mini in zero-shot scores 53.3% accuracy (MAE 0.80), while the fine-tuned model reaches 80.0% (MAE 0.27) — an 18 percentage-point gain. A rule-based lexicon alone reaches 46.7% (MAE 0.93), and a random classifier 20.0% (MAE 1.82).

• Ablations: removing rule anchors drops accuracy to 76.7%; removing the contrastive loss drops it to 77.3%; removing fine-tuning drops it to 62.0%; replacing 4-bit quantization with FP16 leaves accuracy unchanged at 80.0%.

• Speed: 8 seconds for a 30-page script, 34 seconds for 120 pages, 88 seconds for 350 pages, and 116 seconds for 700 pages — roughly 0.29 seconds per scene, against a 120-second target. Per-scene inference is 0.15-0.3 seconds with an 8-12 second cold start.

• Explainability: five professional script editors rated explanations at a mean of 3.8/5 (median 4, standard deviation 0.9). Strengths noted were legal anchors and specific text quotes; weaknesses were occasional overexplaining and missing context for ambiguous cases.

• Case study: an 87-page blind drama script was processed in 41 seconds, rated 16+ against a 16+ ground truth, with 7 flagged scenes covering alcohol, violence, and profanity.

• Against humans: expert review takes 4-8 hours for a 120-page script with 85% inter-rater agreement on rating and 70% on scene flagging; the system ran in 34 seconds with 80% agreement with the gold standard, a speed-up the authors put at 700×.

• Known failure modes: metaphorical language triggers false positives (for example "crushed by defeat," "killed his dreams"), while pragmatic implications are missed — a scene implying a sexual advance was rated 0+ against a 16+ ground truth. Science fiction terms like "disintegrate" are flagged as violence even when aimed at robots.

• Class imbalance: 18+ content makes up only 4.8% of the 4,200 training scenes (0+ is 42.9%, 6+ 19.0%, 12+ 21.4%, 16+ 11.9%), contributing to lower recall on extreme cases.

Methodology in Plain English

The pipeline is a chain of stages. A parser reads .docx, .pdf, or .txt files and detects which of 7 encodings applies (UTF-8, UTF-16, CP1251, KOI8-R, ISO-8859-5, MacRoman, ASCII), prioritizing byte-order-mark detection so that Cyrillic BOM files are not mistaken for UTF-8. A segmentation step then splits the script into scenes using regex patterns and capitalization heuristics, keying on scene headings such as INT./EXT.

Each scene is passed to Phi-3-mini-4k-instruct (3.8B parameters, 32 layers, 32 attention heads, hidden size 3072, 32,000-token SentencePiece vocabulary, 4096-token context). The model was fine-tuned with LoRA adapters (rank 16, alpha 32, targeting q_proj, v_proj, k_proj, o_proj) and loaded in 4-bit NormalFloat quantization, cutting memory from 8GB to 2.2GB per instance. Training used a single A100 40GB, learning rate 2e-4 with 100 warmup steps, AdamW, batch size 4 with gradient accumulation to an effective 32, and 3 epochs with early stopping, for 6 hours 42 minutes total; the best checkpoint was epoch 2 step 1,890 with validation F1 0.823.

The system prompt embeds the statutory rating definitions directly and requires JSON-only output naming one of five categories, which the authors say reduces hallucination. The loss combines cross-entropy with a contrastive term (weight 0.1) that penalizes large embedding shifts between consecutive scenes, encouraging consistent ratings within a document. In parallel, regex lexicons match statutory keywords (for example kill|death|dying for violence, alcohol|vodka|drugs for substances), and these matches are surfaced as "legal anchors" in the explanation.

Document rating uses a maximum-severity rule: the script receives the highest rating assigned to any scene. Category percentages feed an IMDb-style Parents Guide view. Inference uses dynamic batching of 4-8 scenes, explicit CUDA cache clearing, FP16 mixed precision, and sliding-window truncation with majority-vote aggregation for scenes over the context limit. The whole stack runs on Yandex Cloud (g2.2xlarge, 2×A100 40GB) with FastAPI and SQLite, exposing a fast /chat endpoint for snippets under 500 tokens and a /upload endpoint for full documents.

Why This Matters

Research impact. Most content moderation research targets short English social media posts. This paper extends the problem to long-form narrative documents in a morphologically rich language, and shows that a domain-specific dataset of 4,200 scenes is enough to lift a 3.8B model 18 percentage points over its zero-shot baseline. The hybrid neural-plus-rule design is a transferable pattern for any NLP task where outputs must be legally defensible.

Real-world applications.

• Pre-production rating estimation, letting studios check compliance before a script is locked.
• Script editing guidance, where flagged scenes come with concrete rephrasing suggestions.
• Compliance documentation, generating audit-ready reports for classification bodies.
• Editorial triage, letting human reviewers focus on the 12 problematic sentences in a 394-sentence script instead of reading everything.

Industry relevance. The system was built for the Wink streaming hackathon under constraints that mirror real deployment: no external API calls, an 80GB VRAM ceiling, and a 5-minute processing window. The paper also reports a privacy feature relevant to commercially sensitive scripts — strict local-only processing with automatic deletion after 24 hours unless explicitly saved. The authors are explicit that the tool is decision support for editors, not an autonomous gatekeeper, and they discuss risks around censorship, algorithmic bias, accountability, and the disproportionate impact of content moderation on marginalized creators.

Future Directions

1. Larger context windows. Moving to 8k-16k contexts would let the model see entire scenes without the sliding-window truncation and majority-vote workaround currently used for scenes over 2000 tokens.

2. Pragmatic and narrative understanding. Both documented error types — metaphorical violence flagged as real, and implied sexual content missed — point to a need for discourse-level and character-relationship modeling rather than sentence-level lexical cues. The authors suggest semantic role labeling as one partial fix.

3. Balancing the training data. With 18+ scenes at just 4.8% of the corpus, expanding mature-rated content is the stated path to better recall on extreme cases; oversampling and class weighting showed diminishing returns.

4. Multilingual and workflow extension. The paper proposes fine-tuning mBERT or XLM-R for English and Spanish, adding character-level and act-structure analysis, and integrating with screenwriting tools such as Final Draft and Celtx for real-time feedback.

Target Audience

This paper is most useful to NLP researchers working on long-document classification and legally-grounded explainability; media industry engineers and compliance teams evaluating automated age-rating; practitioners interested in running compact fine-tuned LLMs under tight memory and latency budgets; and researchers in Russian-language NLP, for whom the paper states it provides the first open evaluation framework for age-rating Russian narrative text.

Authors’ abstract

We present Qwerty AI, an end-to-end system for automated age-rating and content-safety assessment of Russian-language screenplays according to Federal Law No. 436-FZ. The system processes full-length scripts (up to 700 pages in under 2 minutes), segments them into narrative units, detects content violations across five categories (violence, sexual content, profanity, substances, frightening elements), and assigns age ratings (0+, 6+, 12+, 16+, 18+) with explainable justifications. Our implementation leverages a fine-tuned Phi-3-mini model with 4-bit quantization, achieving 80% rating accuracy and 80-95% segmentation precision (format-dependent). The system was developed under strict constraints: no external API calls, 80GB VRAM limit, and <5 minute processing time for average scripts. Deployed on Yandex Cloud with CUDA acceleration, Qwerty AI demonstrates practical applicability for production workflows. We achieved these results during the Wink hackathon (November 2025), where our solution addressed real editorial challenges in the Russian media industry.