Unchecked or Falsely Accused? Evaluating AI-Generated Text Detectors on Kazakh

AI summary

Overview

Research area: Natural Language Processing — AI-generated text detection for low-resource languages, specifically Kazakh.

Technical level: Intermediate. The paper uses standard detection methods (zero-shot scorers, fine-tuned multilingual encoders, off-the-shelf classifiers) but assumes familiarity with AUROC, false-positive rate (FPR), true-positive rate (TPR), and threshold calibration.

Scope (one sentence): The paper builds a Kazakh benchmark of pre-ChatGPT human text and machine text from 9 generators, measures how five public English-trained detectors and several zero-shot and fine-tuned detectors fail on it, and releases a fixed detector called AINA-KK-Det.

What This Paper Is About

Institutions in Kazakhstan already run AI-text detectors on Kazakh writing, but no prior study checks how those tools behave on Kazakh text whose human authorship predates ChatGPT. The paper asks two questions: do detectors wrongly accuse honest Kazakh writers ("falsely accused"), and do they let machine-written Kazakh through ("unchecked")? To answer them, the author assembles AINA-KK — 3,811 provably pre-ChatGPT human passages and 5,597 machine and gray-zone passages from 9 generators — evaluates a wide range of detectors on it, and then builds and releases a better-calibrated detector, AINA-KK-Det.

Key Contributions

1. AINA-KK benchmark: 3,811 human passages from four domains (encyclopedic, news, web, educational) whose existence before ChatGPT's release (30 Nov 2022) is provable, plus 5,597 machine and gray-zone passages from 9 generators — including the Kazakh-built KazLLM-1.0-8B and Sherkala-8B-Chat — with script (Latin-2021 transliteration, homoglyphs) and length (100-word, 50-word truncation) stress tests.

2. A dual-failure evaluation: Measurement of zero-shot detectors across four scoring-model families, fine-tuned multilingual encoders (XLM-R-base, mDeBERTa-v3-base), and five widely downloaded English-trained public detectors, with false-positive rates broken out by domain and length.

3. Fixes requiring no new data collection: Threshold calibration on held-out human text, script canonicalization, script-augmented and length-matched training, length-specific thresholds, and an OR-combination of a fine-tuned classifier with a zero-shot scorer — released together as AINA-KK-Det.

4. Released artifacts: Data (Hugging Face dataset ajnasiou/aina-kk), code (github.com/ozxmn/aina-kk), and detector (ajnasiou/aina-kk-det), with calibration data and the detector's own measured error rates.

Main Findings

• Public English-trained detectors fail in both directions. TMR RoBERTa flags 81.4% of human passages and ranks human text above machine text (AUROC 0.292). Fakespot flags 25.5% of human passages, 33.3% of student reports and essays, and 87.6% of 50-word excerpts. Desklib DeBERTa flags 79.6% of essays. At a threshold recalibrated to 1% FPR on Kazakh development text, none catches more than 1.3% of machine passages.

• False accusations dominate in realistic conditions. Assuming 5% of submissions are machine-written, 93% of Fakespot's flags and 97% of TMR's would fall on honest writers.

• Zero-shot detectors leave machine text unchecked. The best, Binoculars with Gemma-4-E2B (AUROC 0.931), catches 50.3% of in-distribution machine passages at a calibrated threshold; no other of the 44 stored statistics catches more than 23.7%. With Qwen3-4B as scoring model, Binoculars and Fast-DetectGPT are at chance (AUROC 0.509 and 0.475).

• The scoring model matters more than the statistic. Kazakh continued pretraining of mGPT raises Fast-DetectGPT from AUROC 0.808 to 0.849, but this still falls short of the larger multilingual scorer.

• Fine-tuned encoders rank near-perfectly but mis-set the threshold. mDeBERTa reaches AUROC 0.998 (0.984 on unseen generators), yet at the default 0.5 threshold flags 7.6% of human test passages and 10.3% of web text; XLM-R flags 9.9%.

• Short human text is flagged most. Cut to 50 words, mDeBERTa flags 24.4% of human excerpts at the default threshold against 7.6% at full length. Machine outputs are shorter than human passages (median 153 vs 175 words; 18.5% vs 0.5% under 120 words), so brevity becomes evidence of machine authorship.

• Kazakh-built generators evade. Calibrated mDeBERTa catches at least 96.8% of every global generator's passages but only 47.1% of KazLLM (40 of 85) and 62.7% of Sherkala (47 of 75). Zero-shot Binoculars catches 50.6% of KazLLM and 74.7% of Sherkala, and 43.8% of the 73 Kazakh-built passages mDeBERTa misses — the two detector types err on different texts.

• Latin-2021 transliteration breaks every calibrated threshold. mDeBERTa's ranking inverts (AUROC 0.384), as does Gemma Binoculars' (0.346); XLM-R survives partly (0.875). 17 of 20 detectors catch at most 0.5% of Latin machine text. Replacing 10% of Cyrillic letters with Latin look-alikes cuts mDeBERTa's TPR from 99.0% to 63.3%.

• Gray zones have no stable answer. At the calibrated threshold mDeBERTa flags 82.5% of passages opening with ~50 human words then continued by a model, 7.7% of AI-polished human text, 7.9% of machine translations from Russian, and 100.0% of prompt-based humanizer rewrites.

• AINA-KK-Det results. On test it flags 0.9% of human passages (95% CI 0.4–2.7; 1.5% on development data) and catches 85.2% of machine text from known generators, 92.9% from two unseen Gemma-4 models, 61.3% from the Kazakh-built models (95% CI 53.2–68.8), and 85.2% of Latin-script machine text. At 5% prevalence, 82.9% of its flags would be correct (62% at the upper end of its FPR interval), against 6.9% of Fakespot's.

• Training on Kazakh-built generators is not a dependable fix. Adding KazLLM and Sherkala to training lifts their detection to 76.5% and 80.0%, but the same model flags 82.2% of 50-word human excerpts and 7.9% of essays at the default threshold; with length matching and length-specific thresholds the operating point breaks down (in-distribution TPR 64.8%, KazLLM 41.2%, Sherkala 44.0%).

• Part of the separability is genre, not authorship. A logistic regression on 23 surface statistics reaches AUROC 0.979, because human passages are spans of published pages while machine passages are complete compositions. 86% of human test passages contain a digit against 28% of in-distribution machine passages.

Methodology in Plain English

Building the human half. The author collected Kazakh text that provably existed before ChatGPT launched: a May 2022 Kazakh Wikipedia dump, news pages from 27 outlets captured by Common Crawl before the cutoff, CC100-kk web text from 2018 snapshots, and student reports and essays from 68 educational websites. Passages were kept only if a language-identification tool labeled them Kazakh-in-Cyrillic with confidence ≥ 0.9, near-duplicates were removed, and personal data (emails, phone numbers, ID numbers) was scrubbed. Passages were clustered by topic and whole clusters assigned to train or test, so no topic appears in both.

Building the machine half. Nine generators produced text under six prompt types: writing from a topic, continuing a human opening, polishing human text, translating Russian web text (by an LLM or by NLLB-200), and rewriting machine text under a "humanizing" prompt. Four generators — KazLLM, Sherkala, and two Gemma-4 models — were held out entirely from training.

Stress tests. Test passages were transliterated into the 2021 Latin alphabet, attacked with Latin look-alike letters, and truncated to 100 and 50 words.

Validity audit. A native-speaker author judged 100 items before and 100 after filtering. Only 33 of 52 sampled human passages were valid in the post-freeze sample (66% weighted by domain size; educational 6 of 13), which the paper reports openly.

Detection and calibration. Zero-shot detectors pair a scoring model with a sibling sharing its tokenizer; fine-tuned encoders train on four in-distribution generators for 2 epochs over 3 seeds; five public detectors run as released at threshold 0.5. Thresholds are set to 1% FPR on 391 development human passages from 7 withheld topic clusters and applied unchanged to test, with bootstrap confidence intervals that re-fit the threshold.

Fixes. Calibrating on held-out human text, mapping Latin-2021 back to Cyrillic before scoring, augmenting training with transliterated and homoglyph text, matching training lengths to the machine-output distribution, fitting thresholds per input length, and flagging when either a classifier or Binoculars exceeds its own threshold.

Why This Matters

Impact on research. Multilingual detection benchmarks cover up to 22 languages, but Kazakh appears in none of their training data — only in the COLING 2025 test set (2,471 samples) and as human fact-check text in BLUFF. This paper is the first to evaluate downloadable public detectors on Kazakh text with dated human provenance, and it identifies a failure mode (short human text flagged because of a length imbalance) that is the converse of the known short-machine-text weakness.

Real-world applications:

• Universities in Kazakhstan currently face a vendor marketing Kazakh detection with false positives "reduced to 1%"; this paper gives institutions a measured alternative and a way to check such claims.
• Academic-integrity workflows can adopt length-specific thresholds and published FPR intervals instead of default 0.5 cutoffs.
• Government and media monitoring of machine-generated Kazakh content, where ChatGPT and Gemini account for about 94% of AI-chatbot traffic in the country.
• Any low-resource language planning a script transition (Kazakh is officially moving from Cyrillic to a Latin-based alphabet) can reuse the canonicalization and script-augmentation recipe.

Industry relevance. The results give detector vendors a concrete checklist — calibrate on local human writing, canonicalize script, fit thresholds by length, report the error rate for the input's length — and show that English-trained checkpoints carry no usable signal on Kazakh at a 1% operating point.

Future Directions

• Collect genuine student writing. The educational class is public web text as found, not essays collected under supervised conditions; the author notes that supervised L1/L2 writing sessions and a native-speaker detection study require ethics approval.
• Test against commercial detectors. Turnitin, GPTZero, Copyleaks, Originality.ai, Pangram and the local vendor are not evaluated, because they offer no free API and responsible evaluation means notifying vendors first.
• Disentangle "Kazakh-built" from size and decoding. KazLLM and Sherkala are both 8B Llama-3.1 derivatives decoded locally with a repetition penalty, so being Kazakh-built is confounded with model size, base model and decoding.
• Establish the OR-combination gain. The classifier-plus-Binoculars improvement on Kazakh-built text is consistent but not statistically established (exact McNemar p = 0.06), and it does not recur with the other two classifiers (p = 0.38 and 1.00).
• Move beyond machine transliteration. The Latin test uses text with an exact inverse transliterator, not Latin text written by people; informal spellings and the 2017 and 2018 alphabets are untested.

Target Audience

Researchers in multilingual NLP and AI-text detection; academic-integrity officers and university administrators in Kazakhstan and other low-resource-language regions; developers building or procuring detection tools; and anyone working on script transitions or language-specific benchmarking, who will find the calibration and canonicalization recipe directly transferable.

Authors’ abstract

AI-text detectors are already applied to Kazakh writing, yet no study reports how often publicly available detectors flag Kazakh text whose human authorship is dated before ChatGPT. We present AINA-KK: 3,811 human passages from encyclopedic, news, web and educational pages that provably existed before ChatGPT, and 5,597 machine and gray-zone passages from 9 generators, including the Kazakh-built KazLLM and Sherkala, with script and length stress tests. Five English-trained detectors from the Hugging Face Hub flag up to 81.4% of human passages and 79.6% of published student essays; recalibrated, they catch at most 1.3% of machine text. Calibrated zero-shot detectors catch at most 50.3% of machine text from in-distribution generators. Fine-tuned encoders rank text almost perfectly, but mDeBERTa at its default threshold flags 24.4% of 50-word human excerpts, partly because machine outputs are shorter than human text; calibrated, it catches only 47.1% of KazLLM and 62.7% of Sherkala passages. Without script handling, machine-transliterated Latin-2021 text defeats every detector. Fixes that need no new data (calibration, script canonicalization, script-augmented and length-matched training, and an OR with a zero-shot scorer) form AINA-KK-Det. On test it flags 0.9% of human passages (95% CI 0.4-2.7; 1.5% on development data) and catches 85.2% of machine text from known generators, 92.9% from two unseen Gemini-family Gemma-4 models and 61.3% from the Kazakh-built ones; its configuration was chosen with test results in view. We release the data, code and detector at https://github.com/ozxmn/aina-kk.