Can an Agent Debug a Web Corpus? Agentic Auditing of Slovak FineWeb2

AI summary

Overview

Research area: Natural language processing / multilingual web-corpus curation, with LLM agents used as corpus auditors rather than as classifiers.

Technical level: Intermediate. The paper is readable without deep ML background, but familiarity with pretraining corpora, language identification and LLM tool-use helps.

Scope (one sentence): The authors run a tool-using LLM agent over the 26.5M-document Slovak slice of FineWeb2 to discover, measure and turn into cheap executable tests the systematic defects a uniform multilingual pipeline missed.

What This Paper Is About

Multilingual corpus pipelines such as FineWeb2 apply one recipe to more than a thousand languages, so failures specific to a single language go unnoticed; the FineWeb2 authors themselves ask users to review filtering for the languages they care about. Manually auditing a corpus does not scale, and running an LLM as a classifier turns curation into an inference bill without explaining anything. The paper tests a third option: use an LLM once, as a debugger, which explores the corpus through a small read-only, budgeted API and compiles each finding into a cheap executable test that a native speaker validates.

Key Contributions

1. A reusable audit environment. A DuckDB-over-Parquet corpus view with a FineWeb2 adapter, eleven read-only budgeted tools (aggregation, sampling, document inspection, text search, host and crawl statistics, sentence-level language identification, detector measurement and evaluation, and save_finding), all logged to replayable JSONL trajectories.
2. An agentic auditor with a defect-agnostic prompt. The system prompt names no failure mode, no defect taxonomy and no neighbouring language, so discovery remains attributable to the agent; aggregation is free while every document read costs one inspection from a fixed budget.
3. A three-stage workflow that can fail a finding. Discovery by the agent, blinded validation by a native speaker, and model-proposed verdicts that the annotator adopts; deployment runs the resulting YAML tests with no model or GPU. One of the seven findings was rejected by the labels and ships as candidate.
4. A released artefact set. The framework, the Slovak test suite (7 slice guards plus 2 metadata invariants), the validation package and the agent trajectories are released at github.com/NaiveNeuron/agentic-corpus-debugger, with the paper's tables generated from those artefacts by script.

Main Findings

• Seven findings over four phenomena. Three agent runs (197, 180 and 173 of 300 inspections used) recorded 2, 4 and 3 findings; deduplicating selectors leaves seven, which collapse by document overlap into four phenomena: content farms on free-hosting subdomains, Czech or code-switched documents labelled Slovak, keyword-stuffed adult spam, and a slo.* farm.
• The largest defect is a 5.32% content farm. 1,408,437 documents on *.firebaseapp.com and *.web.app across 2,851 hosts. It is nearly absent before 2022, peaks at 38.7% of the 2023-06 crawl and is still present in the last crawl. At 1,011 words on average against a corpus mean of 438, it contributes 12.3% of all running words. All three runs found it independently.
• The farm defeats the pipeline's own filters. Its mean language_score is 0.98 (the LowConf cutoff is 0.8), its mean MinHash cluster size is 1.00 so no farm document reaches the duplication threshold of 100, and all three auxiliary identifiers agree with the label on 87% of them. Even at stricter thresholds (score at least 0.9, cluster at most two, all identifiers agreeing), 1,220,394 of 1,408,437 documents (86.6%) pass — but so do 52.0% of the remaining documents, so the signals cannot single the farm out.
• Blinded human labels support 6 of 7 findings. Judged defect rates run .53–1.00 against .08 in the 25-document uniform control, so three of the four phenomena are supported. The slo_prefix_machine_translated_farm finding came out at .07, indistinguishable from the control, and ships as candidate.
• The compiled tests are extremely cheap. The suite is metadata-only and exact over all 26,470,482 documents, running in 0.7 s of wall-clock time (about 13 s of CPU time across 20 cores) with no GPU or LLM calls. Classifying the same corpus with the same model costs $3.42 per thousand documents measured on 400-character snippets batched twenty per call — $90,456 per pass, or $12,309 at raw token prices without harness overhead, recurring every crawl.
• Static strategies reach every validated phenomenon. The two Czech findings sit entirely inside the LID-Disagree slice (100% overlap); the adult-spam selectors all threshold the corpus's own profanity count, lying inside the n_bad_words >= 3 slice (0.39% of the corpus); uniform sampling reaches the 5.32% farm. The slo.* farm falls outside every slice (12 disagreements in 73,835 documents), but the labels do not support it anyway. The agent contributed the measurement, characterisation and test, not an unreachable slice.
• The best baseline beats the agent per run. At 300 inspections each, agent runs validated 2–3 selectors over 2–3 phenomenon groups at $2.21–$2.84, while baselines validated 2–4; LowConf validated 4 selectors over 3 groups at $1.03, more than any agent run at under half the cost. Pooled, four baselines cover 4 groups with 6 validated selectors ($4.46) and three agent runs cover 3 groups with 6 ($7.50). What the agent adds is exact corpus-wide prevalence for every selector.
• Two metadata invariants, one found by a stronger model. A fourth claude-opus-5 run outside the protocol reported that 10,691,128 documents (40.4%) carry a language_score above 1.0, up to 1.00001 — a numerical convention rather than contamination, but a trap for anyone re-thresholding with FineWeb2's per-language cutoffs. The invariant fails in 11 of 12 FineWeb2 configurations. A second, written by the authors with the framework, checks identifier uniqueness: 13 identifiers occur twice (26 rows), all in CC-MAIN-2024-10.
• Tests transfer to a held-out split. All 9 tests run unchanged on an independent 28,720-document held-out split the agent never touched, with every slice prevalence landing within 0.86–1.06 times its training value.
• Validation was partial. Of 358 documents drawn for annotation, 255 carry a label (agent slices 122/175, LowConf 26/36, LID-Disagree 30/36, HighDup 28/36, Random 24/35, control 25/40), with wide confidence intervals and a single native-speaker coder.

Methodology in Plain English

The corpus is exposed to the agent as two DuckDB relations: corpus_meta, one row per document with no text, which supports exact prevalence over all 26.5M rows in under a second, and corpus, a deterministic 1-in-13 text subsample of 2,034,894 documents for when reading is needed. The columns are the pipeline's own metadata (URL, crawl, date, language, language_score, minhash_cluster_size, profanity count n_bad_words, the GlotLID top_langs distribution, and fastText/langdetect/Lingua predictions added by the redistributor) plus language-neutral derived columns. Nothing in the schema names a defect or a language pair.

The agent acts only through the eleven tools, under a prompt that states the finding schema and the epistemic standard (systematic, reproducibly retrievable, grounded in inspected documents, quantified) and asks for calibrated precision estimates and counter-evidence. It runs through the Claude Code CLI with no shell, file or web access. Every finding must come with a description, a cheap executable selector, a prevalence measured on the whole corpus, and an expected precision.

Validation is blinded: documents are shuffled and shown without provenance, URL or metadata, excluding anything any agent run had inspected, and labelled for language and usefulness. A document counts as a defect if its language label is cs, mix, other, or its usefulness label is junk. A model then tabulates the evidence per candidate group and proposes a verdict the annotator adopts — 11 approve, 8 reject and 5 unsure across 24 decisions — and only an adopted verdict assigns a status.

For comparison, four static document-selection strategies (Random, LowConf at language_score < 0.8 covering 0.49% of the corpus, LID-Disagree covering 3.12%, and HighDup at minhash_cluster_size >= 100 covering 0.23%) each draw the same 300 documents the agent could have inspected and show them to the same model without tools.

Why This Matters

Research impact. The paper reframes corpus auditing from a one-shot artefact into executable, re-runnable regression tests — "pytest for web corpora" — that any future crawl or pipeline revision can be checked against for the cost of a scan over metadata that already exists. It also delivers a candid, quantitative negative result: at a matched inspection budget, a well-chosen static sample validated more selectors than agentic exploration, which disciplines claims about what LLM agents add to data work.

Real-world applications.

• Pretraining-data curation for mid-resource and low-resource languages that receive no dedicated audit, where a uniform recipe quietly admits spam farms.
• Continuous monitoring of corpus releases, using the released drop-list patcher to remove newly detected slices before training.
• Evaluating new or revised crawl pipelines against a fixed, validated suite of per-language tests at negligible compute cost.
• Providing native-speaker communities a low-cost way to contribute language-specific tests rather than re-doing a full manual audit each crawl.

Industry relevance. The cost profile is the headline for practitioners: $2.21–$2.84 per agent audit run and 0.7 s per full-corpus scan, against $12k–$90k per LLM classification pass that must be repeated every crawl. For teams building multilingual models, that is the difference between a recurring line item and a rounding error, and the tests run without a GPU.

Future Directions

• Language and model coverage. The study covers one language, one corpus and three runs of one model; agentic discovery has been shown only on Slovak, and the eleven other FineWeb2 configurations only re-ran Slovak-written tests. Whether discovery generalises is open.
• Recall and stability. Validation measures precision only; no selector's recall is known, and on a single crawl natural variation trips all seven prevalence guards and can produce false alarms. Calibrating guards across crawls and measuring recall are explicit open problems.
• A stronger comparison. The baselines lack the agent's tools as well as its adaptivity, and the profanity-ranked strategy was analysed only as a slice — the model was never run on it. A tool-equipped, budget-matched baseline that includes a profanity-ranked sample would isolate adaptivity from tool access.
• Better validation. One native-speaker coder limits the precision estimates and cannot measure agreement on ambiguous Czech/Slovak text; multi-annotator validation and a finer defect criterion would tighten the numbers.

Target Audience

Researchers and engineers working on pretraining-data curation and multilingual corpora; practitioners who build or consume pipeline-derived datasets such as FineWeb2; NLP researchers interested in LLM agents applied to data work rather than to modelling; and language communities or national infrastructure teams (the work is framed around CLARIN-SK) who want a cheap, repeatable way to audit their own language's slice.

Authors’ abstract

Multilingual pipelines such as FineWeb2 apply one recipe across a thousand languages, leaving the failure modes of individual languages unanticipated. We use an LLM once, as a corpus debugger rather than a classifier: a tool-using agent explores the corpus through a read-only, budgeted API and compiles each finding into an executable test, which a native speaker then validates. On the 26.5M-document Slovak portion of FineWeb2, three runs costing \$2.21-\$2.84 each found seven findings over four phenomena. The largest is a machine-generated content farm of 1,408,437 documents (5.32%) that passed the pipeline's filters, 86.6% of it passing even stricter thresholds. Blinded labels support 6 of the 7 findings (defect rates .53-1.00 against .08 in a random control) and do not support one. The compiled suite scans the full corpus in 0.7 s of wall-clock time without a GPU, against \$12k-\$90k per LLM classification pass. The best static sampling baseline validates more selectors than any agent run, at under half the cost. The agent contributes exact prevalence, a precision estimate checked against sampled documents, an account of why the pipeline missed each slice, and a reusable test. We release the framework, Slovak test suite, validation data and agent trajectories.