DuoLens: A Framework for Robust Detection of Machine-Generated Multilingual Text and Code

AI summary

Overview

Research area: Natural Language Processing — machine-generated content detection, spanning multilingual natural-language text and source code, with a focus on efficient encoder-only model design.

Technical level: Intermediate. The paper assumes familiarity with transformer encoders, fine-tuning, AUROC/F1 metrics, and binary classification, but its core argument is straightforward and does not require deep architecture expertise.

Scope: The paper proposes DuoLens, a fine-tuned small-encoder framework for binary classification of human-written versus machine-generated multilingual text and code, and benchmarks it against both traditional encoder baselines and large LLMs.

What This Paper Is About

As LLMs generate increasing amounts of prose and code, reliable detectors are needed for academic integrity, misinformation control, and assessment fairness — but existing detectors tend to be either computationally expensive (LLM-based, agent-ensemble, or perplexity-based methods) or too weak, and most work only well in English or only on Python code. The authors argue that fine-tuned encoder-only Small Language Models (SLMs) can outperform LLMs on this binary classification task while consuming a fraction of the compute. They build balanced multilingual and multi-language code datasets, fine-tune encoder models on them, and propose DuoLens, a dual-encoder fusion model combining CodeBERT and CodeBERTa.

Key Contributions

1. Two balanced detection datasets. A multilingual text dataset of 54,520 samples across eight languages (English, Russian, Arabic, Dutch, German, Spanish, Portuguese, Romanian) with an exact 27,260/27,260 split between human-written and machine-generated samples, curated from MULTITuDE, MultiSocial, M4, and HC3. A source-code dataset covering seven programming languages (Python, Java, JavaScript, C, C#, C++, Go), each with 6,000 human-written and 6,000 machine-generated samples, addressing the Python-heavy bias and limited production-code representation of prior resources.

2. The DuoLens architecture. A dual-encoder detector that fuses complementary representations from CodeBERT (pretrained on natural language and code) and CodeBERTa (pretrained on code only) using pooled vectors, a learned gating mechanism that down-weights redundant features and emphasizes encoder-specific cues, and a single linear classifier trained with class-balanced binary cross-entropy.

3. Comprehensive benchmarking. Evaluation across all seven code languages and eight natural languages, including cross-language scenarios, with baselines including CodeBERT, CodeBERTa, XLM-RoBERTa-base, XLM-RoBERTa-large, mBERT, GPT-4o, and Qwen2.5 Coder 3B.

4. Reproducibility package. Training and evaluation scripts with seeds and configurations, plus a reproducibility checklist.

Main Findings

• LLMs perform poorly on this task. GPT-4o achieved only 0.535 AUROC / 0.414 macro-F1 on code detection and 0.573 / 0.490 on multilingual text detection. Qwen2.5 Coder 3B was worse still at 0.492 / 0.388 on code, despite being domain-specialized for code.

• Fine-tuned encoders reach near-ceiling performance. Across both domains, fine-tuned SLMs hit AUROC of 0.97–0.99 and macro-F1 of 0.89–0.94.

• DuoLens leads both domains. It scored 0.985 AUROC / 0.937 macro-F1 on code detection and 0.975 / 0.911 on multilingual text detection, edging out fine-tuned CodeBERT (0.98 / 0.93) and fine-tuned XLM-RoBERTa-large (0.974 / 0.924).

• Large efficiency gains. Compared with LLM-based approaches, the encoders reduce latency by 8–12× and peak VRAM by 3–5× at 512-token inputs.

• Robustness under distribution shift. Under cross-generator shifts and adversarial transformations — paraphrase and back-translation for text, formatting and identifier renaming for code — performance retains at least 92% of clean AUROC.

• Untrained encoders are already competitive. CodeBERT and CodeBERTa without any fine-tuning achieved 0.953 and 0.948 AUROC respectively, roughly matching or exceeding fine-tuned LLM prompting while requiring far less compute.

• Cross-language transfer is uneven and asymmetric. Java-specific and Spanish-specific checkpoints transferred best across their respective domains. English was consistently the hardest target language (0.59–0.67 accuracy when predicted by other-language checkpoints), while Romanian was among the easiest (0.78–0.90). Code transfer was similarly lopsided, with Python samples being poorly detected by checkpoints trained on other languages (as low as 0.467 from the Go checkpoint).

Methodology in Plain English

The authors first assembled datasets by pooling existing human-written and machine-generated corpora, then subsampled to enforce equal class counts within every language — this prevents the model from learning a shortcut based on label or language frequency. For code samples exceeding BERT's 512-token input limit, they applied chunking.

They then fine-tuned pretrained encoders by attaching a classification head and training it to output 0 for human-written and 1 for machine-generated. For code, the candidates were CodeBERT and CodeBERTa; for multilingual text, XLM-RoBERTa-base, XLM-RoBERTa-large, and mBERT. DuoLens itself works by running an input through both CodeBERT and CodeBERTa, pooling each encoder's output (via the [CLS] token or mean pooling), and passing both into a small fusion head. A learned gate decides how much weight each encoder's representation receives — the intent being that CodeBERT contributes natural-language/code alignment cues while CodeBERTa contributes syntactic and structural regularities. The fused vector feeds a single linear classifier. Prediction probabilities are calibrated with temperature scaling, and checkpoints are selected on a development set.

Baselines were established in two ways: probing (using the pretrained encoders without fine-tuning) and few-shot prompting for the LLM baselines, using six examples (three human/machine pairs) across different languages. Metrics reported were overall accuracy plus per-class and per-language accuracy, AUROC, macro-F1, and throughput in samples per second.

Why This Matters

The paper pushes back against a prevailing trend of building detectors out of ever-larger LLMs or multi-agent pipelines. If a fine-tuned, half-size encoder can match or beat GPT-4o on the same binary decision while using an order of magnitude less compute, the field's default architecture choice deserves reconsideration — particularly for researchers without large GPU budgets. It also extends detection work beyond English text and Python code, which is where most prior datasets cluster.

Real-world applications:

• Academic and assessment integrity. Screening student submissions, written exams, and coding assessments for machine-generated content, including in non-English contexts where current detectors produce false positives.
• Hiring and technical interviews. Flagging AI-generated solutions in take-home coding challenges and live coding platforms across multiple programming languages.
• Publishing and newsroom verification. Triaging multilingual news and social media content for machine-generated origin as part of editorial or moderation pipelines.
• Platform content moderation at scale. Deploying detection on high-volume streams where per-request LLM inference cost and latency are prohibitive.

Industry relevance centers on the deployment profile: 8–12× lower latency and 3–5× lower VRAM means detection can run in CI/CD pipelines, learning management systems, or on-premises infrastructure without sending data to third-party APIs or absorbing per-token costs.

Future Directions

• Broadening language and modality coverage. Extending to more natural languages and programming languages than the current eight and seven, particularly given the uneven cross-language transfer results observed.
• Adding user-facing explanations. Encoder-only models produce a label with no rationale. The authors suggest integrating an LLM for sentence-level classification or explanatory output, possibly in a hybrid or agent-based setup.
• Fixing residual dataset imbalance. The multilingual dataset is balanced on labels but not across languages, which the authors identify as a limitation directly tied to the weaker cross-language results.
• Fairness and bias evaluation. DuoLens inherits the biases of its base models, and the authors state that fairness is not guaranteed. Broader generator coverage is also needed — most baselines were open-weight LLMs, with only GPT-4o representing closed-source generators.
• Stronger adversarial evaluation. The paper reports robustness to paraphrase, back-translation, and code formatting changes, but notes that single-signal detection approaches remain evadable; more aggressive adversarial testing is an open problem.

Target Audience

This paper is most useful to NLP researchers working on machine-generated content detection and model provenance; machine learning engineers who need to deploy detectors under real compute and latency constraints; academic integrity and assessment platform developers; and researchers in multilingual NLP or code generation who need balanced benchmark datasets. Readers interested in the practical trade-off between encoder-only SLMs and LLM-based approaches for narrow classification tasks will also find the empirical comparison valuable.

Authors’ abstract

The prevalence of Large Language Models (LLMs) for generating multilingual text and source code has only increased the imperative for machine-generated content detectors to be accurate and efficient across domains. Current detectors, predominantly utilizing zero-shot methods, such as Fast DetectGPT or GPTZero, either incur high computational cost or lack sufficient accuracy, often with a trade-off between the two, leaving room for further improvement. To address these gaps, we propose the fine-tuning of encoder-only Small Language Models (SLMs), in particular, the pre-trained models of RoBERTA and CodeBERTa using specialized datasets on source code and other natural language to prove that for the task of binary classification, SLMs outperform LLMs by a huge margin whilst using a fraction of compute. Our encoders achieve AUROC $= 0.97$ to $0.99$ and macro-F1 $0.89$ to $0.94$ while reducing latency by $8$-$12\times$ and peak VRAM by $3$-$5\times$ at $512$-token inputs. Under cross-generator shifts and adversarial transformations (paraphrase, back-translation; code formatting/renaming), performance retains $\geq 92%$ of clean AUROC. We release training and evaluation scripts with seeds and configs; a reproducibility checklist is also included.