HLPD: Aligning LLMs to Human Language Preference for Machine-Revised Text Detection

AI summary

Overview

• Research area: Natural Language Processing, specifically machine-generated and machine-revised text detection under black-box conditions.
• Technical level: Advanced. The method relies on preference optimization (DPO-style reward modeling), log-probability curvature metrics, and adversarial prompt generation, and assumes familiarity with LLM scoring models and AUROC evaluation.
• Scope: The paper proposes HLPD, a detection framework that aligns a scoring model to human writing style rather than machine style, and evaluates it against logit-based, supervised, and commercial detectors across single-task, adversarial multi-task, multilingual, and iterative-revision settings.

What This Paper Is About

Detectors work well on text that an LLM generated from scratch, but they break down when a human draft is only lightly polished, rewritten, or expanded by a machine, especially when the machine model is unknown (the black-box setting) and the revision prompts are adversarial. The authors start from the hypothesis that human writing has its own distinctive stylistic patterns, and they build a detector that is trained to prefer human-written text over its machine-revised counterpart instead of imitating machine style. The goal is a detector that stays accurate across many revising models, diverse revision tasks, and advanced LLMs.

Key Contributions

1. A human-alignment detection strategy (HLPD): The paper aligns a scoring model to human language style using a reward-based process called Human Language Preference Optimization (HLPO), improving detection of both fully machine-generated and multi-task machine-revised texts in black-box settings. The trained scoring model is additionally applied in an adaptive attack on GPTZero.
2. A five-dimensional adversarial multi-task evaluation framework: A five-dimensional prompt generator (core revision goal, target style, human-like qualities for evasion, operational constraints, auxiliary user requests) combined with multiple state-of-the-art LLMs, fed through DeepSeek-R1 to produce consolidated instructions, expanding the prompt pool to 750 unique prompts.
3. Demonstrated improvements in accuracy, robustness, and multilinguality: Reported gains across machine revision tasks, multiple languages, and several advanced models, all in the black-box setting.
4. Ablation of the optimization design: The loss function's two innovations, a linear contrastive loss replacing the sigmoid activation and a dynamically adjusted beta, are isolated and shown to improve training performance, particularly when data or training epochs are limited.

Main Findings

• Single-task revision on GPT-series models: HLPD achieves a 15.11% relative improvement in AUROC over ImBD, surpasses the logit-based Fast-DetectGPT by 45.56%, and outperforms the supervised RoBERTa-large by 38.40%. It also achieves average improvements of 9.27% over ReMoDetect and 3.21% over Ghostbuster, and a 10.57% relative improvement over GPTZero.
• Full generation by advanced LLMs: HLPD attains the highest average AUROC, exceeding ImBD by 5.53% and Fast-DetectGPT by 34.14%. On the XSum dataset (Table 2), HLPD records 0.9780 (GPT-o3), 0.9850 (DeepSeek-R1), 0.9453 (Gemini-2.5), and 0.9869 (Grok-3), for an average of 0.9738, versus 0.9228 for ImBD, 0.7499 for GPTZero, and 0.6324 for Fast-DetectGPT.
• Overall single-task detection (Table 1): HLPD's overall average AUROC is 0.9144, above Ghostbuster (0.8860), ReMoDetect (0.8368), GPTZero (0.7944), ImBD (0.7870), RoBERTa-large (0.5304), and Fast-DetectGPT (0.4588).
• Adversarial multi-task revisions: HLPD reaches an overall average AUROC of 0.9463, surpassing ImBD by 13.52%, outperforming GPTZero by 24.54%, and beating ReMoDetect by 6.69%. It is highest for all four evaluated source LLMs (GPT-4o, DeepSeek-R1, Claude-3.5, Gemini-2.5-Pro).
• Diverse revision goals: On XSum revised by six LLMs, HLPD achieves the highest average AUROC across all three revision tasks, a 32.38% improvement over Fast-DetectGPT and a 4.11% relative gain over ImBD.
• Iterative revisions: Evaluated on texts revised one to five times by GPT-4o and Gemini-2.5-Pro, HLPD consistently maintains the highest average AUROC.
• Multilingual detection: On generated Chinese, Portuguese, and Spanish dataset variants, HLPD shows a 10% relative advantage over ImBD and an average improvement of 33% against RoBERTa-large across all language versions.
• Inference efficiency: HLPD averages 0.72 seconds per 1,000 words, matching Fast-DetectGPT's 0.72 and far faster than NPR (111.99), DetectGPT (111.33), and DNA-GPT (35.92), on L20 GPU cards.
• Ablation on strategy: Against an unaligned Fast-DetectGPT baseline, HLPD shows an average AUROC improvement of 56%, and outperforms SFT, RLHF, ORPO, and IPO by average margins of 30%, 26%, 23%, and 23%. The optimized HLPO variant secures a 14.5% relative advantage over ImBD.
• Ablation on loss design: The dynamic beta alone (HLPO⋆) gives a 1% relative accuracy improvement, the linear loss alone (HLPO‡) gives 5%, and combining both (HLPO‡⋆) gives 8%. The same modifications lift the ImBD baseline by 6%.
• Humanization adaptive attack: Using GPT-3.5-Turbo to polish 100 XSum paragraphs and generating 100 candidate perturbations per paragraph, the method reduces detection AUROC by an average of 5% per iteration, totaling 20% after four iterations. On the commercial GPTZero platform this corresponds to a 74% decrease in predicted AI probability. On the ImBD demo, AI probability dropped 44.5% (from 81.8% to 37.3%); on the Ghostbuster demo it dropped 61% (from 87% to 26%).

Methodology in Plain English

The approach flips the logic of prior style-based detectors. ImBD trains a scoring model to recognize machine style; HLPD instead trains it to prefer human style.

First, the authors build a dataset of paired texts with identical content: one written by a human and one revised by a machine. Using these pairs, they define a preference relation where the human text is preferred over the machine-revised text, and model that preference with the Bradley-Terry formulation used in DPO. The reward for a text is expressed as the difference between its log probability under the model being trained and under a fixed reference model, scaled by a temperature parameter beta.

Two modifications are made to this standard setup. The sigmoid function is removed, giving a linear contrastive loss that directly enlarges the gap between the log probability of the human text and the machine-revised text, which the authors say avoids saturation and overfitting in small-sample preference tuning. Then beta is not fixed: a lightweight variance-aware scheduler computes the variance of the training margin over a sliding window and increases beta when the model is confident (low variance) and decreases it when the signal is noisy (high variance). Training uses the fine-tuned GPT-Neo-2.7B model as the scoring model, with hyperparameters matching ImBD.

At detection time, the trained scoring model is used to compute Human Language Preference Conditional Probability Curvature (HLP-CPC), adapted from Fast-DetectGPT. A passage is perturbed by sampling from the model's conditional distribution, and the log probability of the original passage is compared to the mean and standard deviation of log probabilities of the perturbed versions. Because the scoring model has been aligned to favor human writing, human text sits in the positive-curvature region and machine-revised text in the negative-curvature region, the reverse of the Fast-DetectGPT situation. A threshold on the negated curvature decides whether a text is flagged as machine-revised.

Evaluation uses single-task datasets covering Rewriting, Expand, Polish, and Generate, plus an adversarial multi-task dataset. Human-written source text comes from XSum, SQuAD, WritingPrompts, PubMedQA, and WikiText. Baselines include five training-based detectors (ReMoDetect, Ghostbuster, RoBERTa-base, RoBERTa-large, GPTZero) and nine logit-based detectors (ImBD, Likelihood, LogRank, Entropy, LRR, NPR, DNA-GPT, DetectGPT, Fast-DetectGPT). Key experiments were repeated five times with different random seeds (42, 199, 410, 2231, 2533), with 95% confidence intervals reported.

Why This Matters

The paper targets the gap between watertight detection of fully generated text and the messier reality of human drafts that machines have touched up. Its framing of the problem in terms of human style rather than machine style is the distinctive move: it argues that because each LLM has its own stylistic fingerprint, imitating one machine's style generalizes poorly, whereas human writing patterns offer a more stable anchor.

Impact on research. The work reframes style alignment for detection: instead of learning machine artifacts, the detector learns human ones. The ablation shows the alignment paradigm matters more than the specific optimizer, with large margins over SFT, RLHF, ORPO, and IPO. The adversarial multi-task framework and the 750-prompt generator also give the field a harder benchmark for revision detection.

Real-world applications (as motivated by the paper):

• Detecting disinformation and misleading content produced by trustworthy-looking LLM output, a risk the authors cite explicitly.
• Protecting high-stakes domains such as legal and medical writing, where the authors argue every word must be deliberately chosen and fully understood, and where even minor AI revisions can introduce subtle errors or vulnerabilities.
• Content provenance and trust decisions for platforms facing LLM-polished submissions.
• Adversarial robustness testing of commercial detectors, as shown by the humanization attack that reduced GPTZero's predicted AI probability by 74%.

Industry relevance. HLPD runs at 0.72 seconds per 1,000 words, matching Fast-DetectGPT and far below several baselines, and works in the black-box setting where the source model is unknown, which is the practical condition for real deployments. The adaptive attack results are also a warning to vendors: a model aligned to human style can be repurposed to help machine text evade detectors, including commercial ones.

Future Directions

• Generalization beyond the tested models and domains: The authors state it remains unclear how the approach generalizes to models and domains not covered in their experiments; closed-source and non-English-specialized models are obvious next targets.
• Short-text detection: The paper notes performance may degrade on very short sentences, where limited context constrains reliable style- and coherence-based detection.
• A two-sided arms race: The same human-aligned scoring model that improves detection also powers an adaptive attack (reducing GPTZero AI probability by 74% over four iterations), raising the question of how detectors and humanization methods co-evolve.
• Semantic fidelity in humanization: The authors caution that repeating the perturbation-and-selection process can introduce more substantial semantic deviations from the original text, so controlling meaning drift is an open problem.

Target Audience

This paper suits researchers and graduate students working on LLM-generated text detection, watermarking, and provenance; practitioners building content moderation or integrity systems who need black-box detection that survives machine revision; security researchers interested in adversarial evasion of detectors; and evaluators who need a benchmark for multi-task adversarial revision. Readers without background in preference optimization or log-probability scoring will find the methodology sections demanding, though the experimental results and the adaptive attack section are readable without it.

Authors’ abstract

To prevent misinformation and social issues arising from trustworthy-looking content generated by LLMs, it is crucial to develop efficient and reliable methods for identifying the source of texts. Previous approaches have demonstrated exceptional performance in detecting texts fully generated by LLMs. However, these methods struggle when confronting more advanced LLM output or text with adversarial multi-task machine revision, especially in the black-box setting, where the generating model is unknown. To address this challenge, grounded in the hypothesis that human writing possesses distinctive stylistic patterns, we propose Human Language Preference Detection (HLPD). HLPD employs a reward-based alignment process, Human Language Preference Optimization (HLPO), to shift the scoring model's token distribution toward human-like writing, making the model more sensitive to human writing, therefore enhancing the identification of machine-revised text. We test HLPD in an adversarial multi-task evaluation framework that leverages a five-dimensional prompt generator and multiple advanced LLMs to create diverse revision scenarios. When detecting texts revised by GPT-series models, HLPD achieves a 15.11% relative improvement in AUROC over ImBD, surpassing Fast-DetectGPT by 45.56%. When evaluated on texts generated by advanced LLMs, HLPD achieves the highest average AUROC, exceeding ImBD by 5.53% and Fast-DetectGPT by 34.14%. Code will be made available at https://github.com/dfq2021/HLPD.