Towards Detecting AI-Assisted Responses in Online Surveys

AI summary

Overview

Research area: Natural Language Processing / AI-generated text detection, applied to social science survey methodology.

Technical level: Intermediate. The paper assumes familiarity with detection metrics such as AUROC and with transformer-based text classifiers, but its core method is deliberately simple and training-free.

Scope: The paper builds a benchmark dataset (ASURRE) of AI-assisted survey responses across three real surveys and three usage strategies, evaluates existing machine-generated-text detectors against it, and proposes a lightweight respondent-level detector called SPABD.

What This Paper Is About

Online surveys are a primary tool for measuring human attitudes and behaviour, but participants increasingly use large language models to write their answers — one study cited here found 34% self-reported LLM use on a major crowdsourcing platform. Existing AI-text detectors work on long-form prose and score each answer in isolation, which is a poor fit for surveys made of many short, fragmented fields. This paper asks whether AI-assisted survey participation can still be detected once the AI is sophisticated enough to impersonate a specific respondent across an entire survey, and proposes a detection method that looks at behavioural patterns across a respondent's full answer set rather than at any single answer.

Key Contributions

1. The ASURRE benchmark dataset. Paired human and AI-assisted responses across three real surveys (Nature PhD Students 2019, Springer Nature Social Media 2017, OSMI Mental Health in Tech 2017–18) and three usage strategies — revision, full generation, and persona-grounded agentic completion — generated by multiple open and proprietary LLMs.

2. A systematic evaluation of existing detectors under increasing realism. The paper shows that full generation is readily detectable, revision is partially detectable, and persona-grounded agentic completion pushes most detectors toward chance — establishing that the most realistic threat is the least detectable one.

3. Four respondent-level behavioural cues. A minimal set of statistics — minimum answer length, length coefficient of variation, mean question-answer similarity, and number of answered fields — that separate human from AI completions without requiring a reference language model, training labels, or population calibration.

4. SPABD, a training-free aggregator. An algorithm that combines the four cues into a single respondent-level score using only five known-human reference responses, improving mean AUROC from 0.61 to 0.75 in agentic-completion settings and outperforming the best existing detector in 11 of 12 configurations.

Main Findings

• Detection difficulty tracks usage realism. Full generation is the easy case, with top existing detectors reaching AUROC 0.79–0.93. Revision drops performance to 0.55–0.74. Persona-grounded agentic completion leaves most detectors between 0.50 and 0.65, effectively near chance.

• Supervised MGT detectors can perform below chance. RADAR scores 0.28 on phd2019 and 0.39 on springer2017, evidence of a domain mismatch between its training distribution and short, open-ended survey answers. The prompt-based zero-shot classifier also sits near chance (roughly 0.45–0.53 on agentic settings).

• Persona agents leave behavioural fingerprints even when their prose looks human. Agents over-answer optional fields (10 vs 2 on OSMI), produce unusually uniform answer lengths (coefficient of variation 0.33 vs 0.67 on PhD2019), respect a minimum-length floor humans freely violate (6 vs 2 words), and answer slightly more on-topic than humans.

• Humans couple answers to question similarity; agents do not. Within a respondent, humans track how similar two questions are when writing their answers (slope β = 0.55). Persona-grounded AI collapses into a narrower, portrait-anchored register (β = 0.16, with non-overlapping confidence intervals).

• No single cue dominates, which is a strength. Coverage cues weaken when agents are told to skip fields; length cues weaken under length-restrictive prompting. Other cues remain informative, and aggregation preserves detection signal under targeted prompting.

• SPABD improves mean AUROC by +0.14. From 0.61 (best existing detector) to 0.75 across the 12 agentic-completion settings, with the only loss occurring in the combined adversarial mode on Springer (0.56 ± 0.10).

• Operating-point gains are large under tight false-positive tolerances. At 5% FPR, SPABD detects 27.7% of AI submissions versus 12.0% for the best existing detector; at 1% FPR, 15.5% versus 4.5%. On the shortest survey, springer2017, existing detectors remain stronger at low FPR.

• Five reference responders suffice. Scaling the known-human reference set from K=5 to K=100 raises mean AUROC only from 0.75 to 0.79 while shrinking seed-level standard deviation from 0.09 to 0.02. Most of the achievable gain is captured at K=5.

• Cross-LLM robustness holds for SPABD but not for baselines. Across Sonnet 4.6, gpt-oss-120b, and Gemini-3-Flash as persona simulators, SPABD stays within ±0.02 (0.886 / 0.909 / 0.895), while the identity of the best baseline detector changes between backbones and Fast-DetectGPT and Log-Likelihood fall below chance on gpt-oss-120b.

• The method degrades gracefully and tolerates mild contamination. Removing half the open-ended fields drops mean AUROC to 0.655, still above the best existing detector's 0.61 across full surveys. Contaminating one of five reference responses drops performance from 0.751 to 0.730; two drops it to 0.692.

• Fairness check favours non-native respondents. SPABD does not rely on fluency, perplexity, or vocabulary, and only one of four cues is length-based. Native and non-native respondents were compared across all three surveys; non-native respondents were flagged less often, not more.

Methodology in Plain English

The researchers started from real surveys collected before 2020, so every original response is verifiably human-written and free of AI contamination. They then simulated three ways a participant might use an LLM.

In revision, the LLM polishes the respondent's own answer without changing meaning, opinions, or details. In full generation, the LLM only sees the question text and invents a plausible answer with no grounding in any real respondent. In agentic completion, a two-stage pipeline reads a real respondent's full answer set, distils it into a structured "persona portrait" covering writing style, emotional register, English level, grammar error rate, seriousness, recurring themes, and a data-derived length habit, and then a fresh simulator fills the entire survey question by question, deciding autonomously which questions to answer or skip. A final humanisation pass suppresses obvious AI stylistic markers. Four prompting modes vary how much autonomy and constraint the agent has.

Because the persona agent's output is visually indistinguishable from human prose at the field level, the detection approach moves up a level. Instead of scoring each answer alone, the method computes four respondent-level statistics over the whole answer set: the shortest field answered, the variability in answer lengths, how closely answers align semantically with their questions, and how many fields were filled in. Each statistic is normalised against a small set of known-human reference responses using median and median-absolute-deviation, direction-adjusted by a prior on which way AI tends to shift it, and combined by taking the maximum standardised deviation. Respondents in the top tail of the resulting score are flagged as likely AI-assisted. The method requires no training, no labels beyond the handful of reference humans, and one small 80-million-parameter sentence encoder.

For comparison, the paper benchmarks six established detectors zero-shot: a prompt-based classifier, three token-probability methods (DetectGPT, Fast-DetectGPT, per-token log-likelihood), Binoculars, and the supervised RADAR classifier. Because all of these score individual texts, their answer-level scores are averaged to produce a respondent-level score.

Why This Matters

Impact on research. Survey-based research in the social sciences rests on validity assumptions — that respondents answer in good faith, with genuine effort, and as themselves. If AI-assisted completion is undetectable, those assumptions break down silently. This paper shows that the most realistic threat, persona-driven agentic completion, is precisely the case where existing detectors fail, and it offers a workable starting point for detecting it without collecting labelled training data for every new survey.

Real-world applications:

• Crowdsourced data collection platforms could screen submissions at the respondent level before payment decisions, using flag rates as a triage signal rather than an automatic rejection.
• Academic and market research panels could audit incoming survey data for AI contamination, particularly for long instruments with multiple open-ended fields.
• Sensitive-topic research such as the OSMI mental health survey could verify that disclosure-style narrative answers are genuine, where fabricated responses carry particular ethical weight.
• Longitudinal studies comparing pre- and post-LLM data could use the behavioural cues to estimate how much AI contamination exists in later waves.

Industry relevance. Any organisation that runs surveys for product research, employee engagement, market sizing, or clinical intake faces the same problem, and the incentives to delegate survey completion to LLMs (effort reduction, compensation maximisation) apply in commercial contexts as much as academic ones. The paper's method is attractive operationally because it needs no training pipeline, no labelled AI data, and only five verified human responses per survey — a practical constraint for organisations that cannot fine-tune a detector per instrument.

Future Directions

• Adversarial robustness. The paper explicitly scopes out targeted adversarial attacks. The published cues are interpretable and fully described, so an adversary who reads the paper could design prompting strategies to flatten each cue simultaneously. The authors note that a blind length attack still leaves SPABD at 0.839 AUROC where mean-z aggregation drops to 0.722, but systematic attack evaluation remains open.

• Extension to single-field and fewer-field instruments. SPABD's cues are undefined for one-field surveys, and detection degrades as open-ended fields are removed. Many widely used survey instruments have few or no open-ended fields, leaving that regime unaddressed.

• Language coverage. Evaluation covers three primarily English surveys. Whether the behavioural cues — particularly length variance and question-answer similarity — carry the same signal in other languages is untested.

• Improving performance under combined adversarial modes. The single setting where SPABD loses to the best baseline is Springer's combined mode, where multiple cues weaken at once, pointing to a need for cues that survive simultaneous suppression.

• Richer behavioural modelling. The current method uses four hand-designed statistics. Whether more expressive respondent-level representations — for instance, modelling the internal consistency of a persona across answers — add signal beyond these four is an open question.

Target Audience

Survey methodologists and social scientists who collect open-ended response data and need to assess data quality in the LLM era will find the practical framing most useful. NLP researchers working on machine-generated text detection will benefit from the dataset and from the demonstration that respondent-level aggregation outperforms per-answer detection on short texts. Trust-and-safety and data-quality engineers at crowdsourcing platforms and research panel providers can apply SPABD directly, since it requires only a handful of verified human responses and no training. Practitioners in adjacent fields — market research, employee experience measurement, clinical and public-health surveying — will find the threat model and the cue descriptions transferable even if they do not adopt the method wholesale.

Authors’ abstract

The use of LLMs to complete online surveys impacts the validity of survey-based research, but detecting such usage remains underexplored. We introduce an initial benchmark dataset, namely ASURRE, for AI-assisted survey participation to capture usage strategies ranging from full generation and revision to persona-grounded agentic completion. Controlled by these strategies, LLM-assisted survey responses are generated using multiple LLMs on three real-world surveys in different disciplines, paired with genuine human responses. Our evaluation of existing machine-generated text (MGT) detectors shows that naive AI usage is readily detectable, whereas persona-grounded agents that mimic entire respondents push detector performance toward chance. We further show that agentic completion cannot fully replicate respondent-level behaviour and leaves distinctive behavioural traces. While individual cues can be circumvented by targeted prompting, a simple few-shot, training-free aggregator over these cues improves mean AUROC by +0.14 over the best existing detector across agentic settings. Our project is available at https://github.com/mike-qz-wang/ASURRE.