The Pulse
Study Finds AI Human-Likeness Is Separate From Task Accuracy
A study of 37 AI systems found that human-likeness did not closely track conventional performance measures in image captioning. Across 72,191 Turing-like tests, human judges often struggled to tell AI responses from human ones in language a
AI.info Team · Written with AI · fact-checked ·
Human judges correctly recognized AI-generated image captions only 46% of the time in one part of a study published October 26, 2026—a result below chance that suggests they often mistook machine responses for human ones. Across six language and vision tasks, the researchers found that an AI system’s ability to imitate people did not closely track its conventional task performance. The findings make a case for measuring human-likeness separately from whether a model gets an answer right.
36,499 responses, six kinds of task
The team assembled 36,499 responses from 636 human participants and 37 AI systems. The tasks covered image captioning, word association and conversation, as well as color estimation, object detection and visual attention. Researchers then ran 72,191 Turing-like trials, asking 1,916 human judges and 10 AI judges to decide whether responses came from people or machines.
In the captioning trials, judges identified human captions as human 67% of the time, but identified AI captions as AI only 46% of the time. In conversation tests, they labeled human exchanges as human 66% of the time and detected AI exchanges 47% of the time. The task-specific results varied: in object detection, for example, judges recognized AI responses 63% of the time.
Human resemblance did not equal accuracy
The authors compared models’ performance on image-captioning benchmarks with how often their captions passed as human. They found that stronger conventional benchmark scores did not reliably mean stronger imitation: BLIP, for instance, imitated human captions better than GIT, although GIT scored higher on captioning quality. The paper’s authors write, “Imitation ability showed minimal correlation with conventional AI performance metrics, suggesting that passing as human is an important independent evaluation criterion.” Mengmi Zhang, Elisa Pavarino et al., Nature Human Behaviour
That distinction matters because a response can be accurate without resembling how a person would describe or interpret something. The researchers give the example of an image caption that correctly describes a man and a child on a beach but misses the unusual visual impression that a human observer highlights. The study therefore treats imitation as a different evaluation target, not as a measure of intelligence.
AI judges found patterns people missed
Simple machine-learning classifiers often distinguished human from AI responses more accurately than human judges, including in image captioning and conversation. Those classifiers were trained on examples from the study, so the comparison is not a test of whether an untrained machine can spot AI; the authors also tested ChatGPT in zero-shot settings, with mixed results across tasks. The paper reports that judge performance depended on the task, and that feedback could improve people’s ability to identify machine-generated responses in some tests.
The authors caution that their combined imitation score can obscure the difference between mistaking AI for human and mistaking human work for AI. Those errors carry different consequences when the goal is to detect synthetic content. They also stress that the tests assess whether outputs resemble human responses under specific conditions, not whether a system thinks or understands as a person does.
Imitation can help—and enable fraud
Human-like output can serve useful purposes, such as matching human judgments in applications where that is the goal. The authors also warn that imitation could support fake information, phishing and other scams. They have released the study’s data and code for evaluating models on the tested tasks, giving researchers a way to compare imitation performance alongside standard benchmarks.
The study’s central distinction is measurable: a model’s accuracy on a task and its resemblance to human responses are not interchangeable scores. In the captioning comparison, one could improve without reliably improving the other.