The Pulse
AI Lifted Consultants’ Data-Science Scores, but Not Their Know-How
A randomized trial of 487 management consultants found that ChatGPT improved performance on unfamiliar data-science tasks. After the experiment, researchers found no evidence that AI users had gained technical knowledge they could apply wit
AI.info Team · Written with AI · fact-checked ·
487 management consultants used ChatGPT on unfamiliar data-science tasks in a randomized trial, but researchers found no evidence that the AI group gained technical knowledge they could demonstrate without the tool. The study, published October 26 in Nature Human Behaviour, puts a boundary around what “upskilling” can mean when workers perform new tasks with an AI assistant: better work with the tool did not translate into measurable independent knowledge after the experiment.
Higher scores with ChatGPT, no offline knowledge gain
The researchers gave consultants access to ChatGPT and training in using it, then compared their performance with a control group. Participants worked on tasks outside their usual skill sets, designed so ChatGPT could not solve them independently. The AI-assisted group scored higher than controls on three data-science tasks, though the study found they did not perform at the level of expert data scientists.
“We find no evidence that treated workers are better at answering technical questions without ChatGPT post-experiment, suggesting that their demonstrated technical capabilities do not imply knowledge acquisition.”
Emma Wiles et al., “Generative AI and the temporary upskilling of knowledge workers”
The researchers’ follow-up questions tested whether participants could answer data-science and coding questions without AI. Their results did not show a reliable advantage for the people who had used ChatGPT. The finding does not establish that AI makes workers lose existing skills; the study tested whether participants acquired knowledge in an area outside their skills.
Python, statistics and a prediction problem
The exercises covered three kinds of work: writing Python code to clean and combine data, interpreting statistical information and correcting modeling errors, and building and assessing a predictive model. Consultants were assigned two of the three tasks and had up to 90 minutes for each. The prediction exercise used soccer-match data, while the coding task asked participants to clean customer and order records and identify customers with the highest average spending.
The trial began with 986 workers allocated between treatment and control groups; 573 started the survey, and 487 completed both tasks used in the main analysis. That distinction matters when interpreting the headline sample: the published abstract’s 487 refers to the analyzed participants, not everyone initially allocated. The supplementary materials also describe the researchers’ separate post-experiment knowledge questions, which were answered by a larger group.
Performance is not the same as learning
The results separate two outcomes that can look similar in workplace evaluations. An employee may complete a technical assignment more successfully with ChatGPT, while still being unable to explain or repeat the underlying work independently. The paper describes AI as an “exoskeleton” that can extend workers’ capabilities, but says those gains depend on continued use of the technology.
That distinction matters for companies that treat AI-assisted output as evidence of employee development. A finished analysis may demonstrate that a person can collaborate with an AI system, not that the person has learned data science. The study measured task performance and post-experiment knowledge; it did not test whether longer-term practice, structured teaching or repeated use of AI with deliberate instruction would change retention.
A BCG study with an OpenAI co-author
The trial involved management consultants recruited through Boston Consulting Group, whose Henderson Institute is listed as the affiliation for several authors. BCG provided employee time and internal resources for the research, and the paper says BCG-affiliated authors took part in its design, data collection, analysis and writing. Co-author Pamela Mishkin is affiliated with OpenAI’s Economic Impacts Research group, which the paper says collaborated on task design.
The study is therefore evidence about one group of consultants working on a narrow set of data-science exercises, not a demonstration that AI erodes skills across occupations. Whether coached AI use can turn better supported performance into retained knowledge—and whether that happens in other jobs—remains outside the trial’s results.