Are Large Language Models Sensitive to the Motives Behind Communication?

AI summary

Overview

Research area: Natural Language Processing / AI evaluation, at the intersection of cognitive science and LLM behavior (motivational vigilance, rational models of social inference).

Technical level: Intermediate. The paper uses a formal Bayesian rational model from cognitive science as a benchmark, but the three experiments and their conclusions are described in accessible terms.

Scope: The paper evaluates whether LLMs discount information from biased or self-interested sources the way humans and a rational Bayesian model do, testing them across three paradigms of increasing realism — a controlled two-player judgment game, a vignette-based advice paradigm, and 300 real YouTube sponsorship segments.

What This Paper Is About

Almost all the text LLMs process was written by someone with a goal, so models that act as agents must be able to tell whether a source is benevolent or self-serving before deciding how much to trust it. The authors ask whether LLMs possess "motivational vigilance" — the human capacity to track the intentions and incentives behind communication and adjust belief updates accordingly. They do this by comparing LLM behavior against a normative rational model (Oktar et al.) and against human data, first in tightly controlled cognitive-science tasks and then in naturalistic online advertising.

Key Contributions

1. A first systematic measurement of motivational vigilance in LLMs, using a rational Bayesian model of learning from motivated testimony as a normative benchmark rather than ad hoc accuracy metrics.
2. A three-paradigm evaluation design that separately tests whether LLMs can discriminate deliberate communication from incidental observation (Experiment 1), calibrate belief updates to speaker benevolence and incentives (Experiment 2), and generalize that vigilance to realistic settings (Experiment 3).
3. A new ecologically valid dataset of 300 randomly sampled YouTube sponsorship segments from SponsorBlock, scraped via the YouTube Data API, with brand and product names censored by GPT-4o to remove pre-existing brand impressions.
4. A simple, effective prompt-steering intervention that increases the salience of speaker intentions and incentives and partially restores rational alignment in the naturalistic setting.

Main Findings

• LLMs distinguish deliberate advice from incidentally observed answers. In the adapted Watson and Morgan paradigm, both GPT-4o and Claude 3.5 Sonnet shifted their estimates less when receiving deliberate "advice" than when they had "spied" Player 1's true answer, consistent with human participants. They also adjusted more under cooperative than competitive payoff structures, showing sensitivity even higher than people.
• Experimental design induced the needed uncertainty. With a noise-based replacement for the original 2-second time constraint, LLM first-guess accuracy was 18–44%, slightly below the 55% of human participants in the original setting, and being correct did not significantly affect how much LLMs updated.
• Chain-of-thought made LLMs more, not less, susceptible. CoT significantly increased shifts toward Player 1 in 3 of 4 cases, exceeding 60%, and deviated from human influence magnitudes of 0.34 for advice and 0.37 for spied information.
• Frontier non-reasoning models are rational and human-like in controlled settings. GPT-4o, Claude 3.5 Sonnet, Gemini 2.0 Flash, and Llama 3.3-70B had Bayesian–LLM correlations of 0.911, 0.845, 0.788, and 0.876 respectively. GPT-4o showed the highest internal rationality at 0.911, and its LLM–Human correlation was 0.943.
• Humans correlated better with these LLMs than the rational model did. For frontier non-reasoning models the human match was statistically significantly better (p < .05) than the model fit on those same LLMs' elicited priors, in over 90% of individual prompt conditions and roles — suggesting LLMs capture human heuristics or biases beyond rational analysis.
• Reasoning models were less vigilant. o1 (0.705) and o3-mini (0.716) correlated around 0.7 with the rational model, while DeepSeek-R1 averaged around 0.326. In the user/assistant role, o-series models dropped by around −0.1, and DeepSeek-R1 fell from 0.793 (first-person) to −0.141, indicating complete insensitivity to character trustworthiness and incentives.
• Vigilance scales with model size. Smaller models — Llama 3.1-8B (0.608), Llama 3.2-3B (0.349), and Gemma 3-4B (0.288) — correlated far worse with both the rational model and human behavior, pointing to a direct lack of vigilance capability rather than a human-like bias.
• Vigilance largely breaks down in realistic YouTube sponsorship settings. Correlations with the Bayesian model fell to r < 0.2, with default-prompt values such as 0.024, 0.008, −0.006, and 0.033 for GPT-4o and 0.121, 0.190, 0.094, and 0.119 for Claude 3.5 Sonnet.
• Prompt steering partially recovers vigilance. Appending a phrase asking the model to consider the channel's intentions and incentives raised correlations in most conditions, significantly so (Fisher r-to-z test, α = .05) in 8 of the 12 listed combinations — for example GPT-4o Direct User rising from −0.006 to 0.312. Other steers based on Gricean or bias-oriented concepts were less effective.
• Longer contexts hurt vigilance. The Bayesian model fit better for the shortest 25% of transcripts (Q1) than the longest 25% (Q4) across nearly every model, prompting technique, and perspective in Table 3.

Methodology in Plain English

The authors took three established experimental designs from psychology and adapted them so LLMs could play the same role human participants had.

In the first experiment, two "players" (both played by the same LLM) looked at images of blue and yellow circles and estimated the difference in counts. Player 1 answered 20 easy images, Player 2 answered 20 hard images with the same answer list. On each trial, Player 1 either deliberately sent a numeric suggestion or had its real answer secretly "spied" by Player 2. Because LLMs are not subject to a human time limit, the researchers added noise to the images to make the models uncertain. They ran 30 trials per model, payoff structure, and prompting method at temperature 1, for both direct responses (max 10 output tokens) and chain-of-thought (max 750 tokens).

The second experiment used the rational model of Oktar et al. as a quantitative benchmark. In that model, a speaker's utterance probability depends on a joint reward combining the listener's outcome and the speaker's own outcome, weighted by a benevolence parameter λ (with λ = 1 meaning purely altruistic and λ = 0 purely self-interested); a vigilant listener inverts this to infer the true quality of the recommended option. The researchers elicited three types of scores from each LLM — influence scores (perceived product quality), incentive scores (perceived benefit to the speaker), and trust scores (perceived benevolence) — across finance, real estate, and medicine settings, with four characters and four incentive values giving 16 character–incentive speakers. They then measured the Pearson correlation between the LLM's influence scores and the Bayesian model's predictions fitted to that same LLM's own elicited priors. Most models were prompted 40 times per setting, reasoning models 10 times due to cost, and GPT-4o 80 times due to available Azure credits.

The third experiment replaced vignettes with real data. The authors pulled YouTube sponsorship segments from SponsorBlock, used the YouTube Data API to scrape titles, channel names and descriptions, and transcript text for 300 randomly selected video IDs, and censored brand and product names with GPT-4o. Each of the three variables was elicited in a separate context window for each sponsorship, with n = 1 per video and prompting combination at temperature 0. They then tested whether re-prompting the model to focus on the channel's intentions and incentives improved alignment with the Bayesian posterior.

Why This Matters

Impact on research. The results argue against the worry that LLM training paradigms make vigilance impossible, and they provide a reusable normative benchmark for measuring it. The paper also introduces a taxonomy organizing future vigilance research around inputs (types of motives and informants), processes (heuristic versus rational accounts), and outputs (text versus nonverbal cues of intent).

Real-world applications:

• AI shopping and browsing agents that need to discount promotional claims in order to avoid scams, as in the paper's example of tracking information in ads when shopping online.
• Financial, real estate, and medical advice assistants, all domains in which the paper notes generative AI systems are actively developed or deployed.
• Attaching credibility weights to sponsored content, online reviews, word-of-mouth promotion, op-eds, and partisan social media posts that LLMs are asked to summarize or act on.
• Multimodal agents navigating distracting online environments such as pop-ups, where prior work has shown misleading stimuli degrade task performance.

Industry relevance. The finding that simple prompt steering raises rational alignment points to a cheap deployment-time mitigation, and the finding that reasoning models underperform in the assistant role bears directly on how agentic systems are built.

Future Directions

• Integrate competence vigilance. The rational model used here covers motivations only; the authors note that psychological research has also produced rational models for inference about a source's competence, and a complete benchmark would combine both.
• Test whether the naturalistic failures stem from competence-related inference that the rational model does not represent, or from purely heuristic processing.
• Explore the full input, process, and output space of vigilance, including relational, romantic, affiliative, and presentational motives; individual versus group informants; heuristic versus rational accounts; and nonverbal cues such as gaze and gestures.
• Explain between-model variance in vigilance, which the authors suggest could be probed using mechanistic interpretability, and decide when it is preferable to align LLMs with normative ideals versus empirical human inference patterns.

Target Audience

Researchers working on LLM evaluation, alignment, and agentic systems; cognitive scientists and psychologists interested in testing theories of social inference against machine behavior; and practitioners building LLM agents that must weigh the credibility of human-generated content.

Authors’ abstract

Human communication is motivated: people speak, write, and create content with a particular communicative intent in mind. As a result, information that large language models (LLMs) and AI agents process is inherently framed by humans' intentions and incentives. People are adept at navigating such nuanced information: we routinely identify benevolent or self-serving motives in order to decide what statements to trust. For LLMs to be effective in the real world, they too must critically evaluate content by factoring in the motivations of the source -- for instance, weighing the credibility of claims made in a sales pitch. In this paper, we undertake a comprehensive study of whether LLMs have this capacity for motivational vigilance. We first employ controlled experiments from cognitive science to verify that LLMs' behavior is consistent with rational models of learning from motivated testimony, and find they successfully discount information from biased sources in a human-like manner. We then extend our evaluation to sponsored online adverts, a more naturalistic reflection of LLM agents' information ecosystems. In these settings, we find that LLMs' inferences do not track the rational models' predictions nearly as closely -- partly due to additional information that distracts them from vigilance-relevant considerations. However, a simple steering intervention that boosts the salience of intentions and incentives substantially increases the correspondence between LLMs and the rational model. These results suggest that LLMs possess a basic sensitivity to the motivations of others, but generalizing to novel real-world settings will require further improvements to these models.