Are they lovers or friends? Evaluating LLMs' Social Reasoning in English and Korean Dialogues

AI summary

Overview

• Research area: Natural Language Processing — evaluation of large language models' social reasoning, with a cross-linguistic (English/Korean) and cross-cultural focus.
• Technical level: Intermediate.
• Scope: The paper introduces SCRIPTS, a 1.1k-dialogue bilingual benchmark built from movie scripts, and uses it to test whether nine LLMs can correctly infer the social relationship between speakers in a conversation.

What This Paper Is About

LLMs are increasingly used in multi-party interactions, so they need to recognize who is talking to whom — a parent, a romantic partner, a coworker — because misreading a relationship can lead to socially inappropriate or unsafe responses. Prior work on this problem used simplified setups: multiple-choice classification, narrow relationship taxonomies, single utterances, or static character labels carried over from movie metadata. This paper builds a more realistic benchmark from multi-turn movie dialogues in English and Korean and shows that current models still fail often, especially in Korean.

Key Contributions

1. The SCRIPTS benchmark. A dataset of 1,147 dialogues (580 English, 567 Korean) sourced from 60 movie scripts (28 English, 32 Korean), annotated with an uncertainty-aware relationship schema: Highly Likely, Less Likely, and Unlikely.
2. A dialogue-level, soft-labeling annotation scheme. Instead of assigning fixed character roles from movie metadata, annotators label relationships as they appear in a specific dialogue, allowing multiple Highly Likely relationships per dialogue (3.67 on average) and a distribution over plausibility.
3. Evaluation of nine LLMs in an open-ended generation setting, showing limited social reasoning in both languages and a substantial rate of Unlikely predictions.
4. A failure-mode analysis and an intervention study. The authors identify four recurring failure types and test whether supplying socio-demographic and relational information reduces implausible predictions.

Main Findings

• Moderate but uneven accuracy. Current LLMs achieve roughly 75–80% on the English dataset and 58–69% in Korean (stated in the abstract as the overall range). GPT-4o is the best-performing model with a Highly Likely rate of 79% in English and 69% in Korean.
• Models frequently produce implausible relationships. Models incorrectly infer an Unlikely relationship in 10.8–31.9% of their responses, and this tendency is amplified in Korean, with the rate increasing by an additional 7.2–16.5 percentage points. The abstract summarizes this as predicting an Unlikely relationship in 10–25% of responses in both languages.
• Large gaps between models on the same language. Even the best model, GPT-4o, still responds with human-annotated Unlikely relationships in 10.9% of cases and produces Less Likely responses in 10.0%. Llama-3.1-8B-Instruct makes Highly Likely predictions in only 41.3% of the English dataset.
• Thinking and chain-of-thought do not help much. Enabling thinking yields mixed results by language; in Korean its effect is negligible or harmful (for example, a 4.4 percentage-point drop in Highly Likely responses for Gemini-2.5-Flash). None of these differences are statistically significant (bootstrap test, p > 0.05). CoT prompting also fails to help consistently: Gemini 2.5 Flash shows a 1.7 percentage-point drop in Highly Likely responses in English, and Llama-3.1-8B-Instruct shows a 3.1 percentage-point rise in Unlikely responses in Korean.
• Static movie labels are misleading. For 19% of cases, the movie-level relationship label was judged irrelevant to the dialogue, and even when applicable, the authors' annotations identify more than three Highly Likely relationships per dialogue on average.
• Four failure modes. (1) Failure to distinguish terms of address from terms of reference — models treat "that's my Dad" as if the speaker is addressing the listener as Dad. (2) Failure to aggregate multiple cues — models latch onto a single cue (such as one speaker complying with an instruction) and ignore contradicting cues like the absence of Korean honorifics. (3) Failure to recognize atypical relationships — models reject non-stereotypical pairings, such as a parent–child relationship conducted in a peer-like tone. (4) Failure to understand language- or culture-specific features — mainly in Korean, where terms of address and honorifics are misinterpreted.
• Error composition differs by language. Among 30 manually examined GPT-4o failures, English errors were dominated by failure to aggregate multiple cues (36.7%), while Korean errors were dominated by language- or culture-specific features (46%).
• Social information reduces Unlikely predictions but does not reliably improve accuracy. Providing human gold labels for demographic cues (age, gender) and relational dimensions (intimacy, hierarchy, formality) yielded no substantial or consistent gains for GPT-4o, but it reduced the proportion of Unlikely predictions. Relational dimension labels shifted predictions toward more intimate or less hierarchical relationships (e.g., Strangers → Romantic Interest at 3.3%; Parent–Children → Friends at 2.9%), sometimes incorrectly.
• Model-generated social information does not help. GPT-4o achieves under 60% accuracy on age and gender and below 75% on relational dimensions, so inferred labels are often wrong. The accuracy of inferred social information was 72.8% when relationship reasoning improved versus 65.5% when it deteriorated for age and gender, and 53.3% versus 50.7% for relational dimensions.
• Korean-specialized models help on Korean. A.X-4.0-Light ranks first in both languages (0.589 English, 0.467 Korean), followed by Qwen-3 (0.565 English, 0.423 Korean). Llama-3.1 ranks third in English (0.413) but last in Korean (0.321), while Exaone-4.0 ranks third in Korean (0.409).

Methodology in Plain English

The researchers collected 60 modern movie scripts — 28 English scripts crawled from IMSDb and 32 Korean scripts gathered through an onsite visit to the Korean Film Archive plus an open-access Korean script community. They kept only films released after 2000, capped at PG-13 for English and 15세 관람가 for Korean, extracted text using OCR where scripts existed only in print, and cleaned the output into structured dialogues. From roughly 23k initial scenes, they filtered down to 1,322 dialogues (698 English, 624 Korean) with at least three turns and two or three participants, prioritizing diverse speaker combinations. All character names were replaced with placeholders like [A] and [B] to prevent contamination and reduce bias.

Annotation ran in four phases with three annotators per dialogue (17 English annotators recruited in the U.S., 14 Korean annotators). First, annotators picked Unlikely relationships from a predefined list of 27 types, requiring at least two of three annotators to agree. Second, they provided up to five open-ended labels for the most plausible relationships, and the union became the Highly Likely set. Third, remaining predefined types that were neither Highly Likely nor Unlikely became Less Likely. Fourth, they annotated auxiliary attributes such as age, gender, formality, hierarchy, and intimacy. Dialogs where annotators' Highly Likely labels had no overlap were discarded, removing 13.2% and leaving 1,147 dialogues.

For evaluation, each model generated the relationship in free-form text rather than choosing from a fixed list, with example relationship types included in the prompt as reference candidates. Each model ran five times per dialogue, and the majority response was scored against ground truth. GPT-4o served as the automatic evaluator, with 92.0% human-validated accuracy in a validation experiment.

Why This Matters

• Impact on research. The paper argues that social reasoning needs evaluation setups that reflect real conversational complexity: multi-turn dialogues, multiple plausible answers, three-speaker scenes (41.8% of the data), and culturally grounded annotation. It also shows that techniques successful on math and science reasoning (CoT, internal thinking) do not transfer to social reasoning, which the authors suggest requires a fundamentally different strategy.
• Real-world applications:
  • Conversational agents and chatbots that need to adjust tone, formality, and content to the relationship between speakers.
  • Multi-agent systems and group chats where agents must track who is speaking to whom.
  • Safety and privacy systems that prevent over-sharing — the paper's example shows a model confusing a clerk–customer exchange for a mother–child relationship and inviting inappropriate personal disclosure.
  • Multilingual deployment, where Korean-specific markers such as honorifics and address terms are mishandled.
• Industry relevance. The benchmark is published with a dataset on Hugging Face and code on GitHub, providing a target for developers of Korean-specialized and multilingual models. The results show that Korean-specialized models outperform similarly sized multilingual models on Korean, which supports language- and culture-specific development rather than English-only approaches.

Future Directions

• Extend beyond two languages. The authors state their analysis is limited to English and Korean and may not generalize to other cultural contexts.
• Move beyond movie scripts. The limitations section calls for more realistic sources, such as privacy-preserving real conversations and human–AI dialogue logs, noting that scripts remain a practical proxy given the privacy and labeling challenges of real dialogue.
• Train for rare and atypical relationships. The paper suggests that broader exposure to rare, atypical, and culturally diverse relationships during training may reduce models' reliance on common relationship patterns.
• Provide explicit social cues at inference time. The intervention study indicates that supplying relational and social information reduces implausible inferences, but the authors note that current models cannot yet infer these dimensions accurately enough to do this on their own. They also caution that CoT traces analyzed in the paper may reflect post-hoc rationalizations rather than the actual mechanisms producing answers.

Target Audience

Researchers and engineers working on LLM evaluation, multilingual NLP, and social or pragmatic reasoning; developers building conversational agents and multi-agent systems that must behave appropriately across relationships and cultures; and Korean-language NLP practitioners interested in how culturally specific markers such as honorifics and terms of address expose model weaknesses.

Authors’ abstract

As LLMs are increasingly deployed in real-world interactions, their social reasoning in interpersonal communication becomes critical. To explore their capabilities, we introduce SCRIPTS, a 1.1k-dialogue dataset in English and Korean, sourced from movie scripts and propose a social reasoning task based on SCRIPTS that evaluates the capacity of LLMs to infer the social relationships (e.g., friends, lovers) between speakers in each dialogue. Evaluating nine models on our task, current LLMs achieve around 75--80% on the English dataset and 58--69% in Korean, and models predict an Unlikely relationship in 10--25% of responses in both languages. Furthermore, we find that thinking models and chain-of-thought prompting provide minimal benefits for social reasoning and occasionally amplify social biases. In sum, there are significant limitations in current LLMs' social reasoning capabilities, especially for Korean, highlighting the need for efforts to develop socially-aware LLMs across languages.