Where Norms and References Collide: Evaluating LLMs on Normative Reasoning

AI summary

Overview

Research area: Natural Language Processing, specifically situated dialogue, reference resolution, and the evaluation of social/normative reasoning in Large Language Models (LLMs), with implications for embodied AI and robotics.

Technical level: Intermediate. The paper assumes familiarity with reference resolution, LLM prompting setups, and evaluation benchmarks, but its core ideas are explained through concrete everyday examples.

One-sentence scope: The paper introduces SNIC (Situated Norms in Context), a human-validated, procedurally augmented 9,000-instance diagnostic testbed, and uses it to measure whether 12 LLM configurations can resolve ambiguous referring expressions by identifying and applying physically grounded social norms.

What This Paper Is About

When someone in a shared kitchen says "Can you pass me a mug?" and three mugs are present (one clean, two with leftover coffee), the intended referent is normally the clean mug because of a shared social norm about not drinking from dirty or unknown mugs. Resolving references this way — the paper calls it norm-based reference resolution (NBRR) — requires inferring implicit normative expectations from physical and social context, not just from language. The authors ask whether current LLMs can do this, and they build a controlled testbed to find out.

Key Contributions

1. SNIC (Situated Norms in Context), a new diagnostic testbed for NBRR. The final artifact contains 9,000 questions divided into nine categories of 1,000 questions each, where each category represents a specific social norm or norm conflict. It emphasizes physically grounded everyday norms (cleaning, tidying, serving, cooking) rather than abstract moral judgments.

2. A human validation study grounding the testbed empirically. 210 participants recruited via Prolific (ages 21 to 69, approximately balanced by gender) rated an initial handcrafted set of 120 examples, which was filtered to 51 seeds based on whether the norm-guided referent was the plurality winner. Each participant saw six scenes in randomized order; the study was IRB approved and compensated at $1.00 for a 5-minute study.

3. A procedural augmentation pipeline inspired by Thierauf, Abrams, and Scheutz (2024) that scales the 51 human-validated seeds to 9,000 examples by varying object types, object properties, and settings while preserving the underlying normative structure, using Python f-string template rules and paired Prolog (first-order logic) formal representations of each scene.

4. A three-setting evaluation of 12 LLM configurations (Granite-3.1 2B/8B, Phi-3 3.8B/14B, Phi-4 3.8B, Llama-3 3B/8B, Qwen-2.5 3B/7B/14B, GPT-4o-mini, GPT-4.1), plus a separate evaluation on the original 51 human-validated items and a Spearman correlation analysis across norm categories.

Main Findings

• LLMs rarely infer implicit norms on their own. With scene description plus the Prolog/FOL formalization, the best model (GPT-4.1) reached 59.62% accuracy, with an average of 44.08% across all models.

• Formalization did not reliably help. Average accuracy with only the scene description was 44.22%, with GPT-4.1 at 56.98%. Two exceptions stood out: Llama-3 3B degraded sharply with the Prolog formalization (12.01% versus 33.71% with description only), while Granite-3 2B improved (46.65% versus 27.52%). The authors suggest LLMs may have weak understanding of Prolog as a descriptive language, or that formalization provides no additional normative hints.

• Supplying the norms explicitly produced the largest gains. When all applicable social norms were listed in the context alongside the description and FOL representation, average accuracy rose to 70.51%. GPT-4.1 reached 99.6% and GPT-4o-mini 93.57%. Llama-3 3B was the only model that did not see a significant increase, reaching 26.6%.

• Human agreement with hypothesized referents was mixed but directionally supportive. Across the filtered 51 questions, full match rates ranged from 31.1% to 64.7% per norm group. Norm 1 yielded 58.6% full match and 33.1% partial match; Norm 5 had the highest full match at 64.7%; Norm Conflict 2 had the lowest at 31.1% full match with 40.0% partial. Adapted Fleiss' kappa values ranged from 0.215 (Norm 1) down to −0.113 (Norm 4), indicating generally modest convergence.

• Partial matches show norms still pull on interpretation. Because participants could select multiple referents, high partial-match rates (e.g., 44.4% for Norm Conflict 1, 42.9% for Norm 6) indicate that annotators frequently included the norm-guided referent alongside other choices even when ambiguity remained.

• Norm categories correlate with each other in model behavior. The Spearman correlation analysis found strong positive correlations between structurally similar norms (the paper cites Norm 1 and Norm 3) and strong negative correlations between some norm/norm-conflict pairs (the paper cites Norm 3 and Norm Conflict 1), which the authors interpret as models imposing a fixed priority ordering even when the situation demands a rearrangement such as prioritizing safety over tidiness.

• Expansion made the task harder. On the original 51 human-validated items with no norms or FOL augmentation, accuracies were generally higher than on the full corpus — for example Llama-3 3B at 66.66%, Qwen-2.5 7B at 68.67%, and Phi-3 14B at 58.84% — while GPT-4o-mini reached 45.09%. The authors read this as evidence that the larger dataset introduces more variation and challenging distractors.

• Composition of the dataset. Object counts: 5 objects 14.13%, 6 objects 42.79%, 7 objects 28.4%, 8 objects 14.69%. Task distribution (over 8,000 scenes): cleaning 50.00%, serving 25.00%, tidying 12.50%, cooking 12.50%. Settings: living room 15.83%, restaurant 15.37%, dining room 14.84%, garden 14.64%, banquet hall 14.12%, kitchen 14.08%, library 11.11%. The corpus has 46 distinct objects, 7 distinct settings, and 7 explicit properties. Referring expressions are split between "hand me/them the X" (5,000) and "pick it up" (4,000).

Methodology in Plain English

The researchers started by hand-writing 120 vignettes, each describing a textual scene with a setting, a task (such as cleaning or cooking), 5–7 candidate objects with properties (like clean or dirty), and an ambiguous imperative utterance such as "hand me the book." Each scene was designed so that the referring expression was ambiguous and only resolveable using a social norm.

They then ran an online human study with 210 participants to check whether people actually pick the norm-guided referent. For each question they measured full match (norm-guided referent chosen exclusively), partial match (chosen alongside others), and no match (not chosen), and adapted Fleiss' kappa by binarizing responses to whether the norm-conforming referent was selected.

Only the 51 scenes where the norm-guided referent was the plurality winner were kept as seeds. These were then expanded with rule-based Python f-string templates that swapped in different objects, properties, and settings while preserving the same normative structure, producing 9,000 examples. Each scene also has a parallel Prolog first-order logic representation that assigns IDs to objects, so the researchers could compare natural-language input against a formally precise one.

Every model was run under three conditions: scene description only; description plus the FOL formalization; and description plus FOL plus an explicit list of the applicable social norms. All models other than the GPT-4 family were run through Ollama using DSPY in 4-bit quantized form at temperature 0 and seed 0 on a single A6000 GPU with 48GB VRAM; GPT-4o variants were evaluated as served by OpenAI. The paper also notes that vignettes are treated as a reasonable proxy for physical scenarios, following prior work such as TextWorld.

Why This Matters

The paper argues that detecting and reasoning about physically grounded social norms in text remains a hard, underexplored capability, and it frames this as a blind spot rather than an oversight of scale. Because many norms are implicit, rarely stated, and culturally shared, the authors suggest the gap cannot simply be fixed by more training on text — they observe there is no large-scale corpus of physically situated, norm-sensitive references. They also connect the explicit-norm-prompting result to alignment: models can follow norms when handed them, but do not reliably produce them, which matters as agentic models begin to interact directly with people.

Real-world applications:

• Assistive and service robotics, where a robot must hand over the clean utensil during cooking or the dirty one during cleanup.
• Household organization and tidying systems that need to respect ownership and cultural preferences rather than overgeneralizing (the paper cites Wu et al. 2023 finding LLMs treating all drawers as equivalent).
• Command and object disambiguation in robot dialogue systems, where referents are ambiguous without context.
• Safety-aware task planning, where agents must prioritize hazards over routine cleanliness goals when norms conflict.

Industry relevance: For developers building conversational assistants, embodied agents, or household robots, the paper signals that prompt-time norm injection is currently the most reliable lever, and that model choice matters — performance varied substantially across families and sizes, and Llama-3 3B even regressed under formalization. The authors note that including social norm understanding in continuous model evaluation could serve as a resource for assessing usability in social and collaborative settings.

Future Directions

• Extend beyond text to multimodal and physically grounded input, since the current testbed does not evaluate perceptual cues or physical affordances that embodied agents rely on.
• Explore alternative training paradigms such as reinforcement learning, human feedback alignment, or explicit norm supervision, which the current work deliberately does not investigate.
• Develop more rigorous norm validation, including structured norm elicitation and cross-participant agreement protocols, to establish stronger consensus on the underlying norms; the authors note the current study lacks these, which limits the granularity of its behavioral grounding.
• Investigate how norms are ranked and how that ranking can be reshaped, given the strong negative correlations between certain norms and norm conflicts (for example Norm 3 and Norm Conflict 1), which suggest models apply fixed precedence that may not reorder appropriately when safety should override tidiness.
• Test other forms of formalization, since the Prolog representation produced inconsistent effects across models and the authors state further testing is required to understand how formalizing language affects LLM behavior.

Target Audience

Researchers in NLP, situated dialogue, and reference resolution; roboticists and embodied AI practitioners who need agents to interpret ambiguous instructions in social settings; AI ethicists and alignment researchers interested in how norms are (or are not) encoded in LLMs; and benchmark/dataset designers, since the paper details a human-validated seed-then-augment pipeline that balances scalability with interpretability.

Authors’ abstract

Embodied agents, such as robots, will need to interact in situated environments where successful communication often depends on reasoning over social norms: shared expectations that constrain what actions are appropriate in context. A key capability in such settings is norm-based reference resolution (NBRR), where interpreting referential expressions requires inferring implicit normative expectations grounded in physical and social context. Yet it remains unclear whether Large Language Models (LLMs) can support this kind of reasoning. In this work, we introduce SNIC (Situated Norms in Context), a human-validated diagnostic testbed designed to probe how well state-of-the-art LLMs can extract and utilize normative principles relevant to NBRR. SNIC emphasizes physically grounded norms that arise in everyday tasks such as cleaning, tidying, and serving. Across a range of controlled evaluations, we find that even the strongest LLMs struggle to consistently identify and apply social norms, particularly when norms are implicit, underspecified, or in conflict. These findings reveal a blind spot in current LLMs and highlight a key challenge for deploying language-based systems in socially situated, embodied settings.