Situation Graph Prediction for User Perspective Modeling

AI summary

Overview

Research area: Artificial Intelligence — perspective-aware user modeling, structured knowledge representation, personal AI agents, and multimodal inference (CCS categories: Human-centered computing / HCI, and Computing methodologies / Knowledge representation and reasoning).

Technical level: Intermediate — the core idea is intuitive, but the paper formalizes it with ontology notation, latent-variable notation, and set-based evaluation metrics that require some background in AI/knowledge graphs.

One-sentence scope: The paper defines a new task called Situation Graph Prediction (SGP), builds a small privacy-preserving synthetic dataset to instantiate it, and runs a diagnostic study across three frontier foundation models to see whether they can infer a user's latent internal state from observable digital artifacts.

What This Paper Is About

Current personal AI agents reason about the world but not from the standpoint of a specific person, because classical user modeling represents people as static preference vectors rather than as evolving internal states such as goals, emotions, and context. The core problem is a data bottleneck: longitudinal digital footprints are siloed and privacy-sensitive, and the internal perspective variables that drive behavior are rarely labeled, so there is no grounded supervision for teaching or evaluating this kind of inference. The paper's goal is to formalize that inference as a structured task, and to show it can be studied without real user data by generating labeled examples synthetically.

Key Contributions

1. Formalization of Situation Graph Prediction (SGP): The authors define a structured inverse inference task that maps observable user data artifacts to ontology-aligned perspective representations, intended to serve as structured, queryable memory units for personal agents. The representation is grounded in the DOLCE Ultralite (DUL) upper ontology, with a schema of 11 node kinds and 14 edge types organized into four semantic strata: participants, spatio-temporal structure, contextual atmosphere, and psychological state.

2. A structure-first synthetic generation strategy: Rather than prompting a language model to freely simulate users, the pipeline inverts the generative process — it samples a valid Situation Graph from the ontology first, then prompts an LLM to render observable artifacts (emails, chat logs, calendar entries) consistent with that structure. This aligns latent labels and observable traces by design and is presented as a privacy-preserving alternative to real labeled perspective data.

3. A pilot dataset: 75 situation instances with 225 unique synthetic artifacts, reviewed by a human expert, centered on a single fictional persona and spanning 75 temporally ordered events from 2021–2025 (approximately 60 months).

4. A diagnostic study across three frontier foundation models: OpenAI GPT-4o, Google Gemini 2.5 Flash, and Anthropic Claude Sonnet 4, evaluated under both zero-shot schema alignment and retrieval-augmented in-context learning (RA-ICL), providing evidence that latent-state inference is consistently harder than surface extraction. Code, data, and supplementary resources are released at https://github.com/flybits/sg-prediction.

Main Findings

• SGP is non-trivial for all three models. Under zero-shot conditions, Strict F1 was 0.016 (± 0.015) for GPT-4o, 0.020 (± 0.014) for Gemini 2.5 Flash, and 0.028 (± 0.019) for Claude Sonnet 4. Soft F1 in the same condition was 0.145 (± 0.072), 0.174 (± 0.046), and 0.291 (± 0.111) respectively.

• Retrieval-augmented in-context learning substantially improves performance. Soft F1 gains from zero-shot to RA-ICL were +0.28 for GPT-4o, +0.31 for Gemini, and +0.33 for Claude. Under RA-ICL, Strict F1 rose to 0.163 (± 0.081) for GPT-4o, 0.221 (± 0.110) for Gemini, and 0.323 (± 0.124) for Claude, while Soft F1 rose to 0.424 (± 0.097), 0.486 (± 0.137), and 0.623 (± 0.118).

• Latent perspective inference is consistently harder than surface extraction. The entropy-normalized latent–surface gap was positive in every condition and grew under RA-ICL: +0.528 (± 0.359) for GPT-4o, +0.668 (± 0.303) for Gemini, and +0.704 (± 0.385) for Claude. In the zero-shot condition the corresponding gaps were +0.139 (± 0.134), +0.210 (± 0.207), and +0.396 (± 0.251). The conclusion restates these RA-ICL gaps as +0.53, +0.67, and +0.70.

• Cross-model consistency suggests the gap is not one model's idiosyncrasy. Because the three models come from different organizations with different pre-training corpora and multimodal stacks, the authors argue the direction of the gap is unlikely to reflect a single model's quirks.

• Entropy normalization widens the gap rather than closing it. The latent vocabulary is much smaller than the surface vocabulary (8 vs. 106 values in the schema; 7 vs. 101 observed in the pilot), which should in principle make latent matching easier. After scaling each category's F1 by its vocabulary entropy ratio, the latent–surface gap widens relative to the raw soft-F1 gap — indicating raw parity is misleading.

• The models occupy distinct behavioral operating points. Claude Sonnet 4 achieved the highest Strict F1 (0.323), the highest Soft F1 (0.623), and the largest normalized gap (+0.704) under RA-ICL, suggesting greater base capability disproportionately benefits surface extraction. Gemini 2.5 Flash showed markedly higher recall than precision (Soft R/P = 0.56/0.44) with a near-zero Predicate Violation Rate (PVR = 0.002), an over-generation tendency tempered by strict ontology compliance. GPT-4o sat between them with balanced precision and recall (Soft R/P = 0.42/0.44) and a moderate violation rate (PVR = 0.065).

• Only static SGP was evaluated. The paper reports Task 1 (Static Situation Graph Prediction) exclusively, deferring Task 2 (Temporal Situation Graph Prediction) because reliable single-timestep grounding is treated as a prerequisite for studying temporal error accumulation.

Methodology in Plain English

The researchers started from the observation that you cannot label a person's inner state directly, and you cannot collect real-world data at scale because it is private. So they flipped the usual pipeline around.

First, they define what a "situation graph" looks like using an established upper ontology (DOLCE Ultralite) plus a domain-specific schema. A situation graph is just a set of triplets — subject, predicate, object — where each node has a kind from a fixed vocabulary and a name from a type-specific list. The schema deliberately separates surface attributes (participants, location types, times, ambience) from latent ones (predicates like feels, evokes, has_valence, conveys_val that point at Emotion and Valence nodes). Typed constraints enforce which subject-kind/object-kind pairs are legal for a given predicate, plus bounded graph size and minimal completeness conditions.

Second, instead of collecting real traces, they generate the graph first and then ask a language model to render matching artifacts — for example, given a graph encoding a job interview situation with a stressed emotion, produce emails, chat logs, and calendar entries consistent with it. This "structure-first" inversion means the label and the evidence are aligned by construction rather than annotated after the fact.

Third, they built a pilot dataset around one fictional persona, Elise Navarro, a 28-year-old Filipino professional living in Toronto and working as a Senior Marketing Analyst. The 75 instances span four domains (professional developments, personal and lifestyle changes, health and physical milestones, social and relational experiences), are primarily text with a subset including images and audio, and were reviewed by a human expert.

Fourth, they ran a three-stage evaluation pipeline. Raw artifacts are first decomposed into a unified text form: text is processed directly, images are converted into descriptive tags and scene summaries via a Vision-Language Model, and audio is turned into exact transcripts plus paralinguistic descriptors such as voice_tremor and loudness. The models then produce triplets under either zero-shot schema alignment (full ontology definitions, no examples) or RA-ICL (the top-3 semantically similar training pairs retrieved with text-embedding-3-large and supplied as demonstrations). Everything uses an 80/20 stratified split with 5-fold cross-validation (60 retrieval, 15 test), identical prompts, retrieval indices, splits, and seeds across models.

Fifth, scoring uses four metrics: Predicate Violation Rate for ontological compliance; Strict F1 via exact string matching; Soft F1 via embedding similarity with text-embedding-3-large at a threshold of 0.8, split into latent and surface subsets; and the latent–surface gap. Because latent and surface vocabularies differ so much in size, they also report an entropy-normalized F1 that scales each category's score by its vocabulary entropy ratio relative to a geometric-mean baseline. Each reported number is a 5-fold cross-validation mean averaged over 3 independent runs, with the ± figure being the mean within-run across-fold standard deviation.

Why This Matters

Impact on research. The paper reframes personalization as an inverse inference problem over latent perspective states rather than preference fitting, and it supplies a concrete task definition, a schema, a dataset, and a baseline comparison across model families. It also offers a methodological argument — structure-first synthesis — for how to study perspective modeling when real labels are ethically and practically unavailable. The consistent positive gap across three independently built models gives the community a reproducible diagnostic signal rather than a single-model anecdote.

Real-world applications:

• Adaptive education — adapting to a learner's evolving goals, confusion, and engagement rather than a static skill profile.
• Health support — recognizing deterioration signals, as in the paper's motivating example of an agent detecting crisis from voice tremors, sparse replies, and avoidance patterns when engagement metrics alone only show a decline.
• Explainable decision-making — grounding adaptation decisions in an inspectable, queryable structured memory rather than an opaque preference vector.
• Bias auditing — since the representation is structured and ontology-aligned, inferred states can be inspected and audited, which the authors explicitly name as a supported application.

Industry relevance. The work targets the gap between general-purpose foundation models and genuinely personal agents in agentic settings. For companies building personal assistants, coaching, CRM, or wellbeing products, it offers a design pattern for long-horizon, ontology-aligned memory that is auditable by construction. The synthetic-data approach is also relevant for organizations that cannot legally or ethically train on real user footprints, and the code and data are openly released under CC BY 4.0. The paper was accepted to PILA 2026: the Workshop on Personal Intelligence in the Agentic AI Era at KDD 2026.

Future Directions

• Scaling the pipeline. The authors state directly that the pilot dataset provides evidence for structure-first generation rather than supporting large-scale training, and that scaling the pipeline is a key next step. They also note that the single-persona design ensures longitudinal coherence but that multi-persona scaling follows the same methodology without changing the task formulation.

• Temporal modeling (Task 2). Long-horizon reasoning is defined but not evaluated. The two proposed modes — single-step transition with oracle history, and multi-step autoregressive trajectory — remain open, with the second specifically designed to test robustness to error accumulation.

• Broader model coverage. The study evaluates three frontier closed-weight foundation models. The authors call for evaluation across open-weight models, specialized multimodal architectures, and fine-tuned regimes to disentangle task difficulty from model-specific behavior, and they leave fully supervised training to future large-scale settings. They also note the formulation naturally supports unsupervised and semi-supervised paradigms, such as treating G as latent in a VAE framework.

• Closing the simulation-to-reality gap, and expanding the ontology. Because synthetic artifacts lack some of the noise and irregularity of real traces, the authors treat their setting as a lower bound on difficulty and expect the gap may widen on noisier real data. Separately, the pipeline is limited by the expressivity of the ontology — phenomena not represented in the schema, such as culturally specific norms, cannot be captured, and any ontology for human perspective is inherently normative and requires interdisciplinary scrutiny.

Target Audience

Researchers and practitioners working on personalization, user modeling, agentic AI, and long-horizon memory for personal agents will get the most from this paper, along with HCI researchers interested in perspective-aware systems and knowledge-representation researchers interested in ontology-grounded multimodal inference. It is also relevant to applied teams building personal assistants, coaching, education, or health-support products who need a privacy-preserving path to labeled perspective data. Because the formal notation and ontology machinery are non-trivial, some familiarity with knowledge graphs and latent-variable formulations helps; readers seeking implementation detail should note that full node and edge taxonomies, per-model handlers, configuration details, and per-model precision/recall breakdowns are placed in the supplementary materials rather than the main text.

Authors’ abstract

Perspective-aware AI requires modeling evolving internal states---goals, emotions, contexts---not merely preferences. Progress is limited by a data bottleneck: digital footprints are privacy-sensitive and perspective states are rarely labeled. We propose Situation Graph Prediction (SGP), a task that frames user perspective modeling as an inverse inference problem: reconstructing structured, ontology-aligned representations of perspective from observable multimodal artifacts, suitable as long-horizon memory for personal agents. To enable grounding without real labels, we use a structure-first synthetic generation strategy that aligns latent labels and observable traces by design. As a pilot, we construct a dataset and run a diagnostic study using retrieval-augmented in-context learning as a proxy for supervision. In our diagnostic study across three frontier foundation models (GPT-4o, Gemini 2.5 Flash, Claude Sonnet 4), we observe a consistent positive gap between surface-level extraction and latent perspective inference---indicating that latent-state inference is consistently harder than surface extraction under our controlled setting. Results suggest SGP is non-trivial and provide evidence for the structure-first data synthesis strategy. For reproducibility and transparency, our code, data, and supplementary resources are available here: https://github.com/flybits/sg-prediction.