Ordered Network Analysis of Epistemic Emotions during Collaborative Problem Solving

AI summary

Overview

Research area: Affective computing and learning analytics, specifically the study of epistemic emotions (curiosity, confusion, frustration, and related states) during in-person collaborative problem solving (CPS), with an emphasis on educational AI systems that can perceive learner affect.

Technical level: Intermediate. The paper is readable without deep mathematics, but it assumes familiarity with network-based learning analytics, particularly epistemic network analysis (ENA) and its ordered variant (ONA), and with concepts such as moving windows, difference networks, and dimensional projection.

Scope: A small-scale, exploratory study that applies ordered network analysis to retrospective cued-recall affect reports from 27 participants in 9 collaborative groups, comparing self-caught versus probe-caught reporting and faster versus slower groups.

What This Paper Is About

Affective states such as confusion and frustration are central to learning, but they cannot be observed directly, and no measurement method is treated as ground truth. The researchers asked participants to watch recordings of their own collaborative sessions and report their emotions, then used ordered network analysis to see which emotions tended to follow which others over time. The goal was to test whether the order and persistence of these states carry useful information that simple frequency summaries miss, and whether those patterns differ by how reports were collected and by how long groups took to finish the task.

Key Contributions

1. First application of ordered network analysis to retrospective cued-recall data in situated CPS. The authors position this as an initial baseline for how specific affective states organize over time in co-situated collaborative problem solving, where datasets are typically small and temporally coarse.

2. Demonstration that ONA surfaces persistence and transition patterns invisible in descriptive summaries. The paper argues that traditional analyses emphasizing frequencies or aggregate transition metrics treat affective states as independent and obscure how they co-occur locally.

3. A comparison of two affect reporting mechanisms within the same task. By building separate ONA models for self-caught and probe-caught reports, the study shows that how a report is elicited shapes the apparent structure of affect, with probe-caught reports producing stronger self-loops among core states and self-caught reports producing more distributed transitions involving higher-arousal states.

4. A task-duration contrast using a difference network. A median split divides groups into shorter- and longer-duration conditions, revealing that slower groups show stronger coupling among confusion, conflict, and frustration, while faster groups show stronger transitions among curiosity and optimism.

Main Findings

• A stable epistemic core. Across ONA models, curiosity, optimism, and confusion were consistently involved in the strongest ordered connections, including self-transitions, indicating short-range persistence rather than frequent switching. These three were also the most frequently reported states overall, followed by surprised, disengaged, frustrated, and conflicted. All seven states appeared across groups.

• Interpretation of the projected dimensions. In the full-data model, the horizontal dimension separates states associated with forward progress (curiosity, optimism, surprise) from states associated with difficulty and struggle (confusion, conflict, frustration). The vertical dimension was less readily interpretable as a single construct; a distinction between struggle and disengagement appeared to be emerging, with disengagement in the highest location, but its proximity to curiosity made this harder to interpret.

• Reporting mechanism differences in distribution. Self-caught reports were more evenly distributed across the seven states, while probe-caught reports were more concentrated among a smaller subset, even though both showed similar overall profiles with curiosity, optimism, and confused most prevalent. The split was 64% probe-caught versus 36% self-caught.

• Faster versus slower groups. Blue edges (stronger in faster groups) showed stronger transitions among curious and optimistic, with faster groups occasionally transitioning from optimistic to disengaged (-0.05). Red edges (stronger in slower groups) showed stronger bidirectional transitions among confused, conflicted, and frustrated; slower groups were much more likely to transition from confused to conflicted (0.11) than faster groups and less likely to transition from confused to disengaged (-0.05).

• Q&A on state positions. Qualitative review of task recordings indicated that states toward the right of the space (optimism, surprise, confusion) tended to occur when expectations or the problem state changed, while states on the left (disengagement, conflict, frustration) occurred around a broader range of experiences including apparent boredom, tension, and disagreement.

• Self-caught versus probe-caught network shapes. Both models retained strong bidirectional transitions and self-loops among curious, optimistic, and confused, but diverged beyond that. The self-caught model appeared rotated but preserved the distinction between conflicted and other states; the curious–confused relationship appeared inverted relative to the full data, with confusion now closest to disengagement. In the probe-caught model, optimistic was the state most separated from the group, appearing at the far left, while confused and curious defined the major distinctions of the vertical axis.

• Persistence versus switching by reporting method. Probe-caught reports exhibited stronger self-loops among the core epistemic states, suggesting continued sampling of an ongoing state and more sustained epistemic engagement. Self-caught reports showed more distributed transitions involving higher-arousal states such as frustration, surprise, and conflict, suggesting these reports occur at moments of perceived affective change. The authors note the self-caught model more closely mirrors the full dataset despite probe-caught reports being considerably more frequent, and caution that each model necessarily omits transitions.

• Model fit statistics. The main model showed correlations of 1 (Pearson) and 1 (Spearman) for the first dimension, and co-registration correlations of 0.98 (Pearson) and 0.97 (Spearman) for the second. The first two dimensions explained 34.3% and 21.3% of variance in the full-data and difference models, 29.2% and 16.5% for the self-caught model, and 31.1% and 24.7% for the probe-caught model.

• Task duration is not treated as performance. All groups arrived at the correct final solution, so task accuracy was not used in the analysis. The authors describe task duration as a descriptive property of collaboration rather than a proxy for group performance.

Methodology in Plain English

The researchers recruited 27 adults (over 18, fluent in English, compensated USD 15 with IRB approval) and placed them into 9 groups of three. Each group worked on the first phase of the Weights Task, a balance-scale activity adapted from prior work in which groups infer the relative weights of five blocks, one of which has a known reference weight, and submit a single consensus response. The sessions were video recorded.

Afterwards, each participant individually rewatched the recording and reported their own affective states using an interactive survey tool, selecting one or more states from a set of seven (Confused, Curious, Frustrated, Disengaged, Optimistic, Surprised, Conflicted) adopted from previous research. Reports could be self-initiated at any point, or triggered by a probe after 60 seconds of inactivity; all reports were timestamped, and each was classified as self-caught or probe-caught.

Rather than counting how often each emotion appeared, the team applied ordered network analysis using the web-based ENA tool. Each participant's sequence of responses was treated as a conversation, groups were the unit of analysis, and a moving window of 2 events captured short-range persistence and transitions. When multiple states were selected in the same survey response, they were ordered by a fixed label sequence, a choice the authors note introduces arbitrary ordering but is unlikely to matter much because most ordered connections arise across successive responses. A sensitivity analysis using window sizes of 2, 3, and 4 produced consistent structures with no substantial changes in node positioning or dominant connection strengths.

The analysis produces directed network graphs where nodes are affective states, node size reflects self-transition strength, and edges reflect directed connection strength. Networks were mean-centered and subjected to singular value decomposition to produce dimensions for visualization. Two models were built: one comparing self-caught versus probe-caught reports (built separately rather than as a difference network because of class imbalance), and one difference model comparing faster and slower groups based on a median split of task duration, with node positions fixed and edges color-coded.

Why This Matters

Impact on research. The paper argues that defaulting to descriptive summaries implicitly treats affective states as independent and discards sequential information, even when reports have limited temporal granularity and scale. It offers ONA as a way to extract ordered structure from small, temporally coarse educational datasets, and it complicates common assumptions about what "successful" collaboration looks like: faster completion may reflect smoother coordination with less conflict-driven affect, but it may equally reflect group dynamics in which one or two members drive the task while others are more reserved. The finding that reporting mechanism shapes the apparent network structure is also a caution for anyone comparing affect data collected under different protocols.

Real-world applications.

• Affect-aware collaborative learning systems: AI tutors and group facilitators could be designed to distinguish productive struggle, which may warrant silence, from struggle-free disengagement, which may require intervention, by monitoring local trajectories of confusion and disengagement over time rather than reacting to isolated signals or summary statistics.

• Classroom and small-group monitoring: Teachers and facilitators could use ordered transition patterns, not just emotion frequencies, to identify groups whose coordination is becoming conflict-driven.

• Learning analytics tooling for collaboration: The comparison of self-caught and probe-caught reporting offers practical guidance for designing affect surveys and deciding when to prompt learners.

• Evaluation of collaborative learning design: The duration-based contrast suggests that pacing and team management are distinct dimensions worth measuring separately from solution correctness.

Industry relevance. The work sits directly in the educational technology and AI in education space, where adaptive systems need defensible ways to interpret learner affect from sparse, noisy, and self-reported signals. It is relevant to companies building collaborative learning platforms, intelligent tutoring systems, and classroom analytics dashboards. The funding acknowledgements, a DARPA FACT program award (HR00112490377) and an NSF subcontract under award DRL 2454151 for the Institute for Student-AI Teaming, indicate interest from both defense-adjacent conversational AI and educational AI research programs in understanding affect during team interaction.

Future Directions

1. Replication with larger, more diverse samples. The dataset of 27 participants had a proportionately large male population and a fixed set of seven predefined labels, which the authors identify as limiting generalizability. They call for larger, more diverse samples and a broader range of affective categories.

2. Triangulating self-reports with other signals. Because retrospective cued-recall is susceptible to recall bias and temporal imprecision, and because ground truth for affective states remains an unresolved problem, the authors suggest combining self-reports with physiological or behavioral signals.

3. Linking affective structure to outcomes. The study included no direct measures of learning or performance beyond task completion time, since all groups solved the task correctly. Future work should examine how ONA-derived representations of affective organization relate to learning outcomes, collaboration quality, and solution quality.

4. Testing adaptive interventions. The authors propose investigating whether interventions built on ordered affect representations, rather than isolated affective signals, actually improve outcomes in affect-aware educational technologies.

Target Audience

Researchers in learning analytics, affective computing, and AI in education who study emotion during collaborative learning and are looking for analysis methods that preserve temporal ordering. It is also useful for graduate students and methodologists interested in ONA as an extension of ENA beyond unordered co-occurrence, and for designers of collaborative learning technologies who need to reason about when confusion signals productive struggle versus disengagement. Readers without background in network-based learning analytics will find the results interpretable but the method descriptions require some familiarity with dimensional projection and difference networks.

Authors’ abstract

Investigating how affective states such as confusion and frustration persist and transition during co-situated collaborative problem solving (CPS) is important for understanding the dynamics of epistemic emotions. However, the accurate identification of affective states remain challenging as there is no gold-standard truth in this space. Here, we analyze affective states collected through retrospective cued-recall during an in-person CPS task. Using ordered network analysis (ONA), we examine (1) the overall ordered structure of affective states and how this structure differs across self-caught and probe-caught reporting methods, and (2) what aspects of this ordered structure are emphasized differently in slower and faster groups. We find that ONA reveals differences in persistence and transition patterns that are not apparent from descriptive summaries alone. In particular, we observe a stable epistemic core linking curiosity, optimism, and confusion, with different reporting methods emphasizing different connections among states. An analysis between faster and slower groups show that roles of confusion and disengagement also shift significantly during collaboration, particularly in their relationship to conflict. We interpret our findings in the context of collaboration and discuss their implications in developing AI systems that support CPS.