Human-Inspired Social Engagement Analysis via Interpretable Mutual Visual Attention

AI summary

Overview

• Research area: Computer vision for non-verbal social interaction analysis; head pose estimation, dyadic visual attention, and group engagement measurement.
• Technical level: Intermediate. The pipeline is modular and geometric rather than a black-box deep model, but understanding it requires familiarity with head pose angles, F1 scoring, and interaction-state classification.
• Scope: The paper proposes a four-stage, training-free framework that infers individual and group social engagement from head-direction cues in small groups of people, evaluated quantitatively on annotated video benchmarks and qualitatively through an interpretable visualization tool.

What This Paper Is About

Most computational work on social behaviour either predicts a task such as head pose estimation or trains an end-to-end model that outputs an engagement score without exposing the reasoning behind it. This paper instead builds a transparent, step-by-step pipeline in which head direction is estimated for each person, mutual visual attention between pairs is classified, and those pairwise events are aggregated into interpretable engagement indices. The goal is to give psychologists, educators, and caregivers a tool that explains why a group looks engaged, rather than only reporting a number.

Key Contributions

1. A modular, video-based framework for hierarchical social interaction analysis that moves from pairwise attention estimation up to group-level engagement, explicitly modelling each reasoning stage instead of learning interactions end-to-end.
2. Two purpose-designed engagement indices: the Individual Engagement Index (IEI), which quantifies one person's participation across interaction states, and the Group Engagement Score (GES), which quantifies group-level interaction against a maximum possible value.
3. An intuitive visualization interface that presents head directions, interaction edges, temporal interaction bars with partner IDs and durations, frame-level interaction counts, and per-person IEI, making multi-level analysis accessible to non-technical users.
4. A live demonstration in a kindergarten, where the tool was used to visually describe children's engagement during a cooperative activity and teachers gave positive qualitative feedback on interpretability and usefulness; the recorded data cannot be shared due to ethical constraints.

Main Findings

• Better interaction classification than the baseline: On GP-Static++ (13,789 frames), the proposed method achieved F1 of 0.70 for bidirectional events, 0.45 for unidirectional events, 0.60 for no interaction, and an average F1 of 0.58, compared with 0.56, 0.44, 0.49, and 0.49 for the frame-based method of [28]. The paper reports superior performance in all scenarios and notes the framework is training-free and uses no task-specific training data.
• Group size affects performance: On the annotated CMU Panoptic Haggling sequences (2,368 frames, three people), the method reached F1 of 0.68 for bidirectional, 0.29 for unidirectional, 0.54 for no interaction, and 0.50 average. The paper attributes the degradation in unidirectional and no-interaction classification to group cardinality, while mutual events show only a small drop, which the authors link to the stronger geometric constraints on bidirectional cases.
• GES tracks interaction scenarios: Reported GES values were 40.9 for a Pexels three-person sequence in which two children interact only during the first 1.3 seconds while a woman observes, 68.9 for a CMU three-peer conversation with turn-taking and more fragmented interaction, 31.8 for a teacher instructing students who are each occupied with their own activity, 68.7 for a sequence similar to the CMU example, and 80.6 for a family engaged in a cooperative activity.
• Engagement weights reflect reciprocity: In IEI the weights are set empirically to 1.00 for bidirectional, 0.30 for unidirectional looker, and 0.05 for unidirectional target, encoding the assumption that mutual events indicate the highest engagement and that a looker is more engaged than a target.
• Qualitative coherence: The visualization outputs are reported as coherent with the underlying situation; frame-level interaction counts confirmed that non-interaction dominated for the children in the Pexels example while the woman mainly observed the child with ID 3.

Methodology in Plain English

The framework runs in four stages, and each stage produces an explicit intermediate representation.

• Stage 1 — find people and faces: Each person is detected and tracked with YOLOv11 (the yolo11n-pose variant) combined with ByteTrack, and assigned a unique ID. Only five facial keypoints are retained: nose, left eye, right eye, left ear, right ear. Each keypoint carries its own confidence, and missing points are set to (0, 0) with zero confidence.
• Stage 2 — estimate head direction: The five 2D keypoints are normalised so the representation is invariant to scale and absolute face position, using the keypoint centroid and the maximum horizontal and vertical distances from it. The normalised keypoints go to HHP-Net, which predicts a head orientation triplet (yaw, pitch, roll) together with uncertainties for each angle. The angles are converted into a 2D unit vector on the image plane using Tait-Bryan angles, with the keypoint centroid as the vector's origin.
• Stage 3 — classify interactions: For every pair of people, the unit vector connecting their head origins and the angle between each person's head direction and the direction to the other person are computed. If both angles are below a threshold τ, the pair is bidirectional; if only one is, it is unidirectional; otherwise it is non-interacting. Because pairwise decisions can be inconsistent, the method resolves states hierarchically: bidirectional pairs are assigned first, then unidirectional pairs (the person below τ is the looker, the other the target), and any remaining people are marked non-interacting. τ was empirically set to ±15 degrees.
• Stage 4 — compute engagement: IEI for a person is 100/N times the weighted sum of frames spent in bidirectional, unidirectional-looker, and unidirectional-target states, where N is the number of frames in which that person is detected; it ranges from 0 to 100. GES is 100 times the predicted engagement divided by the maximum possible engagement over the video, where the maximum per frame is ⌊n/2⌋ bidirectional states and n%2 unidirectional states, weighted by 1 for bidirectional and 0.5 for unidirectional.

Implementation used detection and IoU thresholds of 0.5 for YOLO, tracker match and track thresholds of 0.8 and 0.5, and an official TensorFlow implementation of HHP-Net with pretrained checkpoints and no additional fine-tuning. All experiments ran on an ASUS TUF Gaming F15 laptop with an Intel Core i7 processor and 16 GB RAM.

Why This Matters

• Research impact: The work argues for hierarchical, explicit modelling of social cues as an alternative to end-to-end prediction, and shows that a training-free approach can outperform a frame-based baseline on a dyadic interaction benchmark. It contributes two named indices (IEI and GES) as measurable, interpretable quantities for group engagement.
• Real-world applications:
  • Classroom observation, supporting teachers in seeing which students are participating and who is disengaged during group work.
  • Caregiving and social work settings, where understanding group interaction dynamics matters but technical expertise is limited.
  • Child development and cooperative activity monitoring, as illustrated by the kindergarten demonstration.
  • Behaviour analysis and activity monitoring in general, since the framework targets small- and medium-sized groups in indoor and outdoor settings across ages.
• Industry relevance: Interpretable, modular pipelines are attractive where explanations must be auditable or communicated to non-technical stakeholders. The reported hardware requirements (a consumer laptop with an Intel Core i7 and 16 GB RAM) suggest the approach is intended to be deployable without large-scale training infrastructure.

Future Directions

• Incorporating 3D human representations to address the stated limitation that 2D pose estimation can introduce ambiguities when three-dimensional scenes are projected onto the image plane.
• Adding multimodal cues and speaker attribution, which the authors list as extensions beyond the current visual-only, dyadic focus.
• Modelling interaction patterns beyond dyadic relationships, since pairwise events are currently the only building block for group-level indices.
• Performing a quantitative evaluation of the engagement indices themselves, which the authors note is still a future objective.

Target Audience

Researchers in computer vision and social signal processing who are interested in interpretable, hierarchical models of group behaviour; human-computer interaction and learning-analytics practitioners; and psychologists, educators, teachers, caregivers, and social workers who need tools to observe and understand small-group interaction dynamics without technical expertise.

Authors’ abstract

Understanding social interactions from non-verbal visual data is important for behavior analysis and activity monitoring. We propose an interpretable computational model of social engagement inspired by psychological theories of mutual visual attention. Rather than learning interaction patterns end-to-end, our framework explicitly models dyadic visual attention and aggregates these cues into interpretable measures of individual and group engagement. The resulting modular framework combines state-of-the-art head orientation estimation with lightweight geometric reasoning, producing explanations that remain accessible to non-technical users. We evaluate the proposed approach on a variety of data through quantitative experiments and demonstrate its practical usefulness with qualitative visualizations designed to support teachers, caregivers, and social workers in understanding group interaction dynamics.