VIP-MINGLE: A Corpus for Videoconference and In-Person Multimodal Interaction in Group Language Engagement

AI summary

Overview

Research area: Human-Computer Interaction / multimodal conversation analysis, with ties to computer vision, speech processing, and social psychology.

Technical level: Intermediate. The corpus itself is broadly accessible, but the feature-extraction pipeline and mixed-effects statistical analysis involve domain-specific tooling (diarization, action units, dependency parsing) that assumes some familiarity with multimodal data.

Scope in one sentence: The paper introduces VIP-MINGLE, a 59-hour paired within-subject corpus of 32 groups (105 participants) recorded in both in-person and videoconference settings, and shows that conversational behavior shifts substantially across modalities.

What This Paper Is About

Research on group conversation has largely been split between in-person corpora (such as the AMI and ICSI Meeting Corpora) and videoconferencing corpora (such as the CANDOR and RoomReader corpora), with little data bridging the two. Because the same behavior cannot be compared across different speakers, tasks, and protocols, it has been hard to tell whether videoconferencing genuinely changes how people interact or merely degrades an in-person signal.

The goal of VIP-MINGLE is to provide a controlled bridge: the same groups of participants play an identical collaborative game twice, once face-to-face and once remotely, so that differences can be attributed to the medium rather than to the people or the task.

Key Contributions

1. A paired within-subjects multimodal corpus. 59 hours of recordings from 32 groups and 105 participants, where every group completed both an in-person and a videoconference session, with session order counterbalanced to control for order effects.

2. Layered data at multiple levels of processing. Raw audio and video, participant-level psychometric baselines for personality and mood collected a few days prior, processed multimodal features (diarized speech, Whisper transcripts, OpenFace action units and gaze, DeepFace emotion probabilities), and 7,077 time-resolved human-annotated segments.

3. Empirical evidence of cross-setting domain shift. Exploratory analyses across speech timing, language complexity, facial expression, and human ratings show substantial behavioral distribution shifts between settings, not simple scaling differences.

4. A public release. The scripts and dataset are available at DOI: 10.5281/zenodo.20670131, released under CC BY 4.0, with both raw and processed data so researchers can apply their own models.

Main Findings

• Videoconference speech is slower and more fragmented. Using the Heldner–Edlund model to extract turn boundaries, videoconference sessions showed significantly longer turn-taking gaps (β = 0.113, SE = 0.054, p = .037) and shorter utterance duration (β = -0.094, SE = 0.018, p < .001).

• Videoconference language is syntactically simpler. Mean Dependency Distance (MDD) was significantly higher in-person than in videoconference (p = .017, Wilcoxon signed-rank test). No significant differences appeared for textual lexical diversity, mean token surprisal, or perplexity (p ≥ .410).

• Facial expressions differ qualitatively, not just in magnitude. Wilcoxon signed-rank tests on OpenFace action units showed the majority of AUs were heightened in-person, while a few showed significant gains in videoconference. On DeepFace emotion intensity, happy, surprise, and fear were more salient in-person, while disgust and neutral were more salient in videoconference. The paper notes OpenFace does not support intensity data for AU28.

• In-person sessions were rated more enjoyable. On 7,077 clips rated on a 5-point Likert scale, in-person conversations scored significantly higher in enjoyment (p = .001, sessional-level Wilcoxon signed-rank test), with no significant difference in conversational fluidity (p = .465).

• "Turbulence" correlates with enjoyment. Multi-label event annotations showed interruptions and gaps occurred more frequently in-person, while videoconference sessions were notably more "uneventful." The authors suggest interruptions and gaps are intrinsic elements of natural interaction rather than purely negative artifacts.

• The authors caution against assuming cross-domain generalization. The multifaceted differences lead them to warn that models developed in one setting may not seamlessly transfer to the other, and they link the "cleaner" but less enjoyable remote exchange to the phenomenon of "Zoom fatigue."

Methodology in Plain English

Participants were recruited at New York University and assigned to groups of 2–4. Each group completed two sessions under two hours total: one in person around a shared conference table with roughly 1–3 meters of separation, and one over Zoom with each participant in a separate isolated room. The task was a "Family Feud"-style trivia game, with a researcher acting as the initial host and participants rotating into the host role, to elicit spontaneous turn-taking and collaborative problem-solving in English. Aliases were used throughout.

Both settings were recorded through Zoom (individual tracks, Active Video Only, and Gallery Video Only) using personal laptops and external USB lavalier microphones (MAONO AU-UL10) clipped below the chin. In-person sessions added a 360° camera centered on the table, with laptop screens dimmed and speakers muted to reduce interference; microphone gain was manually lowered during recording to limit audio crosstalk and restored later during signal processing.

Raw recordings were then run through a pipeline. For in-person audio, the pyannote/speaker-diarization pipeline isolated each participant from their own microphone after downsampling to 16 kHz mono WAV, and the output was encoded as mono 32 kHz audio-only MP4 files. Whisper (large) produced transcripts with timestamps that were aligned to diarized speaker labels. For video, individual streams were pulled from Gallery Video Only recordings, and OpenFace extracted 3D head pose, eye-gaze, and action unit intensities and presences at the native frame rate of roughly 25–30 fps (failed detections flagged as missing rather than imputed), while DeepFace produced emotion probability time series synchronized to the same frame index.

Separately, videos were cut into 10-second clips and annotated by 192 qualified annotators on Qualtrics, each viewing 120 clips. Statistical comparisons used linear mixed-effects models for speech timing (to control for within-group dependency, with log-transformed data) and Wilcoxon signed-rank tests for language complexity and facial features.

Why This Matters

Impact on research. The paper argues that isolated in-person and videoconference corpora have produced findings that cannot be compared, because unpaired data confounds the medium with task, protocol, and speaker differences. A paired corpus lets researchers isolate the medium's effect and provides a testbed for measuring domain shift directly. The authors frame VIP-MINGLE as a "controlled, horizontal bridge" connecting research previously siloed in each setting, and as a resource for building domain-aware models.

Real-world applications:

• Videoconferencing system design. The finding that remote sessions are "cleaner" but rated less enjoyable, with fewer interruptions and gaps, points to concrete design targets for platforms trying to reduce Zoom fatigue.
• Hybrid and remote workplaces. Organizations running a mix of in-person and remote collaboration can use the corpus to understand how meeting dynamics differ by medium and where remote formats lose engagement.
• Remote education and telehealth. Any setting where group interaction moves online faces the same shifts in turn-taking pace, syntactic complexity, and facial expressiveness that the corpus documents.
• Automated meeting analytics. Systems that diarize speech, transcribe, and read facial expressions can be tested on data where ground-truth settings are known and paired, revealing whether a model is sensitive to the medium rather than the behavior.

Industry relevance. Companies building videoconferencing platforms, meeting-assistant tools, and emotion- or engagement-sensing products depend on models that generalize across physical and remote settings. This paper's core warning is directly commercial: a model trained on in-person conversational data may not transfer to remote interactions, and the difference is not a matter of signal quality but of a fundamentally restructured interaction pattern.

Future Directions

• Extending beyond the single task. The authors acknowledge that the "Family Feud"-style semi-structured game may not generalize to other conversational formats, and explicitly state that VIP-MINGLE does not aim to maximize task diversity. They propose aligning this corpus with other tasks in each setting to enable broader generalization.

• Domain-aware and domain-adaptive modeling. The observed distribution shifts motivate building models that can adapt robustly across physical and remote group interactions rather than assuming seamless transfer.

• Custom feature extraction. Because both raw and processed data are released, researchers are invited to apply their preferred or future state-of-the-art models rather than the representative subset the authors extracted.

• Connecting the corpus to prior work. The paper positions VIP-MINGLE as being comparable with previous corpora and studies (for example CANDOR, RoomReader, and related modeling work), leaving open the question of how far its findings replicate against those datasets.

Target Audience

This paper is most useful to researchers and practitioners working on multiparty conversation: multimodal machine learning and conversational AI researchers who need paired cross-setting data; HCI and CSCW researchers studying videoconferencing, telepresence, and remote collaboration; speech and computer vision researchers interested in diarization, action units, and emotion recognition in group settings; and social and cognitive scientists studying turn-taking, language complexity, and nonverbal behavior. Product teams building videoconferencing or meeting-analytics systems will find the domain-shift framing and the enjoyment-versus-fluidity result directly relevant.

Authors’ abstract

Group conversations are a fundamental yet complex form of social interaction central to human cognition and telecommunication technology. While understanding and facilitating these interactions has been a long-standing goal, findings are often isolated within specific in-person or videoconferencing settings due to a scarcity of datasets that bridge the two. We introduce VIP-MINGLE, a multimodal dataset comprising 59 hours of recordings (32 groups, 105 participants), featuring paired within-subject sessions in both settings. The dataset includes raw audio/video, psychometric data, processed multimodal features (e.g., diarized speech, facial expressions, transcriptions), and time-resolved human annotations. Our analysis reveals significant behavioral distribution shifts across multiple modalities between settings, reinforcing the need for a cross-setting corpus. VIP-MINGLE serves as a critical resource for developing robust models of group conversations across settings.