ManzaiSet: A Multimodal Dataset of Viewer Responses to Japanese Manzai Comedy

AI summary

Overview

Research area: Affective computing / computer vision / multimodal emotion datasets, with a focus on cross-cultural humor perception and Japanese manzai comedy.

Technical level: Intermediate — the dataset construction and motivation are accessible to newcomers, while the statistical analyses (k-means clustering, Brown–Forsythe tests, GEE marginal models, permutation tests) assume familiarity with quantitative methods.

Scope: The paper introduces ManzaiSet, a multimodal dataset of viewer responses to Japanese manzai comedy, and demonstrates its utility through three baseline analyses of viewer typology, temporal dynamics, and humor-type effectiveness.

What This Paper Is About

Affective computing research is dominated by Western datasets, and models trained on them often fail to generalize to non-Western emotional expressions. Humor is a particularly culture-dependent emotion, and no large-scale dataset has previously captured how individual viewers respond to culturally specific comedy. The authors build ManzaiSet to fill this gap, recording synchronized facial video and audio from Japanese viewers watching the same professional manzai performances, and use it to establish baseline findings about individual differences, temporal dynamics, and humor mechanisms.

Key Contributions

1. The first large-scale multimodal dataset of viewer responses to culturally specific comedy. 241 Japanese participants (228 used in analyses) watched up to 10 professional manzai performances in randomized order, producing 2,301 viewing sessions and 191.8 hours of synchronized facial video (640×360, 25 fps) and audio (48 kHz).

2. A viewer typology with three distinct appreciation profiles. K-means clustering identified "High & Stable Appreciators" (72.8%), "Low & Variable Decliners" (13.2%), and "Variable Improvers" (14.0%), with statistically significant heterogeneity of variance across clusters.

3. Evidence of a positive viewing-order effect. Individual-level slopes show ratings rising across the session, contradicting standard fatigue or habituation assumptions in comedy consumption.

4. A test of whether humor categories differ in effectiveness. Automatic humor classification alongside viewer response modeling found no reliable differences across nine manzai humor types after FDR correction, suggesting execution quality may matter more than categorical mechanism.

Main Findings

• Three viewer types with distinct rating behavior. High & Stable Appreciators (n=166) rated consistently high (mean ≈87.0) with minimal within-person variability (SD ≈4.84). Variable Improvers (n=32) rated moderately (mean ≈71.2) but improved across the session. Low & Variable Decliners (n=30) rated lower (mean ≈65.6) and declined across the session. Cluster stability was high (bootstrapped ARI = 0.865).

• Variance heterogeneity is statistically significant. Brown–Forsythe tests confirmed that the clusters differ in within-person variability for rating SD (p = 1.74 × 10⁻¹⁰), coefficient of variation (p = 2.15 × 10⁻¹³), and rating range (p = 3.65 × 10⁻¹⁰).

• Humor appreciation style is trait-like. Split-half reliability of per-participant mean ratings across randomized halves was r = 0.866 (p < 0.001), indicating that individual appreciation patterns are stable rather than state-dependent.

• Viewers enjoy comedy more over time, not less. Mean ratings rose from 77.95 at position 1 to 83.15 at position 10, a 6.7% gain. The mean individual slope was 0.488 rating points per position (t(227) = 5.42, p < 0.001; permutation p < 0.001), corroborated by a linear mixed-effects model (β ≈ 0.49, 95% CI [0.31, 0.66]).

• Minimal carry-over effects. After detrending, lag-1 autocorrelations within participants averaged r = −0.070 (95% CI [−0.108, −0.032]), indicating negligible sequential dependence.

• No humor type dominates. Across 77 humor instances and 131 labels (multi-label), no category showed a statistically reliable advantage after FDR correction. All odds ratios were below 1.5, and unequal category sizes (e.g., n=5 for physical comedy) limit power to detect smaller effects.

Methodology in Plain English

The researchers built a browser-based system that participants used at home. Each person logged in, answered a short demographic questionnaire, and then watched up to 10 manzai performances in a randomized order. While watching, the system recorded their facial expressions and audio. After each video, they gave a 0–100 enjoyment rating. Real-time feedback helped participants position their camera and check audio quality.

Ten performances were selected from a 2022 Yoshimoto Kogyo anniversary event, chosen for video quality, lack of copyrighted background music, and variety of humor styles. Each clip ran 4–6 minutes.

For analysis, facial data was processed with OpenFace 2.0 to extract action units and head pose. The researchers then ran three studies. First, they computed per-viewer features (mean rating, variability, range, order slope) and applied k-means clustering to group viewers by how they rated. Second, they regressed each viewer's ratings on viewing order to measure change over the session, confirmed with permutation tests and a mixed-effects model. Third, they transcribed the videos with Whisper, used GPT-5-mini to label humor types, detected laughter from the audio channel, and modeled whether a viewer laughed at each humor instance using generalized estimating equations with participant clustering and video fixed effects.

Why This Matters

Impact on research. ManzaiSet directly counters the Western-centric bias in affective computing by providing a large, controlled, non-Western corpus of spontaneous emotional responses. Because it pairs standardized stimuli with individual-level responses, it supports both causal stimulus–response analysis (timing events to specific boke–tsukkomi cycles) and within-subject comparison — a capability missing from both small controlled datasets like DEAP and large in-the-wild datasets like AffectNet. The finding that roughly 27% of viewers respond in ways qualitatively different from the majority challenges models trained on averaged responses.

Real-world applications:

• Personalized entertainment and recommendation. Few-shot conditioning on a viewer's recent ratings and facial cues could outperform global models, and engagement-aware sequencing could exploit the observed positive momentum rather than assuming fatigue.
• Cross-cultural emotion recognition. The corpus offers a non-Western testbed for calibrating and stress-testing models trained on Western data, supporting domain adaptation and generalization-gap measurement.
• Comedy performance training and editing. Response-aligned diagnostics — laughter onset rates and latency, AU dynamics, vocal bursts — could give performers and editors interpretable feedback on pacing and delivery.
• Laugh-aware interactive systems. Agents that detect and anticipate laughable moments, modulate tone, and schedule content could be built on top of this data.

Industry relevance. Streaming platforms, comedy production companies, and developers of emotion-aware interfaces all stand to benefit. The collaboration with Yoshimoto Kogyo, a major Japanese entertainment agency, shows how professional content libraries can be paired with audience measurement to inform both algorithmic personalization and content creation.

Future Directions

• Extend beyond Japanese viewers. The dataset currently contains only Japanese participants, so realizing cross-cultural recognition requires adding other cultural groups or combining with non-Japanese corpora and evaluating domain adaptation explicitly.

• Probe practical equivalence rather than absence of effect. The "no difference" result across humor types reflects a lack of statistical significance, not proven equivalence. Equivalence tests (TOST) with larger, more balanced samples could clarify whether small effects exist.

• Finer temporal and micro-response modeling. Finer localization of laughter, modeling of within-session contagion and carry-over, and moment-to-moment anticipatory engagement scoring remain open.

• Closed-loop interaction studies. The authors propose consented studies where interactive systems adapt in real time to detected amusement, with privacy, safety, and cultural appropriateness treated as first-class constraints.

Target Audience

This paper is most useful for affective computing and multimodal machine learning researchers, particularly those working on emotion recognition, cross-cultural generalization, or dataset construction. It also speaks to computational humor researchers who want audience-level rather than content-level signals, and to HCI and entertainment-technology practitioners building personalization or engagement-aware systems. Psychologists and media researchers studying individual differences in humor appreciation and cultural display rules will find the viewer typology and temporal findings relevant. The paper is approachable for graduate students new to affective computing, though the statistical appendix assumes comfort with clustering, mixed-effects models, and multiple-comparison correction.

Authors’ abstract

We present ManzaiSet, the first large scale multimodal dataset of viewer responses to Japanese manzai comedy, capturing facial videos and audio from 241 participants watching up to 10 professional performances in randomized order (94.6 percent watched &gt;= 8; analyses focus on n=228). This addresses the Western centric bias in affective computing. Three key findings emerge: (1) k means clustering identified three distinct viewer types: High and Stable Appreciators (72.8 percent, n=166), Low and Variable Decliners (13.2 percent, n=30), and Variable Improvers (14.0 percent, n=32), with heterogeneity of variance (Brown Forsythe p &lt; 0.001); (2) individual level analysis revealed a positive viewing order effect (mean slope = 0.488, t(227) = 5.42, p &lt; 0.001, permutation p &lt; 0.001), contradicting fatigue hypotheses; (3) automated humor classification (77 instances, 131 labels) plus viewer level response modeling found no type wise differences after FDR correction. The dataset enables culturally aware emotion AI development and personalized entertainment systems tailored to non Western contexts.