RecruitView: A Multimodal Dataset for Predicting Personality and Interview Performance for Human Resources Applications

AI summary

Overview

• Research area: Multimodal machine learning for computer vision, with applications in human resources, affective computing, and personality psychology.
• Technical level: Advanced. The paper assumes familiarity with multimodal encoders (transformers, self-supervised speech and video models) and with Riemannian geometry concepts such as hyperbolic, spherical, and Euclidean manifolds, tangent spaces, and Mixture-of-Experts routing.
• Scope: The paper introduces RecruitView, a 2,011-clip multimodal interview dataset with 12 continuous psychometric targets, and CRMF, a geometry-aware fusion framework evaluated against three large multimodal model baselines on that dataset.

What This Paper Is About

Automated assessment of personality and interview performance from video is limited by two things: datasets that rely on noisy, subjective absolute ratings, and models that flatten all audio, video, and text signals into a single Euclidean space. The authors build a new interview dataset whose labels come from pairwise comparisons made by clinical psychologists, then design a model that processes multimodal features simultaneously in hyperbolic, spherical, and Euclidean geometries to better match the structure of human behavioral traits.

Key Contributions

1. RecruitView dataset: A multimodal interview corpus of 2,011 video clips from 331 distinct interview sessions and more than 300 participants, each responding to one of 76 curated interview questions, with approximately 27,000 pairwise comparative judgments expanded into 12 continuous targets (Big Five traits, Overall Personality, and six interview performance metrics).
2. Psychometrically grounded labeling protocol: Pairwise comparisons collected from clinical psychologists are converted into continuous scores using a Multinomial Logit model with nuclear-norm regularization (solved with cvxpy and the SCS solver using Barzilai–Borwein step sizes), reducing rater calibration bias relative to direct scalar ratings.
3. CRMF framework: Cross-Modal Regression with Manifold Fusion, a geometric deep learning architecture that projects fused multimodal features into hyperbolic, spherical, and Euclidean spaces, processes each with a geometry-specific expert, and combines them via learned routing and tangent-space fusion.
4. Adaptive routing and comprehensive evaluation: An entropy-regularized router that weights geometric experts by input characteristics, plus experiments showing consistent gains over MiniCPM-o 2.6, VideoLLaMA2.1-AV, and Qwen2.5-Omni baselines on all reported metrics.

Main Findings

• Macro-average improvement over baselines: The best CRMF configuration (VideoMAE + Wav2Vec2) reaches Spearman ρ = 0.5682, Kendall τ-b = 0.4069, C-index = 0.7183, and Pearson r = 0.5475, versus 0.5102, 0.3541, 0.6779, and 0.4935 for the strongest baseline, MiniCPM-o 2.6 (8B). This corresponds to an 11.4% relative gain in Spearman correlation, 14.9% in Kendall τ-b, and 6.0% in concordance index.
• Parameter efficiency: The VideoMAE + Wav2Vec2 CRMF uses 408M total parameters with 172M trainable, compared with roughly 340M trainable for MiniCPM-o, 300M for VideoLLaMA2, and 320M for Qwen2.5-Omni. The authors report training 40–50% fewer trainable parameters while outperforming the baselines.
• Best per-dimension results: Openness shows the strongest CRMF performance (ρ = 0.6384, a 13.4% improvement over the best baseline), while Overall Performance reaches ρ = 0.6521 with a 9.1% improvement. Neuroticism is described as the most challenging target but still improves over baselines by 24.5%.
• Multi-geometry beats single geometry: Hyperbolic-only (ρ = 0.5080), Spherical-only (ρ = 0.5338), and Euclidean-only (ρ = 0.5284) variants all underperform the full model (ρ = 0.5682), with drops of 10.6%, 6.1%, and 7.0% respectively.
• Naive fusion fails: Simple concatenation (ρ = 0.4441) and weighted averaging (ρ = 0.4664) degrade performance by 21.8% and 17.9% relative to full CRMF.
• Adaptive routing matters: Replacing the learned router with uniform weights drops Spearman ρ to 0.5209, an 8.3% degradation.
• Video is the strongest single modality: Video-only (VideoMAE) achieves ρ = 0.4516, text-only (DeBERTa) 0.4247, and audio-only (Wav2Vec2) 0.3792. Even the best unimodal result is far below any multimodal configuration; the jump from video-only to full CRMF is a 25.8% improvement.
• Encoder comparison: VideoMAE generally outperforms TimeSformer, while Wav2Vec2 and HuBERT perform comparably, with Wav2Vec2 holding a slight edge on correlation metrics.
• Label correlations: Overall Performance correlates most strongly with Agreeableness (ρ = 0.80), then Conscientiousness (ρ = 0.74), Openness (ρ = 0.70), and Extraversion (ρ = 0.65), and negatively with Neuroticism (ρ = -0.39). Among performance metrics, Confidence is strongest at ρ = 0.83, followed by Answer Score at ρ = 0.82 and Interview Score at ρ = 0.76.
• Label distribution properties: Speaking Skills (skew ρ ≈ -0.86) and Overall Performance (ρ ≈ -0.75) are negatively skewed while Answer Score (ρ ≈ 0.35) is positively skewed, motivating Huber loss training and rank-based evaluation metrics.

Methodology in Plain English

The authors first built a question bank of 76 interview prompts, drawn from public interview-preparation material, professional networking platforms, and consultations with clinical psychologists, and organized them into 15 sets of five questions plus a standard opening question. Student volunteers from Manipal universities were randomly assigned a set and recorded responses through a custom web platform in uncontrolled settings such as classrooms and homes, producing 2,011 video responses in compressed MP4 at 30 FPS, averaging about 30 seconds each. Audio was extracted as mono tracks, and verbatim transcripts were generated with Whisper-large-v3.

Instead of asking psychologists to score each clip on an absolute scale, the authors showed two clips answering the same question and asked which candidate better demonstrated a given attribute, with ties allowed. Roughly 27,000 such comparisons were converted into continuous scores for all 12 targets using a Multinomial Logit model with a Laplacian-weighted nuclear norm penalty, selected over Elo, Bradley-Terry-Luce, TrueSkill, and Glicko-2 after empirical comparison. Because the resulting label distributions are heavy-tailed and skewed, values beyond ±1.5σ are smoothly compressed toward ±3σ with a tanh-based clipping function.

The CRMF model encodes text with DeBERTa-v3-base, audio with Wav2Vec2 or HuBERT, and video with VideoMAE or TimeSformer, all mapped to a shared 768-dimensional space with the last few encoder layers fine-tuned. A transformer-based pre-fusion module with modality embeddings and attention pooling produces a clip-level representation, which is then projected onto three spaces: a Poincaré ball with curvature 1.0 for hierarchy, a unit sphere for directional patterns, and ordinary Euclidean space for continuous variation. A specialized expert network processes each manifold representation, with attention computed in each geometry's tangent space. A small MLP router outputs softmax weights over the three experts, with negative entropy regularization encouraging the experts to stay complementary rather than collapsing onto one. Final fusion combines the expert outputs in a shared tangent space as a routing-weighted average, which the authors describe as a first-order Fréchet mean approximation, before a shared MLP head with task-specific layers predicts the 12 targets.

Training uses AdamW with an effective batch size of 32 (batch size 4 with gradient accumulation over 8 steps), OneCycleLR with 15% warmup, 30 epochs, and early stopping on validation Spearman correlation with patience 5. Text is truncated at 512 tokens, audio resampled to 16 kHz, and video sampled at 16 FPS with 16 frames per clip resized to 224 × 224. The combined objective blends a Huber regression loss with δ = 1.0, a correlation boosting loss, a covariance alignment loss, and routing regularization terms, weighted adaptively.

Why This Matters

• Research impact: RecruitView provides an in-the-wild interview corpus with pairwise-derived continuous labels, addressing a documented shortage of domain-specific, psychometrically controlled data for personality and interview analysis. CRMF's central claim, that different behavioral attributes are better captured in different geometries, offers a concrete alternative to single-Euclidean-space fusion for multimodal behavioral modeling.
• Real-world applications:
  • Interview preparation and coaching tools that give candidates feedback on confidence, speaking skills, and response structure.
  • HR screening pipelines that surface structured, multi-dimension signals from recorded or asynchronous video interviews.
  • Training and development platforms that track communication competencies over time.
  • Clinical or counseling settings where behavioral traits are assessed from short recorded responses.
• Industry relevance: The dataset and both codebases are released publicly (dataset on Hugging Face and GitHub, CRMF code on GitHub), and the reported 40–50% reduction in trainable parameters versus 7B- to 8B-scale multimodal models suggests a path toward lower-cost deployment. Hiring is a high-stakes domain, so the paper's limitations, including residual annotator bias and limited demographic diversity, matter as much as its accuracy gains.

Future Directions

1. Scale and diversity: The authors identify moderate dataset scale, short clips, and limited demographic diversity as limitations, and propose broadening populations and recording conditions to improve external validity.
2. Longer, multi-turn interactions: Extending from single short responses to extended multi-turn interview dialogues.
3. Target-wise manifold selection: Learning which geometry is appropriate per target rather than applying all three uniformly, alongside stronger self-supervised priors.
4. Causal analysis and deployment: Investigating causal relationships among traits and performance, plus real-time inference and human-in-the-loop calibration.
5. Open questions the paper raises but does not resolve: Whether the geometric advantage persists at larger data scales, whether the reported gains generalize across languages and cultures, and how residual label noise from pairwise judgments propagates to deployed predictions.

Target Audience

This paper is most useful to researchers in multimodal machine learning and computer vision working on human behavior analysis, to HR technology practitioners evaluating automated interview assessment, and to organizational psychologists interested in psychometric label construction. It also suits readers studying geometric deep learning who want to see hyperbolic, spherical, and Euclidean representations combined in an applied regression setting. Readers without a background in manifold-valued neural networks will find the methodology sections demanding, though the dataset description and results are accessible. The provided paper content is truncated before the appendices, so per-dimension tables (referenced as Table 8, Table 9, and Table 10), the full 12 × 12 Spearman correlation matrix, summary statistics, ethics discussion, and complete architecture formulations are cited but not included here; MSE is listed as an evaluation metric but no MSE values appear in the available content.

Authors’ abstract

Automated personality and soft skill assessment from multimodal behavioral data remains challenging due to limited datasets and methods that fail to capture geometric structure inherent in human traits. We introduce RecruitView, a dataset of 2,011 naturalistic video interview clips from 300+ participants with 27,000 pairwise comparative judgments across 12 dimensions: Big Five personality traits, overall personality score, and six interview performance metrics. To leverage this data, we propose Cross-Modal Regression with Manifold Fusion (CRMF), a geometric deep learning framework that explicitly models behavioral representations across hyperbolic, spherical, and Euclidean manifolds. CRMF employs geometry-specific expert networks to capture hierarchical trait structures, directional behavioral patterns, and continuous performance variations simultaneously. An adaptive routing mechanism dynamically weights expert contributions based on input characteristics. Through principled tangent space fusion, CRMF achieves superior performance while training 40-50% fewer trainable parameters than large multimodal models. Extensive experiments demonstrate that CRMF substantially outperforms the selected baselines, achieving up to 11.4% improvement in Spearman correlation and 6.0% in concordance index. Our RecruitView dataset is publicly available at https://huggingface.co/datasets/AI4A-lab/RecruitView