GBU-Palm: A Multimodal Video Dataset and Benchmark for Palm Presentation Attack Detection

AI summary

Overview

Research area: Computer vision for biometric security — specifically palm presentation attack detection (PAD), also called palm anti-spoofing.

Technical level: Intermediate. The paper is readable without deep expertise, but it assumes familiarity with video classification models, AUC/HTER metrics, and the concepts of cross-domain generalization and multimodal fusion.

Scope: The paper introduces GBU-Palm, a large-scale multimodal (RGB + near-infrared) palm video dataset with 21,326 videos from 105 subjects across six acquisition environments, plus a leakage-controlled benchmark and diagnostic analyses of how four video architectures use temporal and spectral evidence.

What This Paper Is About

Palm-based authentication is convenient and contactless, but attackers can hold up a printed palm image or replay palm content on a screen, and many existing palm PAD datasets make this hard to study systematically because they rely on static images, a single sensing spectrum, or limited acquisition conditions. The paper's goal is to provide a single resource that jointly offers native video, synchronized RGB-NIR capture, multiple environments, and traced attack origins, and then to use it to test how well representative video models detect attacks both in familiar conditions and in environments they never saw during training.

Key Contributions

1. A large multimodal video dataset. GBU-Palm contains 21,326 videos from 105 subjects and 210 palms across six acquisition environments (E1–E6), covering bona fide, Print, and Replay presentations, including 6,310 synchronized RGB-NIR samples and attack-lineage annotations that record each attack's source session, generation method, and material identity.

2. Standardized, leakage-controlled benchmark protocols. Two protocols are defined: P1 (In-Env, environment-matched) keeps all six environments in every split, while P2 (Cross-Env) trains on E2, E4, E5, and E6 and withholds E1 and E3 entirely for testing. Both enforce disjoint subjects, palm identities, and attack lineages between training and evaluation.

3. A systematic architecture comparison. Four representative video backbones — R(2+1)D-18, an adapted factorized ViViT baseline, Video Swin-T, and MViT-V2-S — are evaluated under RGB, NIR, and RGB-NIR inputs in both protocols.

4. Diagnostic analyses beyond aggregate accuracy. The paper examines decision outcomes (true accept, true reject, false accept, false reject), attack-family vulnerability, spectral masking through paired RGB/NIR comparison, temporal-order interventions, and frozen-backbone NIR linear probing.

Main Findings

• Cross-environment degradation is architecture-dependent, not uniform. Under P2 Cross-Env, AUC drops by 11.28 points for ViViT, 2.87 for R(2+1)D, 1.75 for Video Swin-T, and 4.03 for MViT, compared with their P1 In-Env results.

• Baseline performance already diverges sharply between architectures. On P1, R(2+1)D-18 reaches 97.48% AUC / 8.39% HTER and MViT-V2-S 97.33% / 8.39%, while Video Swin-T reaches only 72.95% AUC / 32.89% HTER. On P2, R(2+1)D-18 is at 94.61% / 13.05% and ViViT falls to 82.21% / 25.73%.

• No single environment is universally the hardest. Across the six P1 environments, Video Swin-T spans 10.0 AUC points and ViViT 4.6 points, whereas R(2+1)D and MViT vary by only 2.2 and 1.7 points. The best and worst environments differ per architecture. Within P2, E3 yields lower AUC than E1 for all four models, with the largest gap of 2.7 points for MViT.

• Similar AUC changes can hide opposite failure modes. MViT's false accept rate rises from 7.52% on P1 to 26.32% on P2 while its false reject rate falls from 9.26% to 7.68%; R(2+1)D's false accept rate rises from 9.77% to 18.21% with relatively stable false reject; ViViT increases in both.

• Replay attacks are harder to reject than Print attacks. Under MViT P2, Print false accept is 11.47% while Replay false accept reaches 36.08%.

• RGB-NIR fusion is not consistently better than RGB alone. R(2+1)D improves from 4.33% HTER (RGB) to 3.02% (RGB+NIR) on P1 and from 7.97% to 4.08% on P2, and ViViT also benefits, but Video Swin-T worsens from 19.74% to 22.98% HTER on P1.

• NIR-only vulnerability varies enormously between models. On P1, NIR-only Replay false accept reaches 67%, 93%, and 94% for R(2+1)D, ViViT, and Swin respectively, while RGB+NIR reduces these to 6%, 0%, and 38%. MViT maintains 0% under both conditions.

• Video input does not guarantee that temporal order is used. R(2+1)D degrades under shuffling and reversal, and MViT shows smaller but consistent drops, but Video Swin-T shows no stable degradation and the adapted ViViT produces identical AUC under both interventions (93.49 on P1 and 82.21 on P2 in all three conditions), which the authors attribute to its lack of temporal positional encoding and use of temporal mean pooling.

• NIR degradation has different causes in different models. Frozen-backbone linear probing leaves MViT near 99% AUC, improves R(2+1)D from 84.09% to 98.17% on P1, and improves ViViT from 67.71% / 67.14% to 90.42% / 85.01% on P1 / P2 — indicating unused linearly accessible information. Swin reaches only 50.00% AUC with the P1 linear probe, indicating weak linear separability of its NIR representation.

Methodology in Plain English

The researchers first built a data collection protocol rather than a model. They recorded genuine palms, printed copies of palm images, and screen-replayed palm content using consumer phones, tablets, laptops, and a synchronized RGB-NIR capture system, across six lighting settings labeled E1 through E6 that range from uniform indoor light to outdoor directional side and back lighting. Each sample is a continuous 16-second observation stored with its native frame order and timestamps as a 256×256 region of interest, so the video contains real motion and reflection rather than frames assembled from still images. They recorded metadata tracing each attack back to its source session, generation method, and material.

For the benchmark, they framed palm PAD as a binary decision between bona fide and attack. Each model receives 32 unique real frame positions resized to 112×112, with no interpolation or duplicated frames. They defined two evaluation protocols: P1 keeps all six environments represented in training, validation, and test while keeping subjects separate, and P2 holds E1 and E3 out of training entirely so that test performance measures generalization to unseen environments. Both protocols keep palm identities and attack lineages disjoint between splits to prevent leakage.

They trained four video backbones with different temporal modeling designs under identical optimization settings (AdamW, learning rate 2×10⁻⁴, weight decay 10⁻⁴, up to 100 epochs, patience 15), selected checkpoints by validation AUC, and reported AUC and HTER, where HTER averages FAR and FRR. For multimodal input, the two streams share one backbone and their embeddings are simply averaged before the classification head, deliberately avoiding a dedicated fusion module so that comparisons reflect evidence use rather than fusion design.

Beyond headline scores, they ran targeted probes: decomposing decisions into true accept, true reject, false accept, and false reject; breaking false accepts down by attack family; comparing paired RGB, NIR, and RGB+NIR results on identical samples from the same checkpoint; shuffling and reversing frame order while keeping the same 32 frames; and freezing the NIR backbone to train a fresh linear classifier on validation NIR embeddings to test whether the information was present but unused by the original classifier.

Why This Matters

Impact on research. Prior palm PAD resources were largely static-image, single-spectrum, or limited in acquisition conditions — the paper's own comparison table lists VERA (2,000 NIR images, Print), PALMspoof (private, RGB images, Print/display), XJTU-PalmReplay (96,000 RGB images, Replay), and PVASD (1,187,519 NIR images, 2D/3D). GBU-Palm is positioned as the first to combine large-scale native video, synchronized RGB-NIR, six environments, traced attack lineage, and a controlled public benchmark in one resource. The finding that RGB-NIR fusion does not consistently help, and that video input does not guarantee temporal-order use, gives the field concrete evidence that multimodal and video methods must be validated rather than assumed.

Real-world applications.

• Contactless palm authentication at access-control gates, where printed palm images or phone-displayed palm photos are the obvious attack.
• Mobile and kiosk payment or identity verification that must work under indoor, low-light, and outdoor lighting without performance collapsing.
• Anti-spoofing evaluation for devices that ship both RGB and NIR sensors, informing whether the extra NIR channel is worth its cost for a given model.
• Security audits that need to distinguish attack-acceptance risk from user-rejection cost, since the paper shows these move in opposite directions for some architectures.

Industry relevance. The architecture-dependent results, particularly the large variance in false accept rates and the failure of naive multimodal averaging to help every model, are directly relevant to teams choosing a backbone and sensor configuration for a deployed palm PAD system. Because the benchmark separates security risk (false accept) from usability (false reject), it supports risk-based decisions rather than a single leaderboard number. The dataset is described as being released soon, under a CC BY 4.0 license for the paper.

Future Directions

• Extending the dataset to three-dimensional and more challenging attacks, which the authors state as a planned future extension.
• Adding more sensing modalities beyond synchronized RGB and NIR.
• Providing richer spatiotemporal annotations, as mentioned in the conclusion.
• Determining why some architectures fail to use available temporal order and NIR information — the paper shows the gap exists but does not resolve whether better fusion design, temporal positional encoding choices, or representation learning would close it.
• Testing whether the observed architecture-dependent cross-environment degradation pattern holds for models and attack types outside the four backbones and three presentation classes studied here.

Target Audience

Researchers and engineers working on biometric anti-spoofing, presentation attack detection, and liveness verification, especially those focused on palm or hand-based systems. It is also useful for computer vision researchers studying video model generalization, multimodal RGB-NIR fusion, or domain shift, and for practitioners who need a controlled benchmark to choose a video backbone or sensor configuration for a real deployment. Readers looking for a new model architecture will not find one here — the contribution is data, protocol design, and diagnostic evidence.

Authors’ abstract

Existing palm presentation attack detection (PAD) datasets are often limited by static imagery, restricted acquisition conditions, or insufficient multimodal video data, hindering systematic evaluation across environments, modalities, and attack types. We present GBU-Palm, a large-scale multimodal video dataset and benchmark containing 21,326 videos from 105 subjects and 210 palms across six acquisition environments, including bona fide, Print, and Replay presentations, with 6,310 synchronized RGB-NIR samples. We construct leakage-controlled protocols that separate palm identity and attack lineage and benchmark four representative video architectures under environment-matched and held-out-environment settings. Results reveal substantial architecture-dependent degradation under environmental shift and show that RGB-NIR fusion does not consistently outperform RGB-only input. We further analyze model behavior through true accept (TA), true reject (TR), false accept (FA), and false reject (FR) decomposition, spectral masking, temporal-order intervention, and frozen-backbone NIR probing, revealing distinct failure patterns and evidence utilization across architectures. GBU-Palm provides a unified and challenging benchmark for developing and evaluating robust multimodal palm PAD methods under cross-environment conditions.