EPRBench: A High-Quality Benchmark Dataset for Event Stream Based Visual Place Recognition

AI summary

Overview

Research area: Computer vision — event-camera-based Visual Place Recognition (VPR), spanning benchmark dataset construction, multi-modal (event + language) retrieval, and LLM-driven explainability.

Technical level: Intermediate. The paper assumes familiarity with image retrieval metrics (Recall@N), Vision Transformers, contrastive/metric learning, and event-camera data representation.

Scope: The paper introduces EPRBench, a 1280×720 event-stream VPR benchmark with LLM-generated scene descriptions, benchmarks 15 existing VPR algorithms on it, and proposes SG-VPR, a semantic-guided multi-modal fusion baseline that uses text to select and weight event-derived visual tokens.

What This Paper Is About

Visual Place Recognition asks whether an agent has been somewhere before based only on current visual input, and RGB cameras struggle in low light, overexposure, and high-speed motion. Event cameras react to pixel brightness changes asynchronously and offer a dynamic range stated as 120–140 dB, but the authors state that only three event-based VPR datasets are publicly available — Brisbane-Event-VPR (2020), NeuroGPR (2023), and NYC-Event-VPR (2024) — and these are limited in scale and diversity. The paper's goal is to provide a larger, semantically annotated event-stream VPR benchmark (EPRBench) plus a strong, interpretable baseline (SG-VPR) on top of it.

Key Contributions

1. A new high-definition event-stream VPR benchmark, EPRBench. Built from 1280×720 recordings on a PROPHESEE EVK4-HD event camera, containing 13,109 event scene samples (each sample made of five event frames), organized into Campus, Park, and Road scene categories, collected by handheld and vehicle-mounted setups across multiple viewpoints, weather, and lighting conditions.
2. Semantic/language annotations. LLM-generated scene descriptions (Chinese Chain-of-Thought generated with Doubao Pro-1.6, translated with NLLB-200-3B, refined with DeepSeek, and used to fine-tune Qwen2.5-VL-7B via LoRA) with manual verification, enabling language-integrated VPR research. EPRBench is described as the first event-based VPR dataset to provide rich semantic annotations for each scene.
3. A 15-algorithm benchmark. The authors retrain and report CNN-based, Transformer-based, and re-ranking-based VPR models on EPRBench, providing a baseline for future comparisons.
4. SG-VPR, a semantic-guided multi-modal VPR baseline. It uses LLM-generated text to guide spatially attentive token selection, cross-modal fusion, multi-scale aggregation, and adds chain-of-thought reasoning for interpretability.

Main Findings

• EPRBench scale and composition. The dataset is described as containing 10K event sequences and 65K event frames in the abstract, and as 13,109 event scene samples — Campus: 315 scenes / 2,766 samples; Park: 351 scenes / 5,176 samples; Road: 356 scenes / 5,167 samples. Table I lists 1,022 scenes for EPRBench, and the introduction refers to "1K urban scenes." Samples were split 7:1:2 into 9,220 training, 1,273 validation, and 2,616 testing scene samples.
• SG-VPR achieves 94.3% R@1 on EPRBench, with 96.1% R@5 and 97.1% R@10. The paper states this outperforms SuperVLAD (93.3% R@1) and CricaVPR (92.6% R@1). On R@1, FOL reaches 93.8%, PRGS 87.7%, Pair-VPR 91.9%, SALAD 91.3%, BoQ 93.1%, MixVPR 90.2%, EigenPlaces 81.1%, R2Former 80.0%, SelaVPR 89.9%, EMVP 88.7%, ImAge 76.2%, CosPlace 73.2%, and Deep visual geo-loc 63.2%. On R@10, PRGS (96.9%) and FOL (97.1%) are close to SG-VPR's 97.1%.
• On NYC-Event-VPR (Event modality), SG-VPR reports 57.5% R@1, 74.1% R@5, 78.9% R@10. Pair-VPR records the highest R@1 at 64.0%, and R2Former reports 61.2%; the paper describes SG-VPR as second-best overall while exceeding CricaVPR (30.3%) by +27.2% and MixVPR (45.7%) by +11.8% in R@1.
• On NYC-Event-VPR (RGB modality), SG-VPR reports 86.5% R@1, 90.9% R@5, 92.0% R@10, which the paper describes as competitive with leading RGB-based methods. The highest RGB R@1 in the table is Pair-VPR at 89.4%, followed by SALAD at 89.1%.
• Ablation (partial). Table III reports a "Baseline (Vision Only)" at 92.6% R@1 and 95.3% R@5 on EPRBench; the R@10 value and the remaining ablation rows are not present in the provided content.
• Interpretability demonstration. Figures 5 and 6 show qualitative retrieval results and the auxiliary "think" reasoning blocks, where the model gives step-by-step semantic analysis; these are qualitative figures, not reported as numerical metrics.
• Existing dataset comparison. Table I lists DDD17 (2017, DAVIS 346, 346×260, 40 scenes), DDD20 (2020, DAVIS 346, 346×260, 215 scenes), Brisbane-Event-VPR (2020, DAVIS 346, 346×240, 6 scenes), NeuroGPR (2023, DAVIS 346, 346×240, 12 scenes), and NYC-Event-VPR (2024, Prophesee EVK4-HD, 1280×720, 16 scenes). EPRBench is marked as the only one of these with a Text column checked.

Methodology in Plain English

Dataset construction. The authors followed three stated protocols: large scale, two different collection setups (vehicle-mounted for wide road scenes, handheld for fine-grained small-scale scenes), and multi-viewpoint/multi-condition capture. They used a PROPHESEE EVK4-HD event camera at 1280×720 and grouped scenes into Campus (buildings and dense areas such as schools and neighborhoods), Park (non-building open areas not on main roads), and Road (streets and near-highway scenes).

Text annotation pipeline. In Stage 1, 50,000 image sample pairs (half positive, half negative) were constructed from the event data, and Doubao Pro-1.6 generated initial Chinese Chain-of-Thought descriptions via template prompting. In Stage 2, NLLB-200-3B translated the Chinese CoT into English, and DeepSeek was prompted to verify and correct translation accuracy and fluency. In Stage 3, Qwen2.5-VL-7B was fine-tuned with LoRA (supervised fine-tuning) on the resulting high-quality English CoT data to become a "scene model expert" that generates descriptions.

SG-VPR architecture. Event streams are first collapsed into 2D event frames by accumulating events within a fixed time interval according to polarity and coordinates. These frames go through a frozen DINOv2 encoder to produce patch tokens and a global class token. In parallel, the scene description text is tokenized, padded to a fixed length, and passed through a frozen CLIP text encoder to yield word-level and sentence-level features.

Two fusion levels follow. Global fusion concatenates the visual class token with the sentence-level text feature and passes it through an MLP. Local fusion computes, for each visual patch, its maximum cosine similarity to any word token, keeps the top ratio of patches (ρ = 0.25, so k = floor(0.25·N)), and injects the most similar word token into only those selected patches via an element-wise interaction scaled by a learnable factor; unselected patches are left unchanged.

Aggregation. The patch tokens are reshaped into a 2D feature map. Instead of a standard global pooling layer, the global multi-modal anchor replaces it, and GeM pooling is applied on 2×2 and 3×3 grids. The three are concatenated into the final retrieval descriptor.

Training. The framework uses a Multi-Similarity loss for retrieval (α = 1.0, β = 50, λ = 1.0) plus a symmetric InfoNCE loss between the visual class token and sentence token (τ = 0.07), combined with weight γ = 0.15. Training ran in PyTorch on a single NVIDIA RTX 3090, with inputs resized to 224×224 and ImageNet normalization, Adam optimizer at learning rate 1×10⁻⁴, weight decay 1×10⁻³, StepLR decay of 0.5 every 3 epochs, and batch size 24. DINOv2 and CLIP text encoder parameters were frozen; only adapters, projection layers, and the cross-modal fusion module were updated.

Evaluation. Recall@N (N ∈ {1, 5, 10}) is used: a query is a true positive if at least one of the top-N retrieved samples shares the query's location label, and the score is the percentage of such queries over the query set.

Why This Matters

Impact on research. The paper argues the field is held back by only three publicly available event-based VPR datasets that are limited in scale and diversity, by a representation-learning paradigm that underuses semantic cues, and by poor model interpretability that relies on post-hoc activation heatmaps. EPRBench adds scale, scene diversity, and text annotations, and the 15-model benchmark gives future work a common comparison point. The authors state the dataset and source code will be released at https://github.com/Event-AHU/Neuromorphic_ReID.

Real-world applications cited by the paper (bullets):

• Loop closure detection in robotic SLAM
• High-precision localization in autonomous driving
• UAV navigation
• Augmented reality

Industry relevance. The paper positions event cameras as complements to RGB cameras specifically for autonomous and mobile systems that must operate under GPS denial, low illumination, overexposure, high dynamic range, and high-speed motion, where frame-based cameras at roughly 30 FPS suffer motion blur and where high-frame-rate alternatives would increase bandwidth and compute demands. The interpretability branch, which produces natural-language reasoning alongside predictions, is relevant to safety-critical deployments where human-understandable justification for a localization decision is required.

Future Directions

• Closing the gap on challenging event data. SG-VPR is second-best on NYC-Event-VPR Event (57.5% R@1) behind two-stage methods such as Pair-VPR (64.0% R@1), leaving room to improve under severe event sparsity and urban complexity.
• Exploiting the text annotations more fully. The paper provides verified scene descriptions but the proposed baseline uses them mainly for token selection and alignment; richer uses of language supervision for event-based VPR remain open.
• Scaling and diversifying collection. The authors note the dataset was collected with handheld and vehicle-mounted setups; extending to more sensors, more cities, more weather/lighting combinations, and larger scale is a natural continuation.
• Strengthening interpretability evaluation. The chain-of-thought reasoning is demonstrated qualitatively in Figures 5 and 6; the provided content does not report a quantitative metric for reasoning quality or explanation fidelity.
• Verifying the ablation picture. Only the vision-only baseline row (92.6% R@1, 95.3% R@5) of Table III appears in the provided content; the full ablation of the fusion modules is not available here.

Target Audience

Researchers and engineers working on visual place recognition, robot localization and SLAM, autonomous driving perception, and neuromorphic/event-based vision. It is also relevant to practitioners interested in grounding vision models with LLM-generated text and chain-of-thought reasoning, and to teams evaluating whether event cameras can replace or supplement RGB cameras in challenging illumination and high-speed conditions. A background in deep learning, retrieval evaluation, and multi-modal representation learning is helpful but not required to follow the dataset contribution.

Authors’ abstract

Event stream-based Visual Place Recognition (VPR) is an emerging research direction that offers a compelling solution to the instability of conventional visible-light cameras under challenging conditions such as low illumination, overexposure, and high-speed motion. Recognizing the current scarcity of dedicated datasets in this domain, we introduce EPRBench, a high-quality benchmark specifically designed for event stream-based VPR. EPRBench comprises 10K event sequences and 65K event frames, collected using both handheld and vehicle-mounted setups to comprehensively capture real-world challenges across diverse viewpoints, weather conditions, and lighting scenarios. To support semantic-aware and language-integrated VPR research, we provide LLM-generated scene descriptions, subsequently refined through human annotation, establishing a solid foundation for integrating LLMs into event-based perception pipelines. To facilitate systematic evaluation, we implement and benchmark 15 state-of-the-art VPR algorithms on EPRBench, offering a strong baseline for future algorithmic comparisons. Furthermore, we propose a novel multi-modal fusion paradigm for VPR: leveraging LLMs to generate textual scene descriptions from raw event streams, which then guide spatially attentive token selection, cross-modal feature fusion, and multi-scale representation learning. This framework not only achieves highly accurate place recognition but also produces interpretable reasoning processes alongside its predictions, significantly enhancing model transparency and explainability. The dataset and source code will be released on https://github.com/Event-AHU/Neuromorphic_ReID