SparseLaneSTP: Leveraging Spatio-Temporal Priors with Sparse Transformers for 3D Lane Detection

AI summary

Overview

• Research area: Computer vision for autonomous driving — 3D lane detection from monocular images using sparse transformer architectures.
• Technical level: Advanced (assumes familiarity with transformers, attention mechanisms, bird's-eye-view (BEV) transformations, and spline curve representations).
• Scope: A single paper introducing SparseLaneSTP, a sparse 3D lane detector that injects lane-specific spatial priors and temporal information into a transformer, plus a new auto-labeled 3D lane dataset.

What This Paper Is About

Most 3D lane detectors convert front-view images into a dense bird's-eye-view feature map before predicting lanes, but errors in that transformation misalign the features with the true road surface. A newer class of "sparse" detectors avoids the BEV step by treating lane points as queries, yet they ignore known lane geometry and throw away information from previous video frames. This paper builds a sparse lane transformer that deliberately exploits both lane-structure priors and past observations, and accompanies it with a new auto-labeled 3D lane dataset that has accurate long-range labels.

Key Contributions

1. A sparse 3D lane detector integrating spatial and temporal knowledge. SparseLaneSTP combines lane-structure priors with historical observations inside a sparse transformer decoder.
2. A new attention mechanism based on lane structure and history. The spatio-temporal attention (STA) layer decomposes attention into same line attention (SLA), parallel neighbor attention (PNA), and temporal cross-attention (TCA), the last pulling in keys from a memory queue of past lane queries.
3. A continuous lane representation tailored to sparse queries, plus temporal regularization. The model uses Catmull-Rom (CR) splines, whose control points lie exactly on the curve — unlike B-Spline control points — so the sparse query positions correspond to real lane points. A temporal smoothness loss regularizes predictions against an exponentially moving average of past outputs.
4. A new 3D lane dataset with an auto-labeling pipeline. Labels are generated by temporally accumulating confident near-range 2D detections in 3D along an accurate ego-trajectory, yielding consistent labels up to 250 m, plus occlusion-aware visibility labels.
5. State-of-the-art results on OpenLane, ONCE-3DLanes, and the new dataset, across detection and error metrics.

Main Findings

• OpenLane results: SparseLaneSTP reaches 66.1% F1-Score, the highest reported in the paper's comparison table, with the lowest geometric errors (0.203 m X-error near, 0.240 m X-error far, 0.066 m Z-error near, 0.092 m Z-error far). It uses a ResNet-50 backbone at 720×960 resolution, versus the second-best (GroupLane, 64.1% F1) and the most related baseline LATR (61.9% F1).
• ONCE-3DLanes results: highest scores with 82.75% F1, 86.47% precision, 79.33% recall, and the lowest chamfer distance of 0.048 m (LATR: 80.59% F1, 86.12% P, 75.73% R, 0.052 m CD).
• Own dataset results: 68.2% F1 and 81.4% Vis-IoU, ahead of the LATR baseline (65.1% F1, 78.3% Vis-IoU), LaneCPP (63.1% F1, 77.5% Vis-IoU) and PersFormer (59.2% F1, 70.1% Vis-IoU). Extended error intervals for 100–150 m (0.443 / 0.182 m) and 150–200 m (0.646 / 0.263 m) are reported.
• Ablation of contributions (2-layer model, OpenLane): baseline 61.8% F1; adding the continuous CR spline representation 62.9% (+1.1); adding STA 65.0% (+2.1); adding regularization 65.3% (+0.3).
• Attention decomposition matters: global attention gives 62.9% F1, SLA alone 62.8%, PNA 63.5%, SLA+PNA 63.8% (+0.9% over global), and SLA+PNA+TCA 65.0% (+1.2% further). The largest single gain comes from temporal cross-attention.
• Memory length: T = 3 historic frames is best (65.0% F1), versus 63.8% for T = 0, 64.1% for T = 1, 64.6% for T = 2, 64.5% for T = 4, and 63.9% for T = 5. Too few frames lack context; too many add redundancy.
• Efficiency: the six-layer model runs at 11.0 FPS versus LATR at 12.1 FPS. Temporal integration adds a modest 9% overhead, and a two-layer model reaches 65.3% F1 (+3.4% over six-layer LATR) at 16.5 FPS.
• Robustness under occlusion: qualitative comparisons show the temporal model retains lane detections across frames where visibility deteriorates, whereas the non-temporal model loses them; regularization reduces drifting and gradual disappearance.
• Dataset scale: OpenLane comprises 200K images from 1000 sequences; ONCE-3DLanes comprises 211K images; the new dataset comprises 511K images with a 90% / 10% train/test split, gathered from multiple regions and countries.

Methodology in Plain English

The system takes a single RGB image and a video context. A CNN backbone extracts image features, and a lane instance segmentation branch produces initial query embeddings for a fixed number of lane proposals. Each lane is represented by a set of control points that form a Catmull-Rom spline, so the network directly outputs smooth curves rather than discrete points. The x, z, and visibility components of each control point are predicted with separate small networks, while the longitudinal y component is fixed and uniformly spaced to avoid over-parameterization.

A transformer decoder processes these queries. Instead of allowing every query to attend to every other query, the authors split attention into three restricted channels: attention among points on the same lane, attention to points on neighboring (roughly parallel) lanes, and attention to the closest queries from previous frames held in a FIFO memory queue. Past queries are repositioned into the current frame's coordinate system using the vehicle's ego-motion, and their positions and visibility are encoded before being used as keys. Training combines standard detection losses (L1 for x and z, binary cross-entropy for visibility, focal loss for classification) with spatial regularization borrowed and extended from prior work — encouraging parallelism, surface smoothness, and curvature limits — and a temporal smoothness loss that pulls current predictions toward an exponentially moving average of past predictions.

For the dataset, the authors avoid LiDAR entirely. Visual odometry recovers the ego-trajectory, a spline road surface is fitted from trajectory and orientation, and confident near-range 2D lane detections are projected onto that surface and accumulated frame by frame. A semantic segmentation network assigns occlusion labels to each lane point.

Why This Matters

The work argues that two commonly discarded sources of information — the known geometry of lanes and the history of observations — are both cheap to exploit and genuinely useful. It also challenges the assumption that LiDAR-based annotation produces the best 3D ground truth for long-range lanes, and provides an alternative dataset with occlusion-aware visibility labels that extend well beyond what existing benchmarks cover.

Real-world applications:

• Highway and urban lane keeping / lane centering for advanced driver assistance systems, where accurate far-range lane geometry matters at speed.
• Occlusion-resilient perception, such as maintaining a lane estimate when a truck or other traffic participant blocks the view ahead.
• Lane-level localization and mapping, using temporally consistent lane observations and the dataset's track IDs and relational lane information.
• Training and validating perception stacks, using the new 511K-image dataset with long-range, occlusion-annotated labels as an evaluation benchmark.

Industry relevance: the authors are affiliated with Bosch Mobility Solutions, and the paper is framed around the practical failure modes of production-style 3D lane detection — transformation error, occlusion, and label quality. The efficiency analysis (comparable FPS to an existing sparse detector, with a smaller model reaching higher accuracy) is directed at deployment constraints rather than accuracy alone.

Future Directions

• 3D lane tracking: the authors state explicitly that extending the method to 3D lane tracking is future work, to exploit temporal information more fully.
• Tuning the temporal window: T = 3 was best in this study, but the drop at T = 4 and T = 5 suggests the memory mechanism's redundancy handling is an open problem.
• Which priors generalize: the spatial regularization is adapted from prior dense-BEV work (LaneCPP); how well each penalty term transfers to other sparse architectures is not isolated in the reported ablations.
• Dataset adoption and annotation scope: the auto-labeling pipeline is demonstrated on one dataset; whether the approach scales to other regions, sensor setups, and label conventions is left open.

Target Audience

Researchers and engineers working on 3D perception for autonomous driving, particularly those building or evaluating sparse query-based detectors, transformer attention designs, or temporal fusion for video-based lane detection. It is also relevant to teams concerned with lane dataset construction, since the auto-labeling pipeline and occlusion-aware visibility labels are presented as contributions in their own right. Readers without background in BEV transformations and DETR-style query architectures will find the methodology sections dense.

Authors’ abstract

3D lane detection has emerged as a critical challenge in autonomous driving, encompassing identification and localization of lane markings and the 3D road surface. Conventional 3D methods detect lanes from dense birds-eye-viewed (BEV) features, though erroneous transformations often result in a poor feature representation misaligned with the true 3D road surface. While recent sparse lane detectors have surpassed dense BEV approaches, they completely disregard valuable lane-specific priors. Furthermore, existing methods fail to utilize historic lane observations, which yield the potential to resolve ambiguities in situations of poor visibility. To address these challenges, we present SparseLaneSTP, a novel method that integrates both geometric properties of the lane structure and temporal information into a sparse lane transformer. It introduces a new lane-specific spatio-temporal attention mechanism, a continuous lane representation tailored for sparse architectures as well as temporal regularization. Identifying weaknesses of existing 3D lane datasets, we also introduce a precise and consistent 3D lane dataset using a simple yet effective auto-labeling strategy. Our experimental section proves the benefits of our contributions and demonstrates state-of-the-art performance across all detection and error metrics on existing 3D lane detection benchmarks as well as on our novel dataset.