Point Clouds and Three-Dimensional Perception

A lidar scan can be stored as XYZ points, projected into a range image, accumulated into voxels, or rendered into camera views; each representation makes some operations easier and discards or quantizes other structure.

The thing being stored is the same object every time. The nuScenes lidar returns up to 1.4 million points per second at 20 rotations per second — roughly 70,000 points in a single sweep — and the routes below are different answers to the question of what shape that sweep should take in memory.

The choice is not merely a file format: it defines neighborhood, resolution, memory, and invariance.

Each route encodes a different inductive bias about space and computation.

The first route is not an aspiration. PointNet gets permutation invariance out of a single symmetric function — max pooling over per-point features. Reorder the input rows and the output is identical by construction, not by training. Qi and Su, with two co-authors, published it at CVPR in 2017.

The price of that austerity is measurable. ModelNet40 holds 12,311 CAD models in 40 categories, split 9,843 train and 2,468 test. PointNet scores 89.2% overall accuracy on it, and 86.2% average-class accuracy.

The same design is what makes the representation tolerant of missing samples: “As to missing points, when there are 50% points missing, the accuracy only drops by 2.4% and 3.8% w.r.t. furthest and random input sampling.” Throw away half the cloud and the model barely moves.

Those figures survive an independent tabulation. A later survey of deep learning on 3D point clouds, in IEEE Transactions on Pattern Analysis and Machine Intelligence, lists PointNet at the same 89.2% OA and 86.2% mAcc on ModelNet40, at 3.48M parameters, and gives ModelNet40 as 12,311 samples, 40 classes, 9,843 training and 2,468 test, synthetic mesh. Two groups, the same numbers.

Lidar samples become sparser with distance. They can be blocked by foreground surfaces. Reflectivity, incidence angle, weather, and multi-path effects also change returns.

The sensor sheet makes the sparsity concrete. The nuScenes lidar is specified as “Spinning, 32 beams, 20Hz capture frequency, 360° horizontal FOV, −30° to 10° vertical FOV, ≤ 70m range, ±2cm accuracy, up to 1.4M points per second.” Spread 32 beams over a vertical span that narrow and the gap between beams widens with every metre of range. The same car at 10 metres and at 60 comes back as very different numbers of points. One sensor, one sweep.

What that thinning costs a detector has been measured, with the model and the scenes held fixed. On the Waymo Open Dataset, PointPillars scores 63.3 vehicle 3D AP (LEVEL_1) on the 150-scene hidden test set. That single number is an average: 82.3 for objects at 0–30 m, 59.2 at 30–50 m, and 35.7 at 50 m and beyond.

The collapse is not an artefact of one implementation. Seven authors at MIT and Google Research re-implemented the detector in 2020 and reproduced it on the validation set with their own numbers: 54.25 vehicle 3D mAP overall, 76.31 at 0–30 m, 48.08 at 30–50 m, 24.21 at 50 m–Inf.

The benchmark writes the same fact into its difficulty tiers, and it counts points to do it: “Next, we assign LEVEL 2 to examples where either the labeler annotates as hard or if the example has ≤ 5 LiDAR points.” Not occlusion. Not object size. Five returns or fewer.

A network trained on dense nearby objects may fail on the same class at long range; density augmentation and range slices should reflect the real acquisition process.

Shared representations do not make these tasks interchangeable. Registration aligns scans or shapes in a common coordinate system and returns a rigid or deformable transform. 3D detection returns object classes with position, size and orientation, and degrades on sparse distant objects. Point segmentation returns per-point categories and depends on sampling density. Surface reconstruction returns a mesh, an occupancy field or an implicit field, and must assume a topology it cannot observe.

The first of these has the oldest documented failure. A controlled comparison of ICP variants, run at Stanford in 2001, included a test scene the authors call the incised plane. Rusinkiewicz and Levoy write of it: “This is a difficult scene for ICP, and most variants do not converge to the correct alignment, even given the small relative rotation in this starting position.” In that scene, they found, “the closest-point algorithms were the only ones that converged to the correct solution”. Almost everything else got stuck.

The deeper problem is that a locally optimal method cannot report its own failure. Go-ICP, in 2016, states that ICP “is also known for its susceptibility to the problem of local minima”, and that “there is no reliable way to tell whether or not it is trapped in a local minimum”. Removing that uncertainty cost its authors a branch-and-bound search over SE(3), an exhaustive search of the whole pose space. With it they report 100% correct registration on all 2,000 randomly-initialised bunny and dragon tasks, every rotation error under 2 degrees. That is the price of a guarantee that local iteration cannot give.

Use range-aware and coordinate-aware evidence.

The density-and-coverage audit comes first, and weather belongs inside it with a figure attached rather than in a list of confounders. The Seeing Through Fog dataset paired 10,000 km of driving in Germany, Sweden, Denmark and Finland with controlled fog-chamber recordings. The chamber gives the physical limit: “On the other hand, active lidar sensors are highly degraded by scattering media as fog, snow or rain, limiting the maximal perceivable distance at fog densities below 50 m to 25 m”. That is the sensor losing range to physics, before any model sees a point.

A group at ETH Zurich then measured the detection cost on that same data, in 2021. A PV-RCNN trained on the clear-weather split scores 77.05 car 3D AP@0.5 IoU (moderate) on the clear test split, 70.17 on light fog and 46.00 on dense fog. PointPillars scores 74.64 / 68.47 / 35.23. The detector is unchanged. The air is not.

Verifying frames and time comes next, and it is where the corruption benchmarks put by far the largest losses, as the section above quantifies. Transforms, units, timestamps and ego-motion compensation deserve the first hour of a review, not the last.

Matching in three dimensions is a design decision, not a default. nuScenes changed the matching rule rather than the models it evaluates. It defines “a match by thresholding the 2D center distance d on the ground plane instead of intersection over union (IOU)”, because “objects with small footprints, like pedestrians and bikes, if detected with a small translation error, give 0 IOU”. mAP is then averaged “over matching thresholds of D = {0.5, 1, 2, 4} meters and the set of classes C”.

KITTI keeps overlap matching and stratifies its objects instead. Its 3D benchmark “consists of 7481 training images and 7518 test images as well as the corresponding point clouds”. Every annotated object is split into easy, moderate and hard by minimum bounding-box height (40, 25 and 25 px), maximum occlusion level and maximum truncation (15%, 30% and 50%). And “the official KITTI leaderboard is ranked by performance on moderate”. Two benchmarks, two definitions of a hit. The same detector is being asked two different questions.

Slicing by visibility — occluded, truncated, sparse and partially scanned objects reported apart — and testing downstream margins such as mapping drift, collision clearance, volume or localization outcomes close the sequence. A single averaged score hides every one of the gaps above.

Projected image features depend on calibration, synchronization, lens model, rolling shutter, and which surface each sensor observed; a color pixel may correspond to a different moment or depth layer than a lidar point.

Two independent groups have measured which sensor published fusion actually rests on. One of them corrupts a single modality at a time on nuScenes. TransFusion, clean at 66.9 mAP and 70.9 NDS, keeps 65.7 mAP when the camera inputs are corrupted — a robustness ratio of 0.98. Corrupt the lidar instead and it falls to 26.9, a ratio of 0.40.

The paper says so in plain words: “for existing fusion methods, the LiDAR modality is the main modality and the camera modality is auxiliary”. It then describes the limit case: “Especially, in the extreme case where all LiDAR points are missing, current fusion methods fail to predict any objects like the LiDAR-only method.” With the lidar removed entirely the fusion models return 0 mAP and 0 NDS, with every camera still working.

It does not take a total loss. On nuScenes-C, losing only part of the sweep, with all six cameras intact, takes PointPillars from 27.69 to 8.87 mAP, CenterPoint 59.28 → 20.84, FUTR3D 64.17 → 26.32, TransFusion 66.38 → 24.63 and BEVFusion 68.45 → 27.17.

Visualize projected correspondences across the frame and motion range; fusion gains should be compared with single-sensor baselines and sensor-dropout tests. That asymmetry is exactly what a single-modality baseline exposes and what an averaged benchmark number conceals.

A convincing point-cloud demo should survive these operational questions, and each of them has a published answer to be compared against.