Pointy - A Lightweight Transformer for Point Cloud Foundation Models

AI summary

Overview

Research area: 3D computer vision — point cloud representation learning and foundation models.

Technical level: Intermediate. The paper assumes familiarity with transformer architectures, point cloud benchmarks, and standard training protocols, but its core argument is accessible.

Scope: This paper introduces Pointy, a tokenizer-free, lightweight transformer backbone for point cloud classification, and validates it through a replication study that standardizes training conditions across several competing architectures.

What This Paper Is About

Point cloud research has drifted toward ever-larger foundation models trained with cross-modal supervision from images and text, using datasets of hundreds of thousands to over a million samples. Because training protocols, preprocessing, and hyperparameters vary across papers, it is hard to tell whether performance gains come from better architectures, more data, or lucky tuning. The authors build a deliberately small transformer trained on only 39k point clouds and run a controlled, unified comparison to test how far a simple design can go without large-scale cross-modal supervision.

Key Contributions

1. A comprehensive replication study. A benchmarking system for point cloud foundation models that evaluates popular transformer backbones under identical training regimes and evaluation metrics, giving an apples-to-apples comparison of their strengths and limitations.
2. A lightweight yet competitive model. A simple transformer-based backbone that performs well without relying on massive cross-modal supervision or large datasets.
3. Controlled pre-training. Evidence that limited data can compete with much larger data-driven approaches, offering practical insight into scaling strategies and design trade-offs.
4. Open release. Implementation including code, pre-trained models, and training protocols is available at https://github.com/KonradSzafer/Pointy.

Main Findings

• Small model beats bigger-data models. Pointy is trained on 39k point clouds yet outperforms several larger foundation models trained on over 200k training samples, and approaches state-of-the-art results from models that have seen over a million point clouds, images, and text samples.
• Classification accuracy (Table 1). Under identical conditions (batch size 16, 100 epochs, points normalized to [-1,1], random z-axis rotation), Pointy-Small (3.0M parameters, 16.2G FLOPs) reaches 90.4% on ModelNet40 and 80.0% on ScanObjectNN; Pointy-Base (19.4M parameters, 18.0G FLOPs) reaches 90.6% and 78.5%. Classical methods are strong in this regime: PointNet and PointMLP both reach 91.6% on ModelNet40, and PointNet++ reaches 81.7% on ScanObjectNN.
• Pre-training on Objaverse-LVIS (Table 2). After 30 epochs on the 39k-sample subset, Pointy-Small scores 36.4% and Pointy-Base 36.3%, versus PCT at 36.3%, PointMAE at 34.9%, PointTransformer [31] at 34.1%, and PointTransformer [5] at 8.5%.
• Zero-shot transfer (Table 3). Using the 30th-epoch checkpoint with the classification head discarded, Pointy-Base achieves ModelNet40 Top-1 of 85.3% and ScanObjectNN Top-1 of 58.3%, the best among the reproduced models. Uni3D remains highest, at 88.2% and 65.3% respectively, but uses a dataset of nearly one million 3D shapes, 10 million images, and 70 million texts.
• Faster convergence. Training dynamics plots show Pointy converging faster than PCT, PointMAE, and both PointTransformer variants across ModelNet40, ScanObjectNN, and Objaverse-LVIS. PointTransformer [5] consistently underperforms relative to the other methods on all datasets.
• Resolution scaling (Figure 3). PCT performs better in the 256–1024 point range, while Pointy reaches a peak of 89.3% at 2048 points.
• Merging strategy matters (Table 4). Simple token embedding addition (88.9%) outperforms a linear merging layer (88.0%) in the hierarchical transformer, suggesting additive merging preserves geometric relationships without unnecessary parameters.
• Non-hierarchical is competitive. Removing the hierarchical transformer still yields 89.3%, which the authors attribute to the patching scheme and robust positional embeddings guiding attention.
• Configuration effects. Increasing embedding dimension and attention heads consistently improves performance; the best reported configuration in the ablation (90.0%) uses 32 patches with 64 points per patch, though the authors chose 64 patches to balance computational efficiency and representational capacity.
• Positional embeddings help. Removing positional embeddings still gives 89.6%, indicating they are helpful but not essential in this setup.

Methodology in Plain English

Pointy processes raw point clouds directly, skipping the separately trained tokenizers that many competing models rely on.

The pipeline works like this. From a raw point cloud, the model picks anchor points using Farthest Point Sampling, then gathers each anchor's k-nearest neighbors to form local patches. A simplified PointNet-style embedding layer turns each neighborhood into a feature vector. Residual connections let the original coordinates travel alongside the learned features so geometric information is not lost, and a learnable positional embedding encodes where each patch sits spatially.

Those patch tokens go into a hierarchical transformer with six layers of multi-head self-attention. Between transformer blocks, adjacent tokens are merged so the representation progressively coarsens, enabling both local and global feature learning. The core transformer block is left unchanged. The embedding-dimension-to-attention-head ratio is kept low (about 3:1) as an inductive bias suited to 3D coordinate structure. GeLU activations, standard layer normalization, and Kaiming initialization are used throughout. The final output is a P×D dimensional representation, where P is the number of patches and D the embedding dimension. The model accepts either ℝ^(3×N) (coordinates only) or ℝ^(6×N) (coordinates plus normals or color).

Two variants exist: Small (192-dimensional embedding, ~3M parameters) and Base (embedding dimension 510, 20M parameters).

For fair comparison, all methods share one training environment: 2,048 points per shape, normalization to [-1,1], random z-axis rotation as the only augmentation, AdamW at a fixed learning rate of 1×10⁻⁴ with no scheduling, batch size 16, cross-entropy loss, and no voting-based inference. Training runs 100 epochs on ModelNet40 and ScanObjectNN, and 30 epochs on the Objaverse-LVIS subset. Evaluations use ModelNet40 (12,308 CAD models, 40 classes), ScanObjectNN (2,890 real-world scans, 15 categories), and a 39k-sample Objaverse-LVIS subset spanning 1,156 LVIS-annotated categories with an 85%/15% train-test split.

For zero-shot evaluation, the classification head is discarded, final-layer features are extracted from the 30th-epoch checkpoint, and test samples are classified by cosine similarity to class prototypes computed as means from the target dataset's training set.

Why This Matters

Impact on research. The paper argues that data curation, end-to-end optimization, and architecture choice may matter more than sheer dataset scale or cross-modal supervision. Its unified benchmarking framework isolates architectural effects that the literature usually confounds, and it releases code, protocols, and pre-trained weights to support reproducible comparison.

Real-world applications.

• Robotics, where lightweight 3D perception matters for on-board compute constraints.
• Augmented reality, which needs real-time understanding of scanned environments.
• Computational design, where CAD-style 3D models are classified or retrieved.
• Real-world scanning pipelines, since ScanObjectNN uses noisy, cluttered, occluded scans representative of sensor data.

Industry relevance. A 3.0M-parameter model at 16.2G FLOPs that is competitive with far larger models is attractive for deployment where memory and compute are limited. The finding that clean, well-labeled, curated data can substitute for enormous noisy datasets also has direct implications for data collection budgets.

Future Directions

• Denser prediction tasks. The authors note that training under a classification objective may limit the model's ability to exploit 3D geometry for finer-grained tasks, and propose investigating semantic and instance segmentation in real 3D scenes.
• Scaling the training data. Testing whether the approach holds up on larger datasets is listed as future work.
• Pre-training and multimodal strategies. The authors want to verify pre-training strategies and multimodal alignments, such as with images or text, inside the same controlled experimental framework.
• Generalization to harder sensor data. Pre-training on a curated, less noisy dataset may require additional fine-tuning for optimal performance on challenging real-world scanner data — an open question the paper raises but does not resolve.

Target Audience

Researchers and practitioners in 3D computer vision and point cloud representation learning, particularly those interested in efficient model design and reproducible benchmarking. It is also useful for engineers deploying 3D perception on resource-constrained hardware, and for anyone evaluating whether large-scale cross-modal pre-training is justified relative to a well-controlled smaller-scale alternative. Readers should be comfortable with transformer terminology, standard point cloud benchmarks, and classification metrics.

Authors’ abstract

Foundation models for point cloud data have recently grown in capability, often leveraging extensive representation learning from language or vision. In this work, we take a more controlled approach by introducing a lightweight transformer-based point cloud architecture. In contrast to the heavy reliance on cross-modal supervision, our model is trained only on 39k point clouds - yet it outperforms several larger foundation models trained on over 200k training samples. Interestingly, our method approaches state-of-the-art results from models that have seen over a million point clouds, images, and text samples, demonstrating the value of a carefully curated training setup and architecture. To ensure rigorous evaluation, we conduct a comprehensive replication study that standardizes the training regime and benchmarks across multiple point cloud architectures. This unified experimental framework isolates the impact of architectural choices, allowing for transparent comparisons and highlighting the benefits of our design and other tokenizer-free architectures. Our results show that simple backbones can deliver competitive results to more complex or data-rich strategies. The implementation, including code, pre-trained models, and training protocols, is available at https://github.com/KonradSzafer/Pointy.