LLM2R: From Large Language Models to 3D Reassemblers

AI summary

Overview

Research area: Computer Vision / 3D vision — 3D part assembly and geometric reassembly, bridging point-cloud representation learning with pretrained large language models.

Technical level: Advanced. The paper assumes familiarity with Transformers, causal attention, LoRA, SE(3) pose parameterization, quaternions, Chamfer distance, and point-cloud encoders such as Point-BERT.

Scope: The paper introduces LLM2R, a single point-cloud-only framework that repurposes a pretrained decoder-only LLM to predict one rigid SE(3) pose per part for both semantic part assembly and geometric fracture reassembly, and evaluates it on PartNet, Breaking Bad, and PartNeXt.

What This Paper Is About

3D part assembly asks a model to take a set of scattered, unassembled parts and predict the rotation and translation that puts each one back into its place in the complete object. The difficulty is that a part's correct placement is almost never determined by its own geometry — it depends on every other part — so methods built on pairwise matching or on repeated "generate and check" loops become slow and unreliable as objects gain parts. The authors' goal is to treat the whole assembly as one long sequence that a pretrained language model can reason over in a single forward pass.

Key Contributions

1. Assembly as causal sequence modeling. Each part is compressed into a block of geometric tokens by a Point-BERT encoder plus a Q-Former; all blocks are concatenated and processed by a LoRA-adapted pretrained decoder-only LLM (Qwen3.5-0.8B), yielding one point-cloud-only architecture for both semantic and geometric assembly that predicts all poses in a single non-autoregressive pass.
2. Learnable summary tokens. Because a causal decoder only lets a part see the parts before it, the method appends a block of S learnable summary tokens after the last part. These tokens attend to the entire assembly at a constant token cost, and a part–summary fusion head conditions each part's pose on them without modifying the attention mask or the pretrained decoder.
3. Strong empirical results across three benchmarks. Highest part accuracy and success rate on all three PartNet categories, best rotation error, translation error, and part accuracy on PartNeXt, and lowest translation error and fastest inference on Breaking Bad (where only the fracture-specialized GARF is more accurate).
4. Scalability and transfer evidence. The advantage over baselines grows with part count (33.5 points over GARF and 29.1 points over Assembler at 100 parts) and the model transfers zero-shot to unseen categories with more than 16 points higher part accuracy than the strongest of three baselines.

Main Findings

• Breaking Bad (volume-constrained Everyday / Artifact): LLM2R reaches RMSE(T) of 0.46 and 0.53 against 1.34 and 1.77 for GARF, and the fastest inference at 0.092 s and 0.094 s per instance (GARF 0.225 / 0.239 s; Jigsaw 0.185 / 0.171 s). GARF stays ahead on rotation error, PA (94.77% and 93.34% versus 92.57% and 90.11%) and CD, but is fracture-specialized and pretrained on fracture surfaces.
• PartNet (Chair, Table, Lamp): Highest PA on all three categories — 71.42, 70.67, and 52.31 — which is 2.2, 2.2, and 8.3 points above the strongest baseline, and the highest SR among methods that report it (41.40% on Chair versus 22.59% for Assembler). Hierarchical methods (3DHPA, CFPA) still achieve lower CD on every category.
• PartNeXt: Best on all three pose-centric metrics — rotation error 41.12° (versus 45.12 for CRAG), translation RMSE 5.24 (versus 9.13), and PA 67.93% (versus 65.89%) — but CD of 6.37 trails CRAG (2.40) and GARF (3.10). The authors argue CD rewards tight local surface fit while RE/RMSE/PA measure whether each part is correctly posed.
• Zero-shot transfer: Trained only on Chair, Table, and Lamp and tested on Bed, Door, and Storage Furniture, LLM2R reaches PA of 31.03%, 32.49%, and 26.35%, at least 16.7 points above the best baseline (CFPA at 10.29%, 15.78%, and 9.13%), with the lowest CD and lowest rotation error on every category.
• Scaling with part count: With key-value cache reuse, total inference time for 100 PartNet instances grows nearly linearly from 6.33 s at 10 parts (63 ms per instance) to 47.54 s at 100 parts, versus 103.37 s when recomputing the prefix. On Breaking Bad, GARF leads at 30 fragments (83.61% versus 80.13%) but falls to 24.33% at 100 fragments while LLM2R holds 57.85% (33.5 points higher). On PartNet, LLM2R and Assembler are close at 30 parts (52.57% versus 49.34%), but Assembler drops to 7.78% at 100 parts while LLM2R retains 36.83% (29.1 points higher).
• Ablations (PartNet Chair subset): Removing the summary block is the most damaging change for part accuracy (−7.7 points, CD up 2.31, CA nearly unchanged), indicating it mainly improves absolute placement. Replacing the Q-Former with an MLP projector costs 4.9 PA points and 3.5 CD. Removing the natural-language instruction costs 6.1 PA points and 6.6 SR points — the largest SR drop — which the authors cite as direct evidence that language pretraining, not just capacity, contributes.
• Token budget: Quality improves monotonically with the number of geometric tokens K (PA 68.37 at K=16, 70.88 at K=32, 71.42 at K=64, 71.55 at K=128) and saturates beyond K=64, where doubling the token budget buys only 0.13 PA points and 0.08 CD.
• Position in the sequence matters: On 20-part Chair instances, PA is highest for the first two positions (83.4% and 80.9%), between 61.5% and 70.9% for positions 5 to 17, and lowest at positions 18 to 20 (56.4% to 59.6%) — a gradual decline of about 24 points with no collapse at the tail.

Methodology in Plain English

The system turns a pile of parts into a sentence-like sequence and reads the answer off a language model.

• Encode each part. Every part point cloud runs through a frozen Point-BERT encoder (pretrained on ShapeNet) that cuts it into G=64 local neighborhoods of 32 points each. A Q-Former then squeezes those patch descriptors into K=64 compact tokens living in the language model's 1024-dimensional embedding space. The same queries are shared across all parts.
• Build the sequence. The token blocks for all N parts are laid end to end inside a chat-formatted instruction, followed by S=64 learnable summary tokens and a closing template. Parts appear in dataset order with no ordering heuristic.
• Run the decoder once. A causal decoder (the Qwen3.5-0.8B language model, 24 layers, three gated linear-attention layers per one full softmax-attention layer) processes the sequence in a single forward pass. Only LoRA adapters (r=32, α=64) on the attention projections, plus the Q-Former, summary block, and pose head, are trained; the base LLM stays frozen.
• Fix the causality problem. Since a causal model lets a part see only earlier parts, the summary tokens — placed after everything — act as a per-object digest of the whole assembly. A fusion head pairs each part token with the corresponding summary token, passes the pair through a shared linear layer with GELU, and averages over the block to get one feature per part.
• Predict poses. Rotation is produced as a softmax-weighted, re-normalized convex combination of 32 fixed anchor quaternions (which can reach any unit quaternion, so it is a continuous parameterization rather than a classifier); translation comes from a small MLP.
• Train with geometry-aware losses. The objective combines a sign-invariant quaternion loss, a translation MSE, a Chamfer distance between the predicted and target placements (computed on 1,024 random points per part, which sidesteps symmetry ambiguities), and a repulsion term for interchangeable parts (grouped by sorted bounding-box extents and covariance spectrum) that penalizes predicted spacings smaller than the target spacings. All loss weights are 1 and no language-modeling loss is used.
• Train efficiently. 100 epochs (about 10 hours) on 8 NVIDIA H200 GPUs, batch size 32, AdamW at learning rate 10⁻³ in bfloat16, context limit 8,192 tokens. The point encoder stays frozen for the first 80 epochs, during which 42.2M parameters are trainable, then is fine-tuned jointly for the last 20.

Why This Matters

The paper shows that a pretrained language backbone can be repurposed for continuous 3D geometric reasoning — not just text generation or 3D question answering — and that a small block of appended tokens can supply the global context that causal attention otherwise denies. It also reframes scalability: instead of iteratively generating and verifying candidate assemblies, one long-context forward pass handles up to roughly 125 parts at K=64 within the 8,192-token limit, with key-value caching enabling incremental assembly as parts arrive.

Real-world applications:

• Furniture and flat-pack assembly guidance, where a robot or user is handed loose parts with no CAD template.
• Robotic manipulation and manufacturing, where a bin of components must be put back together.
• Archaeological reconstruction of shattered artifacts and fragments.
• Restoration of broken everyday objects from scanned fragments.

Industry relevance: The single-pass, non-autoregressive design gives the lowest inference time among compared methods on Breaking Bad (0.092 s per instance), and the LoRA-based adaptation means a modest 0.8B backbone with 42.2M trainable parameters in the main training stage — an attractive cost profile for deployment on assembly-line or robotic systems where latency and retraining budget matter.

Future Directions

• Order invariance. The model is not permutation invariant because reordering changes the prefixes seen by part and summary states; order-invariant position encodings are proposed as a next step.
• Beyond the context window. About 125 parts is the ceiling at K=64 with an 8,192-token limit, so larger backbones or longer contexts would be needed for very large assemblies.
• Multiple plausible assemblies. The model regresses a single pose per part and so cannot express several valid configurations; distribution-learning or sampling formulations are the natural extension.
• Geometric refinement. The authors propose a lightweight geometric refinement of predicted poses to close the Chamfer-distance gap, which they attribute to the few parts placed far from the surface, and note that PCA-derived pose targets remain ambiguous up to part symmetry and are only partly compensated by the Chamfer objective.

Target Audience

Researchers and graduate students in 3D vision, geometric deep learning, and robot perception who work on shape assembly, pose estimation, or part-based reconstruction; practitioners interested in adapting pretrained language models to continuous 3D outputs; and engineers building assembly, manipulation, or restoration pipelines who need a fast, template-free method that scales to objects with many parts. Readers without a background in Transformers and SE(3) geometry will find the method sections dense, though the problem framing and results tables are accessible.

Authors’ abstract

3D part assembly aims to reconstruct a complete shape from a set of unassembled parts by predicting a rigid SE(3) transformation for each part. Because the placement of a part is rarely determined by its own geometry, the task demands reasoning over the entire part set, and existing methods, which rely on pairwise relations or iterative generation and verification, scale poorly to objects with many parts. We present LLM2R, which casts part assembly as sequence modeling with a pretrained decoder-only large language model (LLM): a Point-BERT encoder and a Q-Former compress each part into a block of geometric tokens, the blocks of all parts are concatenated, and a causal LLM adapted with Low-Rank Adaptation (LoRA) processes the sequence in one forward pass. Causal attention, however, lets the hidden states of a part observe only the parts that precede it. LLM2R therefore appends a block of learnable summary tokens after the last part: these tokens follow every part, so their states aggregate the whole assembly at a constant token cost, and a part-aware pose head fuses each part's states with them before predicting its rotation and translation. One architecture thus handles semantic and geometric assembly and predicts all poses in a single non-autoregressive pass. LLM2R achieves the highest part accuracy and success rate on all three PartNet categories, the best rotation error, translation error, and part accuracy on PartNeXt, and the lowest translation error and fastest inference on Breaking Bad, where only a fracture-specialized method is more accurate. Its advantage grows with the number of parts: at 100 parts it exceeds GARF by 33.5 points in part accuracy on Breaking Bad and Assembler by 29.1 points on PartNet, and it transfers zero-shot to unseen categories with more than 16 points higher part accuracy than the strongest of three baselines.