InstructMesh: Selective Refinement of Generative 3D Models for Fabrication

AI summary

Overview

Research area: Human-computer interaction (HCI) at the intersection of generative 3D modeling and personal digital fabrication, specifically post-generation repair of AI-generated 3D meshes for 3D printing.

Technical level: Intermediate — the paper assumes some familiarity with 3D generative pipelines (encoders, latent voxel grids, decoders) and HCI study methodology, but explains its system in accessible terms.

Scope: The paper presents InstructMesh, a tool that lets novices repair fabrication-relevant geometric flaws in generative 3D models by applying targeted operations to the model's intermediate latent voxel representation, with two interaction modes (natural language via an LLM and parameterized sliders) and a preview visualization. Note that the provided content is truncated during the description of the second user study, so that study's results are not reported here.

What This Paper Is About

Text- and image-to-3D generative models produce models that look convincing but are often unusable when 3D printed — they may have sealed openings, thin walls, fused parts, or disconnected bridges, because these models are trained with image-based supervision that prioritizes appearance over fabrication-ready geometry. Fixing such flaws normally requires expert mesh-editing skills in tools like Blender or MeshMixer, and simply re-prompting the generator is unreliable. InstructMesh lets novices highlight a problematic region on a generated mesh and apply a targeted latent-space edit (for example, opening a sealed lid or thickening a wall) so the model can be decoded into a repaired, functional mesh.

Key Contributions

1. A formative analysis (N = 120 models) identifying common fabrication-relevant geometric flaws in outputs from a state-of-the-art 3D generative model, resulting in a nine-category taxonomy with per-category prevalence rates.
2. A set of canonical latent-space operations (Extrude, Expand, Fill, Trim, Erode, Flatten) that repair those flaws without requiring mesh-manipulation expertise, plus an interaction design combining natural language input, parameterized slider controls, and a preview visualization that shows additive edits in green and subtractive edits in red.
3. A technical evaluation calibrating twelve presets across the six operations on an 80% test split, measuring flaw-correction success, runtime, and LLM operation-selection accuracy.
4. Two user studies (N = 12 each) — the first demonstrating that novices can identify and repair visually identifiable flaws with InstructMesh, and the abstract reporting that hybrid workflows combining sliders with natural language input, alongside preview visualizations, improve usability and transparency.

Main Findings

• Fabrication flaws are pervasive in generative outputs. Across 120 reconstructed models, 94 models (78.3%) exhibited more than one distinct type of issue, with an average of 2.4 issues per model (SD = 1.10). The most common were Extraneous Artifacts (65.8%), Missing Openings (48.6%), and Hollowing Errors (36.9%); the least common were Broken or Missing Bridges (10.8%), Topological Errors (9.9%), and Truncated Features (9.9%). Fused Joints affected 13.5%, Fused / Duplicated Features 27%, and Wall Thickness Issues 20.7%.
• Latent-space operations repair most flaws. On the 80% test split, Missing Openings were repaired at 96.3% and Wall Thickness Issues at 95.65%; the lowest success rate was Fused / Duplicated Features at 83.33%.
• Edits stay under a minute. Trellis image-to-3D generation averaged 31.7s (SD = 3.7s); adding region-specific latent operations raised this to 43.7s (SD = 5.2s).
• Constrained LLM prompting maps language to operations accurately. Using in-context learning with GPT-4, the system achieved 92.1% accuracy in predicting the correct canonical operation on the 80% test split, without fine-tuning.
• Novices can spot flaws. In Study 1, 12 participants identified 90.4% of fabrication-relevant flaws. Detection varied significantly by model (F(4,55)=3.06, p < 0.05; flaws in M1 were easier to detect than in M5, p < 0.05) and by flaw type (F(5,139)=19.24, p < 0.0001), with topological issues hardest to detect and extraneous artifacts and missing openings most readily identified.
• Repair time depends on model complexity and flaw type. Model had a significant effect on repair time (F(4,55)=7.27, p < 0.001), with simpler models such as M1 repaired faster than M3, M4, and M5. Flaw type also mattered (F(5,121)=5.43, p < 0.001): truncated features took 3.00 ± 0.95 min, significantly longer than topological errors (1.50 ± 0.52 min, p < 0.01) and fused features (1.84 ± 0.75 min, p < 0.05); missing openings (2.33 ± 1.24 min) also took longer than topological errors (p < 0.05).
• Repairs were largely successful. An independent expert judged 89.7% of flaws successfully repaired, with only 16 failures out of 156 annotated flaws. Flaw type had a significant effect (F(5,121)=2.33, p < 0.05) but post-hoc tests revealed no significant differences between specific categories, and model had no significant effect (F(4,55)=2.25, p > 0.05).
• Novices' self-assessments matched expert judgment. No significant difference was found (F(1,587)=0.40, p > 0.05); only 11 cases (7.1%) showed disagreement where participants believed a flaw was fixed but the reviewer did not.
• Participants valued the low barrier to entry. P2 noted "I like that you can undo your changes, and I just have to describe my changes in text. It's very easy to get used to the tool." P6 said "I don't feel the need to learn CAD software," and another participant commented "First time 3D modeling - Didn't know it could be easy!"
• Study 2 results are not reported in the available content. The second study compared an LLM interface against a slider interface and examined preview visualization, but the provided text stops during the task description, so its findings cannot be summarized.

Methodology in Plain English

The researchers started by diagnosing the problem. They took the 100 most popular "things" from Thingiverse (120 unique 3D models after preprocessing), rendered one image of each, and reconstructed it with the generative model Trellis. Two expert annotators then compared each reconstruction to the original, described every issue in plain language, iteratively grouped descriptions until nine stable flaw categories emerged, and finally mapped every issue back to those categories.

Based on that taxonomy, they built InstructMesh on top of the Trellis two-stage pipeline, which encodes a shape into a sparse voxel grid (64³) and decodes it into a textured mesh. Rather than editing the mesh, the tool edits the intermediate voxel grid and re-decodes it, letting the model regenerate texture and surface detail while the user's geometric correction is respected. When a user paints a region, the system aligns the mesh bounding box to the voxel grid and extracts the selected faces as sub-meshes, which serve as geometric references for six operations — three additive (Extrude, Expand, Fill) and three subtractive (Trim, Erode, Flatten). Tolerance to imprecise brushing comes from discarding connected components below a minimum size, acting on a proximity radius around the selection, and rendering previews directly from modified voxels without invoking the decoder.

Natural-language instructions are handled by GPT-4 using in-context learning: the model does not touch geometry, it only picks an operation and parameters from a fixed, pre-parameterized vocabulary, with examples drawn from a 20% development split of the formative dataset. The team manually calibrated twelve presets across the six operations using that development set, then evaluated on the remaining 80%, using an independent reviewer to judge success at the flaw level. Finally, they ran two studies with 12 participants each: the first on whether novices can identify and repair flaws, the second on interaction mode (LLM versus sliders) and preview visualization.

Why This Matters

The work targets a gap between generative 3D models that look good and objects that actually work when printed. By making fabrication-relevant repair accessible through region selection rather than mesh surgery, it points toward generative design tools that produce physically usable artifacts, not just visually plausible ones.

Real-world applications:

• Rapid prototyping, where a designer needs a functional part quickly without learning CAD.
• Hobbyist 3D printing, letting makers fix downloaded or generated models before printing.
• Educational settings, where students can explore 3D modeling and fabrication concepts without a steep software learning curve.
• Any downstream use requiring functional geometry — for example, parts meant to contain liquids, support airflow, or bear structural loads.

Industry relevance: The taxonomy of the nine flaw categories gives a concrete diagnostic vocabulary for evaluating and improving 3D generative models, particularly those trained with image-based supervision. The constrained LLM-to-operation mapping also offers a pattern for making generative tools more reliable in human-AI interaction, where scoping an LLM to a fixed operation set keeps edits interpretable and bounded.

Future Directions

• Dynamic and mechanical parts. The authors explicitly scope out Fused Joints and note that current generative methods do not yet support dynamic mechanisms such as hinges or interlocking parts requiring precise geometric and mechanical constraints — extending the operation vocabulary to these is an open problem.
• Better detection of subtle flaws. Topological errors were the hardest flaws for novices to identify, even though they were relatively quick to repair once recognized; tooling that surfaces hard-to-see flaw types could close this detection gap.
• Expanding the operation vocabulary and backends. Because the approach uses in-context learning rather than fine-tuning, new operations can be added by extending the example set, and the authors state the LLM component is model-agnostic — any modern LLM could serve as backend.
• Generalizing beyond the study setup. The formative analysis and reconstruction experiments were scoped to Trellis, even though the authors note nearly all current 3D generative methods use a similar two-stage encoder-decoder pipeline with image-based training objectives. Validating the latent operations across other backbones (including models such as SPAR3D) remains to be shown.

Target Audience

HCI researchers working on human-in-the-loop generative systems; graphics and 3D generative modeling researchers interested in controllability and fabrication constraints; computational fabrication and personal fabrication researchers; and practitioners building design tools who want a concrete flaw taxonomy and an interaction pattern — region selection, constrained LLM operation mapping, and latent-space previews — for making generative outputs physically usable.

Authors’ abstract

Recent advances in generative AI allow users to create 3D models from text or images. However, these models prioritize visual plausibility over geometric accuracy, often generating results with flaws that compromise their intended use post-fabrication. We present InstructMesh, an interactive post-generation refinement tool that enables selective repair of generative 3D models through region selection and targeted operations, such as opening or sealing voids, or adjusting local thickness. Users can invoke edit operations via natural language prompts or slider controls. By operating directly on the intermediate latent representation, InstructMesh allows users to apply robust geometric corrections without requiring expert modeling skills. To inform our design, we first analyze common fabrication-related failure modes in outputs from state-of-the-art generative tools. We then conduct two user studies, demonstrating that novices can identify and perform fabrication-relevant repairs on generative outputs using InstructMesh, and revealing user preference for hybrid interfaces that combine slider controls with natural language input.