Alterbute: Editing Intrinsic Attributes of Objects in Images

AI summary

Overview

Research area: Computer vision, specifically diffusion-based generative image editing with object identity preservation.

Technical level: Intermediate. The paper assumes familiarity with diffusion models, latent-space conditioning, and self-attention architectures, though the core idea is explained conceptually.

Scope: The paper introduces Alterbute, a supervised diffusion method that edits an object's intrinsic attributes (color, texture, material, shape) in a single model while preserving its perceived identity and scene context.

What This Paper Is About

Image editing systems are good at changing how an object looks from the outside (camera angle, lighting, background) but struggle to change what the object fundamentally is — its color, texture, material, or shape — without turning it into a different object. The core problem is that "identity" sits on a spectrum: defining it by broad category ("car") permits almost any edit, while defining it at the instance level permits almost none. Alterbute aims to find the middle ground, editing intrinsic attributes while keeping the object recognizable as the same thing.

Key Contributions

1. A single unified model for intrinsic attribute editing. Alterbute edits color, texture, material, and shape in one model, rather than requiring separate systems per attribute type as prior attribute-specific editors do.

2. Visual Named Entities (VNEs) as a definition of object identity. VNEs are fine-grained visual categories such as "Porsche 911 Carrera" or "IKEA LACK table" that group objects sharing identity-defining features while still allowing natural variation in intrinsic attributes. The authors also contribute an automated VLM-based pipeline for extracting these labels at scale.

3. A relaxed training objective. Instead of training only on intrinsic edits (for which paired data essentially does not exist), the model trains on both intrinsic and extrinsic changes conditioned on an identity reference, a text prompt, and a background/mask pair. At inference, extrinsic changes are restricted by reusing the original background and mask, so only intrinsic attributes change.

4. State-of-the-art results on identity-preserving intrinsic editing, measured through human preference and VLM-based evaluation against seven baselines.

Main Findings

• User preference favors Alterbute across all baselines. In a user study with 166 U.S.-based participants on CloudResearch, preference rates over baselines were: FlowEdit 89.3%, MimicBrush 85.0%, InstructPix2Pix 85.0%, OmniGen 81.2%, UltraEdit 80.0%, MaterialFusion 79.7%, and Diptych 76.2%.

• VLM judges agree with humans. Gemini preference rates ranged from 76.8% (Diptych) to 94.3% (MimicBrush); GPT-4o ranged from 74.8% (Diptych) to 89.8% (MimicBrush); Claude ranged from 77.8% (Diptych) to 92.6% (MimicBrush and FlowEdit).

• Results are statistically significant. A binomial test returned p-values below 1e-10 for all user comparisons, below 1e-12 for all Gemini and GPT-4o comparisons, and between 1e-12 and 6e-12 across Claude comparisons.

• The grid conditioning mechanism is structurally necessary. Replacing the 1×2 spatial grid (which enables cross-image self-attention) with channel-wise concatenation causes the model to produce no-ops at inference — it fails to apply the requested edits and outputs the source largely unchanged.

• VNE identity definition outperforms alternatives. Comparing identity reference strategies — DINOv2 feature retrieval (top-5 cosine-similar objects), instance-retrieval features, in-place editing (using the target image as its own reference), and VNE sampling — the paper reports that instance-retrieval features lack sufficient intrinsic variation, DINOv2 clusters visually similar but identity-distinct objects, and in-place editing fails to decouple identity from attributes.

• VNE curation scale. Applying the pipeline to OpenImages (~9 million images, 16 million object bounding boxes) assigned VNE labels to around 1.5 million objects. After discarding singleton clusters, 69,744 VNE clusters remained, comprising 1,079,442 labeled images. Cluster sizes follow a heavy-tailed distribution; clusters with more than 3,000 instances are grouped into the last histogram bin.

• Alterbute is the only compared method capable of identity-preserving object reshaping, though the paper notes reshaping rigid objects does not always yield desired results.

• Limitations identified by the authors: slight background inconsistencies when using coarse bounding box masks; difficulty reshaping rigid objects whose shape is correlated with identity; potential annotation bias in the VNE pipeline from Gemini's training distribution; and a limited benchmark scale of 30 objects and 100 editing cases.

Methodology in Plain English

The authors fine-tune a pretrained SDXL latent diffusion model (7B parameters) rather than training from scratch. The model is fed three things: a reference image of the object (background masked out, sampled from the same VNE cluster as the target), a text prompt describing the desired intrinsic attributes, and a scene description consisting of a background image plus a binary mask marking where the object should go.

To let the model transfer identity from the reference to the target, the inputs are arranged side-by-side in a 1×2 grid at 512×512 per panel, producing a 512×1024 composite. The left half holds the noisy latent of the target; the right half holds the reference. Self-attention layers in the UNet propagate features across the two halves. The diffusion loss is computed only on the left half.

The key trick is the relaxed training objective. Rather than requiring pairs of the same object with only intrinsic changes — data that essentially does not exist — the authors train on pairs where both intrinsic and extrinsic attributes vary, because such pairs are abundant. At inference, external variation is suppressed by reusing the input image's own background and mask, so only the requested intrinsic attribute changes.

Training data comes from an automated pipeline: Gemini is prompted to assign a VNE label to each detected object in OpenImages based on visual appearance, and to describe each object's color, texture, material, and shape as structured key-value pairs. Those descriptions become the training text prompts. To support reshaping, the model randomly alternates between precise segmentation masks and coarse bounding-box masks during training.

Training ran for 100,000 steps with a learning rate of 10^-5 and batch size 128, taking approximately 24 hours on 128 v4 TPUs. The identity reference is randomly dropped in 10% of samples and the text prompt in another 10%, to force robustness. Classifier-free guidance scales of 7.5 (text) and 2.0 (image) follow InstructPix2Pix. Segmentation masks come from a pretrained model (Ravi et al., 2024). At inference, the user supplies a single key-value prompt for the one attribute to change, and the model modifies only that attribute.

Evaluation used a purpose-built set of 30 objects (15 popular objects from prior literature, primarily from Ruiz et al. 2023 and Gal et al. 2023, plus 15 from underrepresented categories like furniture and vehicles), paired with multiple prompts for 100 total editing samples.

Why This Matters

Impact on research. The paper reframes identity preservation as a spectrum between coarse category and rigid instance, and proposes VNEs as a principled middle point. The relaxed training objective is a general recipe for supervised learning on tasks where aligned paired data does not exist — an obstacle that has pushed much of this field toward unsupervised priors. Alterbute shows a supervised alternative is feasible, and does so on a simpler backbone (SDXL) than competitors like Diptych and FlowEdit, which build on FLUX.

Real-world applications:

• Product photography and e-commerce: re-rendering a product in a new color, material, or finish without reshooting, while keeping it recognizable as the same SKU.
• Interior design and furniture visualization: showing the same chair or table in a different upholstery, wood, or shape.
• Automotive and industrial configurators: previewing vehicle trim, paint, and material options on an existing photo.
• Creative and advertising workflows: rapid iteration on visual concepts where the object must stay on-brand.

Industry relevance. The paper is authored by researchers at Google and the Hebrew University of Jerusalem, with two equal-contribution authors. Google develops Gemini, which was used for data curation and as one of three VLM judges; the authors disclose this, noting Gemini was not used for model training or inference. The method targets non-human objects only — products, furniture, vehicles — and the impact statement notes it is not designed or trained on human faces or biometric features, which limits its applicability for deepfake or identity-fraud misuse.

Future Directions

• Expanding the benchmark. The authors explicitly call out the 30-object, 100-case evaluation set as limited in scale and identify expanding it with more objects, attribute combinations, and editing scenarios as an important future direction.

• Improving shape editing for rigid objects. Reshaping rigid objects remains challenging because geometry is often correlated with identity-defining features; generated shapes can lack realism. The paper suggests pre-removing the object to provide a clean background as a remedy for a related artifact issue.

• Reducing VNE annotation bias. Because VNE labeling relies on Gemini, categories prominent in its training distribution may get better coverage. The authors note high-confidence filtering mitigates spurious labels and that the pipeline is fully replicable via the prompts in Appendix F.

• Moving beyond limited-scale benchmarks and proprietary comparisons. The authors note that some proprietary systems can perform comparable intrinsic manipulations but lack published papers, so the evaluation focuses on open academic baselines — an open question is how the method stacks up against unpublished systems.

Target Audience

This paper is most useful to computer vision researchers and graduate students working on diffusion-based image editing, subject-driven generation, and identity preservation. It also suits applied machine learning engineers building generative editing tools for product, furniture, or automotive imagery, who need to understand both the mechanics and the limits of intrinsic attribute editing. Practitioners familiar with diffusion models will get the most out of the architectural details; those newer to the area can still follow the conceptual framing of identity as a spectrum between category and instance.

Authors’ abstract

We introduce Alterbute, a diffusion-based method for editing an object's intrinsic attributes in an image. We allow changing color, texture, material, and even the shape of an object, while preserving its perceived identity and scene context. Existing approaches either rely on unsupervised priors that often fail to preserve identity or use overly restrictive supervision that prevents meaningful intrinsic variations. Our method relies on: (i) a relaxed training objective that allows the model to change both intrinsic and extrinsic attributes conditioned on an identity reference image, a textual prompt describing the target intrinsic attributes, and a background image and object mask defining the extrinsic context. At inference, we restrict extrinsic changes by reusing the original background and object mask, thereby ensuring that only the desired intrinsic attributes are altered; (ii) Visual Named Entities (VNEs) - fine-grained visual identity categories (e.g., ''Porsche 911 Carrera'') that group objects sharing identity-defining features while allowing variation in intrinsic attributes. We use a vision-language model to automatically extract VNE labels and intrinsic attribute descriptions from a large public image dataset, enabling scalable, identity-preserving supervision. Alterbute outperforms existing methods on identity-preserving object intrinsic attribute editing.