HYPE-EDIT-1: Benchmark for Measuring Reliability in Frontier Image Editing Models

AI summary

Overview

Research area: Computer Vision — evaluation and benchmarking of instruction-following image editing models, with an emphasis on reliability and operational cost.

Technical level: Intermediate. The paper is readable without a deep learning background, but it uses pass@k-style reliability metrics, a retry-cap model, and an explicit cost formula that require some care.

Scope in one sentence: HYPE-EDIT-1 is a 100-task benchmark of reference-based marketing and design edits that scores each task with 10 independent attempts under binary pass/fail judging, so that models can be compared on reliability and effective cost per successful edit rather than on best-case samples.

What This Paper Is About

Public demos of image editing models usually show a single cherry-picked output, but real marketing and design workflows involve generating many candidates, reviewing them, and retrying until one is usable. That retry and review overhead is invisible in typical evaluations, even though it dominates the practical cost of using a model. HYPE-EDIT-1 measures this directly by running each editing task 10 times, judging each output independently, and combining per-attempt price with human review time to produce a cost per successful edit.

Key Contributions

1. A 100-task reliability-focused benchmark for reference-based marketing and design edits. Tasks are split into 50 public and 50 held-out private tasks, cover four task types (change, remove, restructure, enhance), and include 89 single-image and 11 multi-image tasks, with input resolutions from 2048–5504 by 1728–5504 pixels.

2. A repeated-trial evaluation protocol with independent judging. Each of 7 evaluated models is run 10 times per task (1,000 outputs per model on the full benchmark), and every output is labeled PASS or FAIL by a majority vote of five human raters who do not see which model produced it, with a Gemini 3 Flash (gemini-3-flash-preview) VLM judge used as a secondary check.

3. A cost model that folds human review time into the price of an edit. The benchmark combines per-candidate model price with an estimated review cost of $0.278 per image (a $50 hourly rate and 20 seconds of inspection), then scales it by the expected number of attempts under a 4-retry cap to yield an effective cost per successful edit.

4. New reporting metrics, including the "Hype Gap." The paper reports Pass Rate (P@1), Pass@10, Pass@4, expected attempts, effective cost, and the Hype Gap (Best-of-10 Uplift), defined as pass@10 minus pass@1, which quantifies how much best-of sampling inflates apparent performance over typical performance.

Main Findings

• Reliability varies enormously across models. Per-attempt pass rates span 34–83% across the evaluated models, and effective cost per successful edit spans $0.66–$1.42.

• Low per-image price does not mean low effective cost. A model priced at $0.03 per image still reaches about $1.42 per successful edit once retries and review are accounted for, while a higher-priced model can be cheaper per success — effective cost is not proportional to per-image cost.

• On the combined split, riverflow-2-b1 leads on every reported reliability and cost metric. It records an 82.7% pass rate, 90.5% pass@4, 1.40 expected attempts, and $0.66 effective cost.

• The ranking is not the same as a per-image price ranking. gemini-3-pro-preview reaches 63.8% pass rate, 79.9% pass@4, 1.85 expected attempts, and $0.95 effective cost; gpt-image-1.5 reaches 61.2%, 70.3%, 2.04 attempts, and $1.30, despite having the highest per-candidate cost in the table at $0.17.

• Mid- and low-pass-rate models cluster tightly on cost. flux-2-max (45.7% pass rate, 2.38 attempts, $1.41), qwen-image-edit-2511 (45.4%, 2.48 attempts, $1.33), seedream-4.0 (35.6%, 2.64 attempts, $1.42), and seedream-4.5 (34.4%, 2.63 attempts, $1.39) all land between $1.33 and $1.42 effective cost.

• Best-of sampling substantially inflates perceived quality. The paper states that the gap between pass rate and pass@10 illustrates how strongly best-of sampling can inflate perceived performance, and repeated collages of 10 outputs per task show partial instruction compliance and structure drift even on narrowly scoped edits such as remove or swap.

• The VLM judge is stricter than humans. The Gemini 3 Flash judge agrees with the human majority in roughly 80% of cases and tends to fail subtle changes that human reviewers accept.

• Public and private split results are shown graphically, not numerically. Figures 4 and 5 report public and private split performance across pass rate, pass@10, pass@4, expected attempts, effective cost, and hype gap, but the paper text does not report numeric values for those two splits.

Methodology in Plain English

The researchers collected 100 real editing tasks from marketing and design workflows — for example, "Add a handle to the mug. Keep everything else the same." or "Swap the middle two drinks and their associated text below it." Each task ships with one or two reference images and a short instruction, stored in a JSON entry with fields for task_id, instruction, task_type, input_images, and target width/height.

For every model and every task, the team generated 10 separate outputs from the same prompt, yielding 1,000 outputs per model across the full benchmark. Each output was then shown to five human raters, who voted PASS or FAIL without knowing which model made the image; the majority vote became the label. A visual-language model judge was run in parallel as a cross-check.

From the repeated labels, the benchmark estimates a per-task success probability, then derives the probability of at least one success across 10 attempts (pass@10) and across 4 attempts (pass@4), plus the expected number of attempts needed. The cost side adds together the model's per-candidate price (ranging from $0.03 to $0.17 in the reported table) and the $0.278 estimated cost of a human reviewing one image, then multiplies by the expected number of attempts and divides by the pass@4 success probability to get effective cost per success. A retry cap of 4 attempts reflects a practical user's patience limit. The difference between pass@10 and pass@1 is reported separately as the Hype Gap.

Why This Matters

Impact on research: Most image editing benchmarks report a single output per task, which hides how often a model fails. By evaluating 10 attempts per task and publishing a public/private split, HYPE-EDIT-1 pushes evaluation toward reliability and contamination-resistant, reproducible comparison — and the Hype Gap gives the field a simple, named way to report the gap between a model's best case and its typical case.

Real-world applications:

• Product marketing: teams generating brand-compliant packshots and lifestyle imagery need to know how many generations and how much review time a usable asset actually costs.
• Brand design and creative production: layout restructuring and product-orientation edits benefit from knowing which models hold structure across repeated attempts.
• Model procurement: buyers comparing $0.03/image against $0.17/image models can use effective cost per success instead of sticker price.
• Internal workflow budgeting: the $50/hour, 20-second review assumption makes the hidden labor cost of retries explicit and adjustable.

Industry relevance: The paper's central commercial claim is that pricing per image is a misleading proxy for affordability. Because models that diverge from the instruction on most attempts burn retries and reviewer attention, the practical cost ranking can differ from the price list — which matters for any team deciding which editing model to deploy at volume.

Future Directions

• Scale and coverage: the authors plan a larger follow-on release that expands task count, broadens marketing constraints (layout, typography, product packshots, brand compliance), and adds more multi-image edits.
• Judging robustness: evaluating with multiple independent VLM judges, releasing a larger human-labeled subset to measure judge correlation and bias, and reporting judge disagreement rates and borderline-case analysis.
• Richer reporting: breakdowns by task type and difficulty, plus confidence intervals via bootstrap resampling across tasks.
• Operational sensitivity: reporting how effective cost changes with review time, labor rate, and retry-cap budget, and exploring alternative workflows such as selecting the best-of-N candidates rather than retrying until a pass.

Target Audience

This paper is most useful to evaluation researchers and benchmark designers working on image generation and editing, to applied machine learning engineers choosing between editing models for production pipelines, and to product or creative-operations teams at marketing and design organizations who need to reason about the true cost of generating a usable asset. Readers who only care about peak image quality will find less here; readers who care about failure rates, retry budgets, and review overhead are the intended audience.

Authors’ abstract

Public demos of image editing models are typically best-case samples; real workflows pay for retries and review time. We introduce HYPE-EDIT-1, a 100-task benchmark of reference-based marketing/design edits with binary pass/fail judging. For each task we generate 10 independent outputs to estimate per-attempt pass rate, pass@10, expected attempts under a retry cap, and an effective cost per successful edit that combines model price with human review time. We release 50 public tasks and maintain a 50-task held-out private split for server-side evaluation, plus a standardized JSON schema and tooling for VLM and human-based judging. Across the evaluated models, per-attempt pass rates span 34-83 percent and effective cost per success spans USD 0.66-1.42. Models that have low per-image pricing are more expensive when you consider the total effective cost of retries and human reviews.