LongBanana: An Expert-Verified Benchmark for Long-Context Multi-Reference Image Synthesis

AI summary

Overview

• Research area: Computer vision, specifically evaluation of text-and-reference-conditioned image generation and agent-driven image-generation workflows.
• Technical level: Intermediate. Readers need familiarity with image-generation models, benchmark design, and basic agent concepts (retrieval, planning, self-revision), but the paper's core ideas are explained conceptually.
• Scope: LongBanana is an expert-designed, expert-verified benchmark of 90 tasks for long-context multi-reference image synthesis, paired with a graph-grounded checklist evaluator and a study of agent harnesses that wrap a fixed image generator.

What This Paper Is About

Image generators are increasingly asked to build one coherent image from many reference images, where the needed evidence is scattered across dozens of files and mixed with visually similar but irrelevant distractors. Existing benchmarks either use few references or score the final image with a single holistic quality judgment, so they cannot tell whether a system picked the right references, attached attributes to the right entities, or leaked content that nothing in the instruction or references supports. LongBanana addresses this by turning each task into an entity–relation graph, converting that graph into thousands of atomic, individually verifiable checklist items, and validating an automatic checklist judge against reviewed human labels.

Key Contributions

1. A benchmark setting and dataset. LongBanana-Bench contains 90 expert-designed cases in a 3×3 taxonomy (10 cases per subclass), with roughly 32 reference images and more than 20 cross-reference constraints per typical case, plus at least 30 cases containing two or more hard-negative references. Experts specified 4,670 atomic checklist items in total.
2. A graph-grounded evaluation design. LongBanana-Eval maps each task's entity–relation graph to checklist items at three levels: nodes (Content Fidelity), edges (Relational Binding), and whole-graph obligations (Global Constraint Satisfaction), so failures are localized rather than absorbed into one score.
3. Expert verification and an audit protocol. The authors provide reviewed expert checklist labels, a 15% double-labeled subset with 89.4% exact agreement and 0.83 quadratic weighted kappa, evidence traces, and validation of the automatic scorer against human labels.
4. An agent-driven generation study. The paper introduces LongBanana-Gen, an agent-harness framework, and compares direct generation, three minimal agent harnesses, and ReAct–Plan–Exec–Reflect (RPR) variants across ten generation conditions, arguing that current systems fail in structured, diagnosable ways rather than along a single image-quality axis.

Main Findings

• The agentic judge tracks humans better than a single-pass judge. On 900 audit-pool outputs with 46,700 reviewed human labels, LongBananaEval scored Pearson 0.8707, Spearman 0.8602, and Kendall 0.6727 against human overall scores, versus 0.8413, 0.8248, and 0.6437 for the VLM-as-judge baseline. At the item level it raised exact ternary agreement by 15.82 percentage points and cut opposite-end 0↔2 errors by 19.28 points.
• Human labels are consistent. The 7,005 independently double-labeled judgments (15% of 46,700) reached 89.4% exact agreement with a quadratic weighted kappa of 0.83.
• Graph-informed scheduling is a strong non-agentic baseline. With Nano Banana 2 fixed, PD-Direct (which receives privileged access to the construction graph and solves its staging schedule with integer programming) reached a 74.6 overall average and 35.6 P@70%, versus 68.4 and 24.4 for FP-Direct. It also had the best Global Appearance Conditioning score (98.8) and the second-best Layout Arrangement Consistency score (62.6) among compared systems.
• Agentic workflows win more broadly without the graph. OpenClaw + RPR reached the best overall average of 78.4 and a joint-best P@70% of 47.8, surpassing PD-Direct on eight of nine subclasses with a 3.8-point average-score gain and a 12.2-percentage-point gain in task pass rate.
• No single harness dominates. Best per-subclass scores are spread across direct prompting, minimal harnesses, and RPR. The minimal OpenClaw harness (76.1 average, 40.0 P@70%) remained stronger than OpenClaw + RPR on temporal and state reasoning; OpenCode (77.3, 47.8) and Hermes-Agent (73.9, 43.3) lost task pass rate when RPR was added even as their averages rose slightly or fell.
• RPR gains depend on the harness. The full workflow gave OpenClaw its strongest result, with the clearest improvements over the minimal harness in scene composition, reference selection, and aggregation; the paper attributes this to reliable evidence grounding and repair, and notes drift when the harness cannot exploit retrieval or revision feedback.
• Component ablations separate the roles. With OpenClaw and Nano Banana 2 fixed, Plan–Exec alone was strongest on global appearance conditioning and local appearance binding, ReAct alone improved reference selection over the minimal harness, and Reflect alone improved layout, aggregation, and task pass rate; the full workflow was strongest overall. The per-component numbers are in Table 11, which is not included in the excerpt.
• The image backend still matters a lot. With OpenClaw + RPR fixed, Nano Banana 2 led all nine subclasses with a 78.4 average and 47.8 P@70%, versus Seedream 4 at 63.6 and 18.9 and GPT Image 1 at 61.8 and 17.8. Seedream 4 beat GPT Image 1 on scene composition, temporal/state reasoning, and aggregation, while GPT Image 1 was more competitive on visual appearance and reference selection.

Methodology in Plain English

The authors built each benchmark case around a graph instead of a loose prompt. Experts first wrote a realistic scenario (for example, an accessories retailer advertising handbags with dual parallel top handles). A language model then structured that scenario into nodes (entities and attributes that must or must not appear), edges (spatial, interaction, ordering, and local-appearance relations), and whole-graph constraints (layout, counts, temporal state, which references to select). That graph drove both the creation of the instruction and reference images and the derivation of the checklist.

Each checklist item is atomic and tied to exactly one graph unit. It records what to observe, which references are relevant, and how to score it, and items can be positive (required content present) or negative (distractor or unsupported content absent). Human curators then reviewed instruction, references, and checklist together for feasibility and verifiability before a case was included.

Scoring is done by LongBananaEval, an agentic checklist judge. For every item it assembles an evidence packet (the output image plus the relevant reference images, optionally cropped), routes the item to one of three critics matching the rubric dimension, and lets the critic call deterministic tools such as OCR, detection or counting, layout routines, and identity or feature matching. Each item gets a label of 2 (satisfied), 1 (partial or uncertain), or 0 (unsatisfied); for negative items, complete absence of forbidden content scores 2. Item labels are averaged and rescaled to a 0–10 overall and per-dimension score, with evidence traces stored for audit. Generation tables report the checklist-average on a 0–100 scale plus P@70%, a difficulty-weighted task pass rate using item weights computed once from human scores on a fixed reference-system pool and then held fixed.

For the generation study, the authors evaluated ten conditions: two direct baselines (FP-Direct, which slides an eight-reference window over the inventory with a fixed full prompt, and PD-Direct), three minimal agent harnesses (Hermes-Agent, OpenCode, OpenClaw, all using Claude Sonnet 4.6 as the agent model), and their RPR variants, plus a backend comparison fixing OpenClaw + RPR and swapping in Seedream 4, GPT Image 1, or Nano Banana 2. Human annotators labeled the 900 generated outputs against the full checklists, giving the 46,700 reviewed judgments used to validate the automatic judge.

Why This Matters

Impact on research. The paper reframes multi-reference image synthesis as a provenance problem: the question is not whether the image looks realistic but whether each piece of content can be attributed to a specific reference or to the instruction. By tying every checklist item to a graph unit, it makes failures diagnosable (wrong reference selected versus wrong binding versus unmet global constraint) and gives the field a shared, auditable target rather than a single opaque quality score. It also supplies a validated automatic scorer, which is a prerequisite for cheap, reproducible comparison of future systems.

Real-world applications:

• Product and catalog advertising: assembling a compliant image from an inventory of item photos while rejecting look-alike products and preserving counts, layouts, and appearance details.
• Storyboarding and previsualization: synthesizing a shot that must respect established character, prop, and set references across many images.
• E-commerce and retail QA: automatically auditing generated marketing images against source references for wrong-item or unsupported-content errors before publication.
• Agent and tooling development: benchmarking orchestration layers that sit between a user's reference library and a black-box image model.

Industry relevance. The results separate two levers that companies often conflate: the harness (evidence selection, constraint organization, revision) and the image backend (fidelity of realization). The backend comparison shows a large gap under a fixed workflow (78.4 average for Nano Banana 2 versus 63.6 and 61.8), while the harness comparison shows an agentic workflow beating a graph-privileged static plan by 3.8 average points and 12.2 points of pass rate. That framing is directly useful for teams deciding whether to invest in orchestration or in a stronger generator, and the finding that RPR helps some harnesses while hurting others argues against assuming that more agent scaffolding is always better.

Future Directions

• Coordinating retrieval, planning, and repair reliably. The authors identify reliable coordination as the next design priority: retrieved evidence must stay attached to the correct entities and constraints through planning, generation, and revision, since RPR improved OpenClaw but reduced task pass rate for OpenCode and Hermes-Agent.
• Reasoning over the evolving visual state, not just the task plan. The gap between PD-Direct's privileged graph schedule and adaptive workflows suggests agents should inspect intermediate images, detect which constraints survived, and target only unmet requirements during revision.
• Improving reference selection and state reasoning specifically. These are the subclasses where the structurally privileged direct baseline stayed competitive, and where the minimal OpenClaw harness beat its RPR variant on temporal and state reasoning.
• Extending scale and coverage. The benchmark has 90 cases and 4,670 items; whether these diagnostics and judge agreement hold over larger, more diverse task pools, and how the difficulty weighting generalizes beyond the fixed reference-system pool used here, are open questions.

Target Audience

Researchers and engineers working on image-generation evaluation, multi-reference and personalized image synthesis, and agentic or tool-using systems built around image models. It is also useful for practitioners who need to audit generated marketing or product imagery against source references, and for benchmark designers interested in how to convert a structured task representation into atomic, evidence-linked scoring items. Readers looking for new image-generation architectures rather than evaluation and orchestration will find less here, since the generator is treated as a fixed component throughout.

Authors’ abstract

We introduce LongBanana, to our knowledge the first expert-designed and expert-verified benchmark for long-context multi-reference image synthesis in the age of agent-driven image generation. The task requires retrieving dispersed visual evidence, rejecting distractors, binding entities and relations, and composing one coherent final image from multiple references. LongBanana contains 90 carefully designed tasks organized by a 3-by-3 taxonomy; each task includes about 32 reference images, 20 constraints, and 50 evaluation items, yielding 4,670 expert-specified atomic evaluation items in total. Its graph-grounded evaluation design instantiates each task with reference images and an entity-relation graph, then converts node, edge, and whole-graph obligations into checklist items for Content Fidelity, Relational Binding, and Global Constraint Satisfaction. Reviewed expert labels include a 15% double-labeled subset with 89.4% exact agreement and 0.83 quadratic weighted kappa. We further introduce LongBanana-Gen, an agent-harness framework for long-horizon image generation; experiments show that minimal agent harnesses improve over direct generation, while a ReAct-Plan-Exec-Reflect (RPR) workflow yields harness-dependent gains by strengthening evidence grounding and repair. Together, LongBanana provides an auditable testbed for measuring dense visual evidence use and a shared target for developing stronger agent-driven image-generation systems. Code and data are available at the project page: https://long-banana.com/.