Co-Design Bench: Multimodal Collaborative Coding Evaluation for Agentic Models

AI summary

Overview

Research area: AI agent evaluation, specifically coding agents that revise web applications in response to real users' multimodal requests (text plus image, audio, video, or document inputs).

Technical level: Intermediate. The core ideas are accessible, but the evaluation protocol (criterion-level rubric grading, adversarial refutation, evidence audits) assumes some familiarity with LLM benchmarking.

Scope: The paper introduces Co-Design Bench, a 273-task, reference-free benchmark built from real user coding sessions that measures whether coding agents can turn multimodal revision requests into working application edits, and it reports results for ten model endpoints.

What This Paper Is About

Coding agents are increasingly used to modify existing web applications when a user asks for a change, often attaching a screenshot, a screen recording, or a document alongside the request. Existing benchmarks mostly test rebuilding a page from a reference image, applying predefined synthetic edits, or fixing repository issues, and none of them capture compound revisions recovered from real user sessions that combine multimodal evidence with coordinated changes across a repository. The paper's goal is to build a benchmark from genuine user requests with runnable starting applications and hidden expert-written acceptance criteria, then measure both how many individual requirements an agent satisfies and how often it fully completes a task.

Key Contributions

1. A benchmark built from real user sessions. Co-Design Bench contains 273 tasks curated from 37,463 deduplicated queries drawn from 26,314 real user coding sessions, each paired with a validated runnable pre-edit application, a normalized request, retained session context, and attachments.

2. Two complementary tracks with a shared interface. Multimodal Editing (ME, 100 tasks) covers localized visual, structural, and content edits with optional image, audio, video, or document inputs; Repository Editing (RE, 173 tasks) covers coordinated changes across components, routes, state, and data flow. Together they carry 2,331 hidden expert-authored acceptance criteria.

3. A reference-free "judge-then-refute" protocol. Hidden criteria are first graded using source diffs, rendered pages, and Playwright traces; a separate LLM verifier then challenges every Pass and Partial verdict, checking for stubs, hardcoded outputs, unreachable entry points, and request mismatches. Mandatory audits abort scoring when required screenshots or browser-action logs are missing.

4. A dual-metric characterization of agent performance. The paper reports Strict Completion, Graded Completion, and Full-Pass Rate across ten model endpoints (2,730 model–task pairs), showing that broad partial credit can coexist with low complete-task success, plus analyses by input modality, edit breadth, and capability dimension.

Main Findings

• High criterion coverage, low complete delivery. The strongest endpoint, gpt-6-astra, reaches mean strict completion of 0.754 on ME and 0.667 on RE, but fully passes only 41.0% and 17.9% of tasks respectively. Across all 273 tasks its mean strict completion is 0.699 with a full-pass rate of 26.4%, meaning 73.6% of tasks retain at least one unsatisfied criterion.

• Rankings shift between tracks. Strict-completion rankings correlate across tracks (Spearman ρ = 0.818), but kimi-k3 moves from fourth on ME to sixth on RE, while Qwen3.8-27B moves from fifth to second. Endpoint mean strict scores span 28.7 percentage points on ME but only 13.0 on RE, so a combined leaderboard can hide track-specific strengths.

• Broader edits are harder. On RE, tasks spanning three or four edit types score 0.591 pooled strict completion versus 0.654 for one or two types, a 6.3-point gap, and every one of the ten endpoints scores lower on the broader group. Zero-pass rates rise from 0.9% to 5.8%, while full-pass rates stay similar (9.1% versus 9.7%).

• Modality changes the picture but not monotonically. Pooled strict completion ranges narrowly from 0.584 (audio) to 0.652 (documents). Documents have the highest pooled strict score but only 18.2% of their model–task pairs fully pass, compared with 37.0% for text-only tasks. Qwen3.8-27B leads text-only tasks (0.779), opus-5 leads image tasks (0.891), and gpt-6-astra leads video, audio, and document tasks.

• Capability groups show the same coverage-versus-delivery split. Functional Implementation is the largest group (126 tasks, 46.2%) and has the highest pooled strict completion (0.619) but the lowest full-pass rate (10.3%), versus Grounded Understanding (0.604; 20.2%) and UI Transformation (0.615; 13.6%). gpt-6-astra leads all three at 0.738, 0.692, and 0.673 respectively.

• Agentic execution inflates scores relative to single-shot. On the same 150 WebCompass-edit tasks with rubric, judge, and render pipeline held fixed, qwen3.7-plus rises from 73.6% to 95.1% mean graded feature completion, a 21.5-point gain, with all-pass tasks increasing from 49 to 104 (32.7% to 69.3%). opus-4-8 reaches 99.4% under agentic execution on those tasks.

• More effort does not buy more completion. On ME, gpt-6-astra reaches 0.754 with 32 median turns and 1.24M tokens per case, while GLM-5.2 uses 69 turns and 7.29M tokens for 0.572. On RE, Qwen3.8-27B uses 57 turns and 6.72M tokens for 0.645, versus 48 turns and 2.33M tokens for gpt-6-astra at 0.667.

• Execution profiles differ by track and modality. ME video tasks have a median of 44 turns versus 27 for text-only tasks; media interactions account for 82.8% of activity in the upper-quartile longest rollouts; image-inspection calls average 14.47 per run on ME and 2.78 on RE, while the Edit/Write share rises from 7.3% to 27.6%.

• Agents reach for external perception tools. OpenCV appears in 68.5% of video trajectories, ffmpeg in 31.5% of video and 25.0% of audio runs, and faster-whisper in 4.4% of audio runs; some agents also run pretrained models to obtain descriptions, transcripts, class scores, or detection boxes.

Methodology in Plain English

The authors started from a production corpus of 39,325 queries across 26,314 coding sessions, deduplicated it to 37,463 candidates, and anchored each candidate to a specific edit turn with its repository, context, and attachments. They then reconstructed runnable starting applications from session logs, repository archives, or HTML bundles, requiring each to pass dependency, build, container launch, and Playwright rendering checks; tasks needing unrecoverable services were dropped. Hard filters removed environment-blocked, non-actionable, and non-editing requests, and the survivors were ranked by structural difficulty, verifiability, and historical failure signals such as reformulations and rollbacks. Human experts refined the requests and wrote hidden criterion-level rubrics that specify requested outcomes while allowing multiple valid implementations, so no gold code or target screenshot is needed.

Evaluation proceeds in stages. A candidate application is built and rendered in isolation, and a fixed agentic evaluator assigns Pass, Partial, or Fail to each criterion using source diffs, rendered pages, and Playwright traces. A second LLM verifier then attacks every positive verdict, and an evidence audit enforces that visual claims are backed by screenshots and interaction claims by recorded browser actions, aborting scoring when evidence is missing. Scores are aggregated per task so long rubrics cannot dominate: Strict Completion counts only Pass verdicts, Graded Completion gives Partial 0.5 credit, and Full-Pass Rate counts tasks where every criterion passes. All ten models ran under the same Claude Code harness with identical container manifests, prompts, and tool interfaces, capped at three hours and 120 turns.

Why This Matters

For research, the benchmark supplies a task format where the starting artifact is real, the criteria are hidden and expert-authored, and positive judgments are actively challenged. That makes the measured gap between partial credit and full delivery visible, and gives a common scale across localized multimodal edits and repository-scale changes. The paper also demonstrates that measured completion is strongly sensitive to execution regime (a 21.5-point swing on a fixed task set) and to the choice of harness, which has implications for how any agentic benchmark score should be read.

Real-world applications:

• AI coding assistants in IDEs and browser tools that modify existing apps rather than generating from scratch, where the user's intent arrives as a screenshot, voice note, or spec document.
• Design-to-code and prototyping workflows, where a designer requests a localized change and needs the result to respect dependent values and preserved content.
• QA and release gating for agentic coding products, using criterion-level evidence to distinguish genuinely implemented features from stubs, hardcoded outputs, and unreachable code paths.
• Failure triage in production agent deployments, where the paper's attribution step separates model limitations from environmental blockage and defects in the starting artifact.

For industry, the results argue against reporting a single average score for coding agents. A model that passes roughly 70% of criteria while fully completing roughly a quarter of tasks would look far stronger on a coverage metric than it is in delivery terms, and the track-dependent ranking reversals mean a leaderboard position may not transfer from localized edits to repository-scale work.

Future Directions

• Closing the completion gap. All ten endpoints show large gaps between criterion coverage and full-task success, and the paper does not identify which specific failure mode (request grounding, change propagation, or final-state verification) dominates, leaving that analysis open.

• Separating perception from editing effects. The paper states explicitly that its modality and tool-use profiles cannot isolate causal effects of perception or editing tools because task content differs across comparisons; controlled studies would be needed.

• Harness-robust evaluation. Cross-harness experiments reveal strong model × harness interactions, so a natural next step is standardizing or reporting execution conditions as part of any agentic benchmark score.

• Reducing evaluator error. The judge-then-refute design makes positive credit traceable but, as the authors note, cannot eliminate judgment error; the sensitivity check and the 200 task–model outputs reviewed by two frontend experts suggest room for further validation of automated verdicts.

Target Audience

Researchers and engineers working on coding agents, agentic evaluation, and multimodal grounding will get the most from this paper, particularly those designing benchmarks or building judges for artifact-level assessment. Product teams shipping AI coding assistants should read the results section for the coverage-versus-completion distinction and the track-dependent rankings. Readers interested in human–AI co-design and front-end engineering workflows will find the task construction and worked examples (receipt value reconciliation, a clip editor with coupled limits and state recovery) directly relevant.

Authors’ abstract

Evaluating human-AI co-design requires testing whether coding agents can translate real users' multimodal requests into working application revisions. We introduce Co-Design Bench, a reference-free benchmark of 273 tasks curated from 37,463 deduplicated queries across 26,314 real user coding sessions. Its two tracks, Multimodal Editing (ME) and Repository Editing (RE), cover localized edits with optional image, audio, video, or document inputs and coordinated repository revisions. Tasks span three capability dimensions: Grounded Understanding, UI Transformation, and Functional Implementation. The tasks pair normalized requests and runnable pre-edit applications with 2,331 hidden expert-authored acceptance criteria. Our judge-then-refute protocol grades criteria using source and browser evidence, challenges positive verdicts with a separate LLM verifier, and audits visual and interaction claims without gold code or target screenshots. Under a shared agentic harness, the strongest of ten model endpoints passes an average of 69.9% of criteria per task but fully satisfies only 26.4% of tasks. All ten endpoints achieve lower criterion completion on repository edits spanning three or four change types than on those spanning one or two. Co-Design Bench provides a testbed for studying the gap between partial requirement satisfaction and complete task success under agentic execution for real-user application revisions.