FronTalk: Benchmarking Front-End Development as Conversational Code Generation with Multi-Modal Feedback

AI summary

Overview

Research area: Natural Language Processing / code generation benchmarks, specifically multi-turn, multi-modal conversational coding for front-end web development.

Technical level: Intermediate. The paper assumes familiarity with LLM/VLM evaluation, benchmark design, and web agents, but the core ideas (multi-turn forgetting, visual vs. textual instructions, agent-based critique) are accessible with basic machine learning background.

Scope: The paper introduces FronTalk, a benchmark of 100 multi-turn dialogues (1,000 turns, 3,676 test cases) for front-end code generation with paired textual and visual user feedback, plus an agent-based evaluation framework and a critique-based method (AceCoder) that reduces feature forgetting.

What This Paper Is About

Real developers rarely build a website in one prompt. They iterate over many turns, mixing written instructions ("make the button larger") with visual ones (a sketch of a layout, an annotated screenshot). Existing code-generation benchmarks mostly test single-turn or text-only interaction, so they miss this realistic workflow.

This paper builds a benchmark that captures multi-turn, multi-modal front-end coding, then diagnoses where current models break down and proposes a method that fixes the most prominent failure mode.

Key Contributions

1. FronTalk benchmark: 100 multi-turn dialogues derived from real-world websites in the C4 dataset, spanning 1,000 conversational turns and 3,676 human-refined test cases. Each turn carries both a textual instruction and an equivalent visual instruction (sketch or annotated screenshot).

2. Two new diagnostic metrics: a pass rate for instruction-following (verified by an interactive web agent) and a forgetting rate that quantifies how much functionality correctly implemented in earlier turns is overwritten by later turns.

3. Agent-based evaluation framework: an interactive web agent augmented with image-manipulation tools (cropping, image comparison) to verify test cases, plus a pairwise usability protocol where one LLM simulates a first-time user and a second LLM judges which of two websites offers a better experience.

4. AceCoder: a method where a web agent autonomously critiques the built website against both current and prior instructions, and the critique is used to regenerate a better implementation.

Main Findings

• Models fall far short of the task. Even the strongest model, Gemini-2.5-Pro, is 25.0% below perfect pass rate with textual feedback and 31.3% below with visual feedback. Top open-source models trail by 37.5% (textual) and 55.4% (visual).

• A pervasive forgetting issue. Every evaluated model overwrites previously implemented features across turns, with forgetting rates ranging from 4.3% to 44.6% (up to 46% degradation). This is distinct from long-context degradation — some models handle long prompts fine yet still forget, because the problem is resolving implementation-level conflicts between old and new requirements, not memorizing instructions.

• A visual interpretation gap. Visual feedback consistently underperforms equivalent textual feedback, especially for open-source VLMs. Manual analysis of 608 failure cases reveals three error types: (1) literal replication of sketch layouts without implementing the underlying functionality (most common), (2) missing textual annotations in densely annotated images, and (3) misinterpretation of ambiguous visual clues.

• Proprietary models lead, and the gap widens with visuals. The proprietary–open-source performance gap is 12.5% with textual feedback but 24.1% with visual feedback, suggesting proprietary models generalize better to multimodal code tasks.

• Usability determinants shift with model strength. For weaker models, poor usability comes from broken functionality; for stronger models, it comes from design flaws such as inefficient navigation or missing interactive feedback — i.e., implicit design common sense that users never state explicitly.

• Ambiguous users are much harder. When the user simulator only answers direct questions (clarification-only) or reveals intent solely through choices among model-generated options (preference-only), GPT-4o's pass rate drops from 56.0% to 23.9% and 16.4% respectively, showing weak intent-elicitation ability.

• AceCoder works. It reduces the forgetting rate to nearly zero (e.g., GPT-4o: 21.4% → 0.4%) and improves the overall pass rate by up to 9.3% with textual instructions (56.0 → 65.3) and up to 5.2% with visual instructions (55.0 → 60.2). Ablations show that critiquing past turns, using an interactive agent (not a static LLM), and regenerating the whole site are all necessary for the best results.

• The framework scales with better models. Replacing GPT-4o with Gemini-3.0-Flash in the visual instruction simulator raises fidelity from 76% to 95%, approaching the 98% of the textual simulator — indicating the visual simulator's limits were model capability, not framework design.

• Human agreement is moderate to substantial. LLM evaluation matches human judgment at 80.2% accuracy (Cohen's κ = 58.9) for pass rate and 69.3% (κ = 56.3) for usability, with lower agreement on the inherently subjective usability dimension.

Methodology in Plain English

Building the data. The authors sampled 10,000 web pages from C4, summarized them with GPT-4o, clustered the summaries with BERTopic, and hand-picked 100 representative sites across domains like e-commerce, finance, and digital art. For each site, GPT-4o generated a 10-turn sequence of user intents and matching test cases (short, checkable statements like "a badge appears next to each thread title"). Humans then cleaned up the test cases — removing 66 and editing 2,783 — which raised the validity rate from 90.1% to 94.7%.

Simulating realistic users. Raw intents are not fed directly to the model. A user simulator rewrites each intent based on the current state of the website, resolving ambiguity ("the submit button" → "the submit button on the right") and removing redundancy ("add a nav bar" → "modify the existing nav bar"). Two versions exist: a text simulator (an LLM) and a visual simulator (a VLM with drawing tools that produces sketches and annotations).

Evaluating with agents. Instead of relying on static screenshots or code inspection, the authors use an interactive web agent adapted from WebVoyager, augmented with image-cropping and comparison tools. In "expert" mode it verifies each test case against the live site to compute pass rate. In "naive user" mode a separate agent explores a site, invents tasks, and produces a trajectory; a judge LLM then compares trajectories from two sites to decide which is more usable.

Fixing forgetting with AceCoder. After the model generates code for a turn, a web agent visits the rendered site and checks whether both the new instruction and all prior instructions are satisfied. The resulting critique is appended to the prompt, and the model regenerates the full website.

Why This Matters

Impact on research. FronTalk is the first benchmark to formalize multi-turn front-end coding with genuinely multi-modal feedback (visual instructions in every turn, not just the first). It exposes the forgetting problem as a distinct failure mode of multi-turn code generation and provides metrics — forgetting rate in particular — that future work can build on.

Real-world applications:

• AI coding assistants (Copilot, Cursor, and similar tools) can adopt AceCoder-style critique loops to stop overwriting a user's earlier features during long editing sessions.
• Design-to-code pipelines benefit from a clearer diagnosis of why models misread mockups — they copy layout but skip the functionality behind it.
• Accessibility and UI quality auditing, where the naive-user agent protocol could be reused to test how learnable a generated interface is before shipping.
• Low-code website builders that accept sketches as input, where the visual interpretation gap directly limits product usefulness.

Industry relevance. The 24.1% proprietary–open-source gap under visual feedback signals that multimodal front-end coding is currently a differentiator for frontier labs. Meanwhile, the finding that strong models fail on unstated design principles — not broken buttons — tells product teams where to invest: better defaults and latent design heuristics, not just more capable code synthesis.

Future Directions

• Closing the visual interpretation gap. The dominant failure mode is implementing the look of a sketch without the behavior it implies. Better vision-language grounding, or explicit reasoning about what a layout element must do, is needed.

• Handling ambiguous users. The sharp performance drop under clarification-only and preference-only users points to a largely unsolved question: how should a model proactively ask the right questions to elicit a user's real intent?

• Preserving cross-turn state more fundamentally. AceCoder patches forgetting with an external critique loop. Whether models can learn to maintain and reconcile code state internally — without an agent inspecting the site each turn — remains open.

• Extending the simulator stack. The paper shows fidelity improves when the simulator uses a stronger model (Gemini-3.0-Flash). Continuous upgrades to simulators and evaluators, plus coverage of more languages and UI frameworks beyond the static web, are natural next steps.

Target Audience

• LLM/VLM evaluation researchers designing benchmarks for multi-turn or multimodal code generation.
• Agent researchers interested in interactive evaluation, web agents, and self-critique loops.
• Industry practitioners building coding assistants or design-to-code tools, who can apply the forgetting-rate metric and the AceCoder pattern directly.
• HCI and usability researchers curious about using LLM-driven simulated users and pairwise trajectory comparison as an evaluation methodology.

Authors’ abstract

We present FronTalk, a benchmark for front-end code generation that pioneers the study of a unique interaction dynamic: conversational code generation with multi-modal feedback. In front-end development, visual artifacts such as sketches, mockups and annotated creenshots are essential for conveying design intent, yet their role in multi-turn code generation remains largely unexplored. To address this gap, we focus on the front-end development task and curate FronTalk, a collection of 100 multi-turn dialogues derived from real-world websites across diverse domains such as news, finance, and art. Each turn features both a textual instruction and an equivalent visual instruction, each representing the same user intent. To comprehensively evaluate model performance, we propose a novel agent-based evaluation framework leveraging a web agent to simulate users and explore the website, and thus measuring both functional correctness and user experience. Evaluation of 20 models reveals two key challenges that are under-explored systematically in the literature: (1) a significant forgetting issue where models overwrite previously implemented features, resulting in task failures, and (2) a persistent challenge in interpreting visual feedback, especially for open-source vision-language models (VLMs). We propose a strong baseline to tackle the forgetting issue with AceCoder, a method that critiques the implementation of every past instruction using an autonomous web agent. This approach significantly reduces forgetting to nearly zero and improves the performance by up to 9.3% (56.0% to 65.3%). Overall, we aim to provide a solid foundation for future research in front-end development and the general interaction dynamics of multi-turn, multi-modal code generation. Code and data are released at https://github.com/shirley-wu/frontalk