From Prompt to Product: A Human-Centered Benchmark of Agentic App Generation Systems

AI summary

Overview

Research area: Human-Computer Interaction / agentic AI evaluation — specifically, human-centered benchmarking of "prompt-to-app" systems that generate full-stack web applications from natural language.

Technical level: Intermediate. The conceptual framing and results are accessible, but the paper leans on statistical machinery (Linear Mixed-Effects Models, Bradley-Terry tournament models, Cliff's Delta effect sizes, Kruskal-Wallis and Wilcoxon signed-rank tests) that assumes some familiarity with experimental design.

Scope: A large-scale comparative benchmark of three commercial prompt-to-app platforms (Replit, Bolt, Firebase Studio) using 96 prompts, 288 generated artifacts, and a human-rater study with 1,071 quality-filtered pairwise comparisons across four quality dimensions.

What This Paper Is About

Prompt-to-app agents promise to build entire deployable web applications from a single natural language description, but there is no rigorous way to tell how good their outputs actually are. Existing code-generation benchmarks test small function-level snippets, which cannot capture whether a generated app is functional, trustworthy, or pleasant to use. This paper builds a human-centered benchmark that evaluates the end products of these systems through real people interacting with real deployed apps.

Key Contributions

1. A mixed-methods evaluation framework for prompt-to-app agents that combines an automated deployment audit with task-based human assessment, rather than relying on static code benchmarks alone.
2. A comparative study of three leading systems — Replit, Bolt, and Firebase Studio — evaluated under standardized, single-shot, default-configuration conditions.
3. A public dataset of 288 generated application artifacts (96 from each platform), generated from 96 prompts and deployed to a standardized containerized environment (Quome Cloud).
4. Two headline empirical findings: isolated (one-app-at-a-time) evaluations of these systems are statistically indistinguishable, whereas side-by-side comparisons yield clear and consistent user preferences; and a single system (Firebase Studio) can dominate across every measured human-evaluated dimension, indicating the market is not yet commoditized.

Main Findings

• Isolated ratings converge; comparisons diverge. On clarity, all three platforms scored nearly identically: Replit 3.96 (95% CI [3.82, 4.09]), Bolt 3.95 (95% CI [3.82, 4.09]), Firebase 3.92 (95% CI [3.78, 4.06]), with pairwise differences not statistically significant (Firebase vs. Bolt β = −0.037, p = 0.50; Replit vs. Bolt β = 0.004, p = 0.94). Once users compared systems side by side on the same task, Firebase led on all four comparison metrics.

• Replit led on isolated ease of use. In isolated ratings, Replit scored highest at 3.85 (95% CI [3.71, 3.99]), then Bolt 3.81 (95% CI [3.67, 3.96]), then Firebase 3.68 (95% CI [3.54, 3.82]). The only statistically meaningful isolated difference was Firebase scoring lower than Bolt (β = −0.130, p = 0.029); the Replit-over-Bolt gap was not significant (β = 0.039, p = 0.51).

• Firebase Studio won every pairwise dimension. Win rates with 95% Wilson intervals: Ease Overall 42.5% [0.388, 0.463]; Trust 41.2% [0.375, 0.450]; Visual Appeal 45.5% [0.418, 0.493]; Visual Appropriateness 45.1% [0.413, 0.489]. Firebase ranked 1st on all four.

• Bolt placed second across the board. Bolt's win rates were Ease Overall 31.7% [0.284, 0.352], Trust 29.2% [0.259, 0.326], Visual Appeal 36.8% [0.333, 0.404], and Visual Appropriateness 33.9% [0.306, 0.375], ranking 2nd on every metric.

• Replit placed third across the board. Replit's win rates were Ease Overall 26.0% [0.230, 0.292], Trust 22.7% [0.199, 0.258], Visual Appeal 28.9% [0.258, 0.322], and Visual Appropriateness 28.9% [0.258, 0.322], ranking 3rd on every metric.

• Bradley-Terry confirms the ordering. Firebase showed statistically significant advantages over Bolt in Trust (β = 0.051, p < 0.05), Visual Appropriateness (β = 0.049, p < 0.05), Visual Appeal (β = 0.042, p < 0.05), and Ease Overall (β = 0.034, p < 0.05). Replit performed significantly worse than Bolt, notably in Trust (β = −0.038) and Ease Overall (β = −0.033). In power-score terms, Firebase's advantage over an average opponent ranged from 53.4% (Ease Overall) to 54.7% (Trust), while Replit's ranged from 45.8% (Trust) to 47.1% (Visual Appropriateness).

• Effect sizes were small but consistent. Cliff's Delta was most pronounced for Firebase over Bolt in Visual Appropriateness (Δ = −0.234, small) and Visual Appeal (Δ = −0.212, small), and for Firebase over Replit in Trust (Δ = 0.198, small), using the paper's thresholds where |Δ| < 0.147 is negligible, 0.147 ≤ |Δ| < 0.33 is small, 0.33 ≤ |Δ| < 0.474 is medium, and |Δ| ≥ 0.474 is large.

• Composite hierarchy. Firebase Studio achieved the highest composite score (1.933), followed by Bolt (1.561), then Replit (1.302).

• Deployment audit validated the human ratings. Automated testing agreed with participant-reported appearance on 88.2% of URLs (254 of 288), with Cohen's Kappa 0.691 ("substantial"), Pearson r = 0.706 (p < 0.001), and a chi-square of 133.9 (p < 0.001). The overall confusion matrix was: appeared/success 197 (68.4%), not appeared/success 16 (5.6%), appeared/failure 18 (6.2%), not appeared/failure 57 (19.8%).

• Platform-level deployment disagreement varied. Agreement was highest for Replit (92.7%), then Bolt (90.6%), then Firebase (81.2%). Firebase showed the largest gap between automated and participant reports: 84.4% automated success versus 68.2% participant-reported appearance, a 16.2 percentage point difference. Bolt's rates were nearly identical (64.6% automated vs. 64.8% participant); Replit's were 75.0% automated versus 68.4% participant.

Methodology in Plain English

The researchers picked three commercial prompt-to-app tools — Replit, Bolt, and Firebase Studio — and wrote 96 prompts covering realistic web-app requests. These prompts were organized along four axes: category (healthcare, legal, real estate, financial services, government, education), difficulty (easy, hard), specificity (specific, ambiguous), and complexity (low, high). Each prompt was submitted to each system once, producing 288 application artifacts total.

Prompts were submitted verbatim with no system-specific tuning, using each platform's default settings, in a single generation pass with no iterative refinement or manual code editing. Where an agent asked for clarifying input (which happened with Bolt and Replit), a choice was picked at random to simulate a minimally engaged user. Firebase Studio generations were automated with a Puppeteer browser script around September 5, 2025; Bolt and Replit generations were triggered manually around September 28, 2025.

All artifacts were pushed to GitHub and deployed into a standardized containerized environment. Because the tools produce different project structures, a helper program triaged each artifact, added an appropriate Dockerfile, built a container image, and deployed it — with some Replit artifacts needing manual API keys and some database-dependent apps needing a manually added SQLite volume.

An automated audit checked deployment by HTTP GET requests (catching connection errors, timeouts, SSL errors) followed by a Playwright browser check for visible DOM elements or crashes.

For the human study, 282 participants were recruited through the CINT Exchange platform with no technical-background eligibility criteria; the abstract reports the analysis involving 205 participants. Each rater moved through a three-phase structure: interact with App A alone, interact with App B alone, then view both side by side. A practice tutorial with two placeholder websites preceded the real evaluation. Participants rated isolated apps on clarity and ease of use using five-point scales (1 = very poor, 5 = very good), and rated pairs on a five-point preference scale (1 = strongly prefer App A, 3 = no preference, 5 = strongly prefer App B) for visual appeal, visual appropriateness, comparative ease, and trust, plus a flag for whether each app successfully appeared.

Response quality was enforced with minimum time-on-task, a cap on total time to prevent fatigue, a maximum time-per-question to deter bots, and a control question with a known answer. Pairwise analysis was restricted to the 1,071 comparisons where both apps successfully appeared.

Statistically, the team used Kruskal-Wallis and Friedman tests for isolated ratings, Wilcoxon signed-rank tests and Bradley-Terry models for pairwise data, and Linear Mixed-Effects Models with platform and presentation position as fixed effects and random intercepts for participants (to absorb harsh-versus-lenient grading bias) and prompts (to absorb task difficulty).

Why This Matters

Impact on research. The paper argues that reference-free, one-app-at-a-time evaluation is unreliable for this class of system: isolated clarity differences were statistically indistinguishable (p > .5), and the isolated ease-of-use ordering even reversed relative to the side-by-side result. That makes a methodological claim with implications beyond this benchmark — if pairwise comparison is the only reliable way to surface meaningful differences, then static or single-artifact benchmarks may systematically under-detect quality gaps in generative software.

Real-world applications:

• Tool selection by teams and individuals choosing among Replit, Bolt, and Firebase Studio, with evidence that the platforms are not interchangeable and that preference patterns differ by dimension.
• Citizen-developer prototyping, since the benchmark targets non-expert users building functional web apps from plain-language descriptions.
• Vendor product development, given that Firebase's advantage was concentrated in trust and visual appropriateness rather than raw fluency, suggesting where competing tools have room to improve.
• Benchmark design more broadly, offering a reusable template of prompt taxonomy, deployment standardization, quality filtering, and mixed-effects analysis that could be ported to other agentic generation domains.

Industry relevance. The headline conclusion — that one system can lead across trust, ease of use, and visual appeal simultaneously — contradicts the common assumption that agentic generation tools must trade visual polish against functional reliability. The authors read this as evidence that the market is not yet commoditized. The deployment audit also flags a practical measurement gap: Firebase's automated 84.4% success rate versus a 68.2% participant-reported appearance rate suggests automation can miss problems (slow loading, rendering issues, usability failures) that real users notice immediately.

Future Directions

• Expanded benchmark coverage. Extend the prompt set to more complex workflows, multi-user roles, third-party API integrations, and prompts that stress long-lived state, permissions, and multi-step transactions.
• Stability and multi-sample evaluation. Run each system-prompt pair multiple times to measure output variance, robustness to prompt perturbations, and consistency across generations, addressing the paper's acknowledged limitation of single-snapshot generation.
• Stronger security and reliability analysis. Add runtime testing, dynamic analysis, type-flow consistency checks, and backend-frontend contract validation, since the current automated pipeline covered only deployability, framework usage, and basic correctness signals and should be read as a lower-bound indicator of functional robustness.
• Developer-centered and longitudinal studies. Study how developers modify, extend, and debug generated applications over longer horizons, quantifying "agentic ROI" in terms of time-to-prototype, maintenance effort, and reduced developer workload — the present study captured only short, task-focused end-user interactions and did not measure learnability, accessibility, or maintainability.

The authors also state they intend to release the benchmark framework, generated artifacts, and survey instruments to support reproducible evaluation and community extension.

Target Audience

Researchers in human-computer interaction and generative AI evaluation will get the most from the framework and methodology. Practitioners and technical decision-makers evaluating prompt-to-app platforms for prototyping or internal tooling will find the comparative results directly actionable. Benchmark designers and LLM evaluation researchers will find the isolated-versus-pairwise methodological finding and the deployment auditing approach reusable. Product managers at low-code, no-code, and developer-platform companies can use the dimension-level breakdowns (trust, ease, visual appeal, visual appropriateness) as a competitive reference. The paper assumes enough statistical literacy to interpret mixed-effects and Bradley-Terry results, so readers without that background will need to lean on the reported win rates and confidence intervals.

Authors’ abstract

Agentic AI systems capable of generating full-stack web applications from natural language prompts ("prompt- to-app") represent a significant shift in software development. However, evaluating these systems remains challenging, as visual polish, functional correctness, and user trust are often misaligned. As a result, it is unclear how existing prompt-to-app tools compare under realistic, human-centered evaluation criteria. In this paper, we introduce a human-centered benchmark for evaluating prompt-to-app systems and conduct a large-scale comparative study of three widely used platforms: Replit, Bolt, and Firebase Studio. Using a diverse set of 96 prompts spanning common web application tasks, we generate 288 unique application artifacts. We evaluate these systems through a large-scale human-rater study involving 205 participants and 1,071 quality-filtered pairwise comparisons, assessing task-based ease of use, visual appeal, perceived completeness, and user trust. Our results show that these systems are not interchangeable: Firebase Studio consistently outperforms competing platforms across all human-evaluated dimensions, achieving the highest win rates for ease of use, trust, visual appeal, and visual appropriateness. Bolt performs competitively on visual appeal but trails Firebase on usability and trust, while Replit underperforms relative to both across most metrics. These findings highlight a persistent gap between visual polish and functional reliability in prompt-to-app systems and demonstrate the necessity of interactive, task-based evaluation. We release our benchmark framework, prompt set, and generated artifacts to support reproducible evaluation and future research in agentic application generation.