Exploring LLMs for User Story Extraction from Mockups

AI summary

Overview

Research area: Software engineering, specifically requirements engineering, at the intersection of agile artifacts (user stories, high-fidelity mockups) and large language models. arXiv:2602.16997v1 [cs.SE], published 19 Feb 2026.

Technical level: Intermediate. No model internals or training details are required to follow the argument, but familiarity with requirements engineering artifacts (user stories, glossaries, traceability) helps.

Scope in one sentence: A case study on whether LLMs can derive user stories from high-fidelity mockups of existing web applications, and whether supplying a Language Extended Lexicon (LEL) glossary in the prompt makes those user stories more accurate and domain-appropriate.

What This Paper Is About

User stories are concise, user-centric descriptions of functionality and are among the most widely used artifacts for defining functional requirements; high-fidelity mockups let end users express their needs with greater precision about the user interface. However, mockups and user stories are often insufficient on their own, and keeping them consistent with one another is a recognized challenge. The authors ask whether an LLM can generate a user story directly from an image of a mockup, and whether adding a domain glossary, the LEL, to the prompt improves the result.

Key Contributions

1. A collaborative workflow for LLM-assisted user story generation. End users create high-fidelity mockups by modifying existing web applications through a browser add-on; requirements engineers build an LEL glossary of the application domain; the product owner combines both in a prompt and obtains a user story that is then managed (for example, on a Kanban board) and passed to developers.

2. A three-part context for the prompt. The screenshot of the original application, the mockup image, and the LEL glossary are all fed to the LLM, so that visual requirements and domain terminology are handled together rather than separately.

3. Two illustrative use cases. One on YouTube, where the authors considered specifying an LEL unnecessary and tested the LLM's ability to interpret a mockup on its own, and one on LeafLab, a niche botanical domain where the LEL was expected to matter most.

4. A preliminary validation across five categories of mockup actions. The authors tested whether the LLM produces an appropriate user story for each action type, with and without the LEL, and scored each generated story on a five-point scale.

Main Findings

• The LEL improved the generated user stories. The authors state that incorporating the LEL significantly enhances the accuracy and suitability of the generated user stories, and that in the preliminary validation the score with LEL was greater than or equal to the score without LEL.

• The LeafLab example shows the difference concretely. Without the LEL, the LLM described the sorting criterion in terms of "score" and produced the phrase "So that I can quickly see the most relevant or highest-scoring species without the need for manual searching or sorting". With the LEL, the same requirement was expressed using domain terminology: "So that I can quickly view the most relevant species based on the number of points where they were found, without the need for manual sorting". In the second version, "points" refers to the number of times a species was recorded in the surveyed field rather than to a score.

• For a well-known application, the LLM managed without a glossary. In the YouTube case, the user added a button for channel statistics by dragging and dropping it onto the channel interface and setting its label to "stats". From the original screenshot and the mockup alone, the LLM produced a user story stating that a new button was desired, that it should sit next to the existing buttons, and that it should look the same as the originals.

• The LEL used in the case study was incomplete. The authors state that the provided lexicon was incomplete and only sufficient for that specific use case.

• Preliminary validation scores (Table 4), by type of requirement action, without LEL versus with LEL: A (add a new, not previously existing element) 3 → 5; B (add an existing element) 3 → 5; C (remove existing elements) 4 → 4; D (compose elements from other parts of the application) 3 → 5; E (compose elements from other applications) 3 → 5.

• The scoring scale used was: 1 = does not distinguish any element of the requirement; 2 = distinguishes only partially the elements that compose the requirement; 3 = distinguishes the elements of the requirement but does not describe them adequately; 4 = distinguishes the elements of the requirement and describes some of them adequately; 5 = distinguishes the elements of the requirement and describes them adequately in their entirety.

• Models, datasets and benchmarks are not reported. The paper refers only to "an LLM" and does not name a specific model, version, prompt-testing dataset size, or any comparison against non-LLM baselines.

• An internal inconsistency appears in the LEL example. The body text and the Visit entry describe a transect as walked for approximately 300 meters with species recorded every 3 meters, resulting in 100 points per transect, and the body text describes 100 equidistant sampling points along a 300-meter line; the Transect entry in Table 3 instead states that each transect is composed of 300 points.

Methodology in Plain English

The authors designed a process rather than training a model. End users work on top of an existing web application, using it as a canvas through a browser add-on mockup tool. They can do three things: select pre-existing elements that are relevant to their need, add new components (widgets, elements from other parts of the application, or elements from other applications), and remove unwanted elements. Modified and added elements are rendered in a hand-drawn mockup font so they stand out from the untouched parts of the original interface.

In parallel, requirements engineers describe the application domain as an LEL: a structured glossary that textually describes all the actors (subjects), their actions (verbs), the concepts (objects) involved, the situations (states), a notion (meaning) for each, and the relationships or behavioral responses between them.

The product owner then writes a prompt that includes the screenshot of the original application, the mockup, and the LEL, and the LLM returns a user story. The story can be pushed into a requirements management platform such as a Kanban board shared with developers. The authors note the tool can interact with the LLM through an API once mockups are finalized.

For the evaluation, the authors took the five fundamental action types (A through E listed above), built a mockup incorporating different categories of actions, generated user stories with and without the LEL, and scored each story on the 1-to-5 scale. Both the LEL and the mockups used in the validation were developed by the team responsible for the application, who had experience in both the domain and the technologies involved.

Why This Matters

Impact on research. The paper targets a gap the authors identify: relatively few studies have explored the direct derivation of user stories from high-fidelity mockups. It also connects two research threads that are usually separate, LLM-based requirements generation and domain lexicons, by showing that domain-specific terminology can refine LLM output. It builds on prior work on traceability between user stories and graphical artifacts, mockup augmentation, and requirement elicitation.

Real-world applications.

• Agile teams that already prototype on live web applications could turn annotated mockups directly into backlog-ready user stories.
• Regulated or specialist domains (the paper's example is botany and field survey work) could use a glossary in the prompt to keep generated requirements in the vocabulary experts actually use.
• Product owners could reduce the manual translation step between what users draw and what developers implement.
• Requirements engineers' domain analysis could gain a second use: the LEL they already produce becomes prompt context rather than documentation that goes unused.

Industry relevance. The authors frame the approach as reducing ambiguity in requirement definition and speeding up the initial phases of development, while acknowledging that extra work is needed to build the LEL glossary and that the benefit is likely largest in complex domains or teams without experienced workers.

Future Directions

• Broader validation. The authors call for further validation to assess adaptability to diverse application domains and robustness in handling complex requirements, since the reported evidence comes from two case studies plus a five-action preliminary test.
• Bias and alignment. Addressing potential biases in language models and refining the automation process so that output aligns with stakeholder expectations are described as key open challenges.
• Empirical studies in real settings. Future work includes empirical studies to measure how effective the integration is in real-world software development scenarios.
• Hybrid approaches. The authors propose exploring combinations of AI-generated content with human validation to improve reliability.

Target Audience

Requirements engineers and analysts who produce domain glossaries and user stories, product owners and agile practitioners looking to shorten the path from prototype to backlog, researchers working on LLM applications in software engineering and on automated requirements elicitation, teams working in specialized or niche domains where terminology precision matters, and students or newcomers to requirements engineering who want a compact, example-driven view of how mockups, glossaries, and LLMs can be combined. Readers looking for model comparisons, benchmark numbers, or reproducible quantitative evaluation will not find them here; this is a conceptual process proposal with illustrative case studies and a small preliminary scoring exercise.

Authors’ abstract

User stories are one of the most widely used artifacts in the software industry to define functional requirements. In parallel, the use of high-fidelity mockups facilitates end-user participation in defining their needs. In this work, we explore how combining these techniques with large language models (LLMs) enables agile and automated generation of user stories from mockups. To this end, we present a case study that analyzes the ability of LLMs to extract user stories from high-fidelity mockups, both with and without the inclusion of a glossary of the Language Extended Lexicon (LEL) in the prompts. Our results demonstrate that incorporating the LEL significantly enhances the accuracy and suitability of the generated user stories. This approach represents a step forward in the integration of AI into requirements engineering, with the potential to improve communication between users and developers.