TalkSketch: Multimodal Generative AI for Real-time Sketch Ideation with Speech

AI summary

Overview

Research area: Human-Computer Interaction (HCI), specifically creativity-support tools, conversational user interfaces, and multimodal generative AI for design ideation.

Technical level: Intermediate. The paper is readable for design and HCI audiences; it describes a working system built with web canvas libraries and commercial multimodal model APIs, but assumes some familiarity with generative AI workflows.

Scope: The paper reports a formative study with six designers on how they use generative AI during early-stage sketching, and then presents TalkSketch, an embedded sketching interface that captures speech while a user draws and feeds both sketch and transcript into a multimodal AI chatbot.

What This Paper Is About

Designers increasingly reach for generative AI chatbots during early-stage ideation, but they struggle to write effective prompts and find text alone a poor fit for expressing visual concepts that are still evolving. The authors ran a formative study (N=6) showing that text-based prompting interrupts creative flow and forces constant switching between sketching apps and AI tools. They respond by building TalkSketch, a system where freehand drawing and spontaneous speech are captured together and passed to an embedded multimodal AI chatbot that generates context-aware responses.

Key Contributions

1. Formative study findings (N=6) on the challenges designers face when using current generative AI tools for early-stage sketch ideation, drawn from a design task plus pre- and post-task interviews. (The paper lists this as its first explicit contribution.)

2. TalkSketch, a sketching interface with a multimodal AI chatbot that lets users draw and verbalize ideas simultaneously. (The paper lists this as its second explicit contribution.)

3. Three design goals derived from the post-task interviews, which the authors use directly as the specification for the system: integrate AI into sketching tools, reduce fatigue from long prompts, and make the AI more proactive and context-aware.

4. A concrete system architecture and prompt design: a Fabric.js canvas, automatic voice capture transcribed through Google Cloud Speech-to-Text, an AI Insights panel and chat interface running on Gemini models, and two reusable prompt templates (Kickoff and Refine) grounded in the Double Diamond framework.

Main Findings

• Text prompts interrupt creative flow. The formative study revealed that text-based prompting often breaks the designer's creative flow and creates a disconnect between thinking up ideas and sketching them.

• Three recurring usage patterns appeared in the design task. Participants used GenAI for research and ideation (for example asking about common toaster problems), used it to render sketch-based ideas (uploading sketches to ChatGPT Image or Gemini), and moved in iterative loops among sketching, prompting, and referencing tools.

• AI responses were often too generic. P3 described initial AI responses as "too generic," noting they only became actionable after repeated prompt refinement.

• Generated images frequently failed to match intent. P1 called the results "kind of crazy," P2 concluded "I might have to redraw the whole thing so AI can understand," and P5 said "It takes too much time, I'd rather just draw." Many participants returned to manual sketching for clarity and control.

• Workflow fragmentation was a consistent pain point. Frequent switching between tools slowed the process, and P1 noted that "uploading sketches to ChatGPT was not so easy to operate on iPad."

• Three specific challenges summarized across patterns. Too-generic AI responses requiring extensive refinement; mismatched or low-quality image outputs; and fragmented workflows from tool switching.

• Designers asked for less typing. P2 said he was "too lazy to retype the whole thing," and P5 preferred to "just draw" rather than describe every detail. Participants wanted alternatives such as annotated sketches, voice commands, or real-time drawing.

• Designers wanted a more proactive AI. P3 proposed the assistant "should know what I'm trying to do," P4 suggested it should "see what I'm drawing and just give suggestions," and P2 wanted a system that could "first discuss the sketch with me" before rendering anything.

• The paper reports no user evaluation of TalkSketch itself. Section 5 is titled "Potential Results" and describes expected benefits for intent expression and creative flow rather than measured outcomes. Section 7 states that a controlled user study remains future work.

Methodology in Plain English

The work proceeds in three stages.

First, a formative study with six designers (2 female, 4 male), recruited with at least one completed design-related course such as Urban Sketching or Design Thinking and Innovation. The group included three students, two entry-level designers, and one experienced design practitioner, with backgrounds spanning architecture, furniture, interior, robotics, and electronic product design, and 2–5 years of part- or full-time design experience. Each participant received approximately USD 7.8 for a 1-hour session split into a 10-minute pre-task interview, a 30-minute design task, and a 20-minute post-task interview. In the design task they were asked to design a household bread toaster and express as many ideas as possible in 30 minutes using sketching apps such as Goodnotes or Procreate, on their own devices or on an Apple iPad Pro 13" with Apple Pencil, alongside GenAI tools such as ChatGPT (GPT-4o), Gemini, and Midjourney. The team analyzed qualitative data from both the task interactions and the interview reflections.

Second, the authors turned those findings into three design goals and built TalkSketch around them. The sketching module uses Fabric.js and supports drawing, erasing, selection, undo/redo, and canvas reset, plus a Save to Gallery feature. The talking module starts audio recording automatically whenever the user sketches without the chatbot open, and stops when the chatbot opens, streaming the audio for low-latency transcription via Google Cloud Speech-to-Text. The multimodal chatbot has two parts: AI Insights, which automatically produces structured reflective feedback from the latest voice transcript and current canvas, and a chat interface supporting text prompts, the iPad's voice dictation, and selected canvas regions exported as image inputs. Text conversation runs on Gemini 2.0 Flash; image generation runs on Gemini 2.5 Flash Image; AI Insights runs on Gemini 2.5 Flash. A unified conversation history persists across both models.

Third, the authors illustrate the workflow with a walkthrough featuring a fictional user named Sky designing a toaster, showing her sketch, her spoken description, the automatically triggered AI Insight, follow-up text and image prompts, and exporting an AI-generated image back onto the canvas as a reference.

The two AI Insights prompt templates are stated verbatim in the paper. The Kickoff Prompt asks the model to act as a design thinking expert, use the Double Diamond framework to highlight user needs, pain points, and framing questions, and offer 3–4 concise design directions in an encouraging tone at around 100 words. The Refine Prompt asks the model to act as a design thinking collaborator, summarize the current design in one or two sentences, suggest 1–2 small ways to expand or clarify it, and end with 1–2 open-ended questions in around 80–100 words.

Why This Matters

Impact on research. The paper shifts attention from AI-generated output to the design process itself, arguing that generative tools should engage with how designers actually think rather than only what they produce. It contributes empirical evidence about prompt fatigue and flow disruption in early-stage design, and it proposes sketch-plus-speech as a combined input modality for multimodal language models, a combination the authors state has not been systematically studied for its effect on subsequent GenAI interaction.

Real-world applications.

• Embedded design assistants inside sketching apps that respond to rough visuals and thinking-aloud narration instead of typed prompts.
• Live demonstrations and presentations where a speaker sketches while explaining, and rapid idea communication matters.
• Online classrooms where instructors draw while narrating concepts and want contextual AI support.
• Collaborative design reviews where teams annotate evolving visuals and need shared, context-aware feedback.

Industry relevance. Commercial platforms such as Adobe Firefly, Canva Magic Studio, and Figma AI already push generative models into design workflows, and research prototypes like DesignPrompt, DesignWeaver, Inkspire, and SketchAI explore richer input for image generation. TalkSketch suggests a direction for product teams building creativity tools: capture the spontaneous speech that already happens during sketching as a first-class input, rather than requiring users to translate visual thinking into text.

Future Directions

• A controlled user study to systematically evaluate how TalkSketch affects the naturalness of human–AI interaction and the perceived creativity of the design process, which the authors explicitly name as the next step.
• Finer-grained alignment between speech and sketch regions. The system currently treats speech and sketch as one combined chunk over a period of time, so it cannot tell which part of a spoken description refers to which part of the drawing, and important links between what is said and what is drawn may be missed.
• Robustness to transcription failure and silent sketching. The authors note that background noise, unclear pronunciation, or mishearing can corrupt the speech input, and that some users may prefer not to speak at all, in which case the system falls back to sketches and explicit prompts with a weaker signal for understanding intent.
• Generalizing the interaction paradigm beyond creativity support, including live demonstrations, online classrooms, and collaborative design reviews, to see whether sketch-and-speech interaction works as a more versatile interface style.

Target Audience

This paper is most useful for HCI researchers working on creativity-support systems, multimodal interaction, and conversational user interfaces; designers and design educators curious about how generative AI can fit into early-stage ideation rather than final production; and product teams building sketching or design tools who want a concrete implementation pattern combining canvas input, automatic speech capture, and multimodal model APIs. Readers looking for quantitative evaluation results should note that the paper reports a formative study of six participants and an implemented system with a walkthrough, while the empirical evaluation of TalkSketch itself is left to future work.

Authors’ abstract

Sketching is a widely used medium for generating and exploring early-stage design concepts. While generative AI (GenAI) chatbots are increasingly used for idea generation, designers often struggle to craft effective prompts and find it difficult to express evolving visual concepts through text alone. In the formative study (N=6), we examined how designers use GenAI during ideation, revealing that text-based prompting disrupts creative flow. To address these issues, we developed TalkSketch, an embedded multimodal AI sketching system that integrates freehand drawing with real-time speech input. TalkSketch aims to support a more fluid ideation process through capturing verbal descriptions during sketching and generating context-aware AI responses. Our work highlights the potential of GenAI tools to engage the design process itself rather than focusing on output.