Node-Based Editing for Multimodal Generation of Text, Audio, Image, and Video

AI summary

Overview

Research area: Human-Computer Interaction (HCI), specifically creative AI interfaces for multimodal content generation (text, audio, image, video), with a focus on narrative authoring tools.

Technical level: Intermediate. The paper describes a system architecture and a small evaluation rather than new model training, so the core ideas are accessible, though familiarity with LLM orchestration and generative media pipelines helps.

Scope: This paper presents a node-based (graph) storytelling interface that lets users generate, edit, branch, and export multimodal narratives by combining natural-language prompts with direct node-level editing.

What This Paper Is About

Most generative AI tools rely on single prompts, which the authors argue do not fully capture what users intend, and existing AI storytelling systems tend to produce linear, non-iterative workflows. This paper builds a system where a story is a graph of nodes, each node holding a text segment and its associated audio, images, or video, so creators can make targeted edits, branch parallel storylines, and refine content iteratively. The goal is to give creators explicit control over narrative structure while still using generative models for text, audio, images, and video.

Key Contributions

1. A node-based framework for AI-assisted storytelling: Stories are represented as directed graphs where each node is a scene or event, making narrative structure explicit and allowing users to branch, reorder, or refine content.
2. A task-selection orchestration layer: A task selection agent routes user requests and node-diagram changes to specialized LLM-driven tasks — Generator, Reasoner, Diagrammer, Editor, and Context Generator — built on GPT-4.1.
3. Selective, node-level multimodal media editing: Users can generate audio (GPT-4o text-to-speech), images (GPT-Image-1), and video (OpenAI's Sora) for a selected subset of nodes, with a rolling story context carried forward for visual consistency.
4. Iterative refinement through branching and comparison: Users can duplicate nodes or branches to produce alternative versions of a storyline and compare them side by side, and export results as a compiled video, a visual storyboard, or a JSON graph.

Main Findings

• Automated story node generation was evaluated on 20 prompts: 10 prompts for single (linear) narratives and 10 for branching narratives, each constrained to roughly 8 to 12 nodes.
• Linear stories succeeded in 8 out of 10 trials: The system produced linear story graphs without unintended branching in 8/10 trials (80%, 95% CI [44%–97%]).
• Branching stories succeeded in 10 out of 10 trials: For branching prompts, the system generated parallel paths in 10/10 trials (100%, 95% CI [69%–100%]). The two failing linear prompts in the appendix were "A group of astronauts lands on Mars..." and "A teacher prepares their students for an important exam...".
• Manual and AI editing served different purposes: Manual editing was most effective for targeted changes such as specific objects, settings, or scene details; AI-assisted editing was more useful for structural or stylistic revisions, such as tone changes ("make this sound mysterious"), extending descriptions ("add the fact that her backpack is on the ground"), or condensing narration.
• Global edits preserved graph structure: Users could select all nodes and request a rewrite — for example, "make these parts shorter and sound adventurous" — which altered narrative details across the story while keeping the branching structure intact.
• Branching enabled non-destructive comparison: Duplicating nodes or branches allowed side-by-side comparison of alternative media and text versions, which the authors state was not possible in linear prompt-based systems and which they argue is valuable given the time and computational cost of video generation.
• Compute was modest: All experiments used deployed models via Azure endpoints (Sora hosted on Azure) and OpenAI endpoints; orchestration, interface, and evaluations ran locally on a standard laptop without GPUs. Most node-based image and video generation tasks completed within 10–30 seconds per API call.
• Not reported: The paper does not report a human-subjects or crowdsourced user study, nor any measurements of cross-node media consistency, long-narrative performance, or cost.

Methodology in Plain English

The researchers built an interactive system in which a story is drawn as a graph of cards (nodes) connected by edges. When a user types a story request, a "task selection agent" decides which specialized task should handle it: the Generator writes narrative text, the Reasoner breaks that text into nodes and assigns edges for narrative flow, and the Diagrammer formats the result into a strict JSON structure with node titles, node segments, and edges. Targeted user requests go to the Editor, which regenerates selected nodes while keeping the surrounding structure.

Each node's text segment then doubles as the prompt for media generation. A rolling story context is passed to image and video generation to promote consistency across nodes, audio narration is produced from each node's text, and users can select any subset of nodes to generate or regenerate media without changing the text. Manual edits made directly in a node immediately propagate to the exported outputs.

To evaluate the approach, the authors generated 20 test prompts with an LLM (10 linear, 10 branching, each describing roughly 8–12 events) and checked whether the system's generated graphs matched the intended structure. Success rates and binomial confidence intervals were reported, and the appendix lists every test prompt with a pass/fail result. Additional observations about editing workflows and branching behaviour were qualitative rather than measured.

Why This Matters

Impact on research: The paper argues that one-shot prompting is insufficient for controllability, and it demonstrates a concrete alternative: making narrative structure an explicit, editable graph that sits between the user and the generative models. It contributes a way to connect high-level prompt-based editing with low-level, node-targeted edits, and it frames controllability as a way to preserve human agency in creative AI loops.

Real-world applications (as discussed in the paper):

• Filmmaking and cinematic clip production, including scene-level continuity and storyboard-style review before export.
• Game design and interactive media, where branching, parallel timelines, and multi-perspective narratives are natural fits.
• Education and community-driven storytelling, by lowering the barrier for non-technical users to direct advanced models.
• Independent media production, allowing creators without professional production pipelines to produce expressive multimodal works.

Industry relevance: The system orchestrates existing commercial model APIs (Azure-hosted Sora, OpenAI GPT-4.1, GPT-Image-1, GPT-4o TTS) rather than training new models, and it runs its interface on a standard laptop, which makes the interaction paradigm directly relevant to product teams building creator tools. It also connects to an existing lineage of node-graph editors and storyboarding tools, positioning graph-based, non-destructive editing as a competitor to linear timeline metaphors.

Future Directions

1. Image grounding for cross-node consistency: The authors identify reliance on text-based context grounding as a key limitation and propose integrating image grounding, or grounding generated media in real-world data, for more coherent media across multiple nodes.
2. Scaling to longer narratives: Handling longer text and larger node graphs is an open problem; hierarchical generation or subgraph-based approaches are proposed to preserve clarity and narrative coherence.
3. User studies with creators: The authors plan to conduct user studies with content creators to gather feedback on usability and the interface's impact on their creative process.
4. Responsible deployment safeguards: The paper notes risks of misuse for misinformation or low-quality mass content generation, and suggests mitigation through provenance tracking, attribution, watermarking, and fingerprinting — though it does not report implementing these in the current system.

Target Audience

This paper is most useful to HCI researchers and interface designers working on human-in-the-loop creative AI, to practitioners building generative content tools who want a concrete alternative to linear prompt-and-timeline interfaces, and to creators in film, games, and interactive media who want structural control over AI-generated narratives. Researchers interested in controllability, iterative refinement, and multimodal orchestration will find the task-routing architecture and the small structural evaluation most relevant.

Authors’ abstract

We present a node-based storytelling system for multimodal content generation. The system represents stories as graphs of nodes that can be expanded, edited, and iteratively refined through direct user edits and natural-language prompts. Each node can integrate text, images, audio, and video, allowing creators to compose multimodal narratives. A task selection agent routes between specialized generative tasks that handle story generation, node structure reasoning, node diagram formatting, and context generation. The interface supports targeted editing of individual nodes, automatic branching for parallel storylines, and node-based iterative refinement. Our results demonstrate that node-based editing supports control over narrative structure and iterative generation of text, images, audio, and video. We report quantitative outcomes on automatic story outline generation and qualitative observations of editing workflows. Finally, we discuss current limitations such as scalability to longer narratives and consistency across multiple nodes, and outline future work toward human-in-the-loop and user-centered creative AI tools.