On Improvisation and Open-Endedness: Insights for Experiential AI

AI summary

Overview

Research area: Human-Computer Interaction (cs.HC), bridging computational creativity, artificial life (ALife), and the study of human improvisational arts.

Technical level: Intermediate. The paper is qualitative and conceptual rather than quantitative — there are no benchmarks, datasets, or model evaluations — but it assumes familiarity with ideas from open-endedness, complex systems, and human-AI interaction research.

Scope: A work-in-progress paper reporting themes from interviews with six expert improvisers to derive design principles for future "Experiential AI" that could improvise alone, with humans, or with other AI agents.

Author: Botao 'Amber' Hu (University of Oxford, Oxford, UK; Reality Design Lab, New York City, USA). Posted as arXiv:2511.00529v2 [cs.HC] on 05 Nov 2025 under a CC BY-NC-ND 4.0 license.

What This Paper Is About

Improvisation is a spontaneous, unrepeatable form of human creativity found in jazz, dance, contact improvisation, and everyday life, and its open-ended character — producing continuous novelty without a scripted endpoint — is closely related to the open-endedness (OE) prized in artificial life research. The paper asks what makes a "good" improvisation for AI, and whether AI can learn to improvise in a genuinely open-ended, experientially creative way rather than merely simulating improvisation. To answer this, the authors interview expert practitioners and translate their wisdom into design principles for future AI agents.

Key Contributions

1. An interview study with six expert improvisers. One-on-one, in-depth, semi-structured interviews were conducted with participants P1–P6, spanning contemporary dance, jazz music, live coding performance, and contact improvisation, each with over 10 years of improvising experience. Interviews lasted 60–120 minutes and were conducted in participants' preferred language, with translation as needed.

2. Eight distilled themes of improvisational practice. Through open coding of transcripts refined into higher-level conceptual categories, the authors identify eight major themes: active listening (umwelt and awareness), being in the time (mindfulness and ephemerality), embracing the unknown (source of randomness and serendipity), non-judgmental flow (acceptance and dynamical stability), balancing structure and surprise (unpredictable criticality at the edge of chaos), imaginative metaphor (synaesthesia and planning), empathy/trust/boundary/care (mutual theory of mind), and playfulness and intrinsic motivation (maintaining interestingness).

3. A bridge between improvisational art and open-ended AI research. The paper connects practitioner insights to concepts from complex systems, cognitive science, and HCI, arguing that current generative AI can appear open-ended but does not truly achieve open-endedness — it often repeats patterns or clichés, essentially regurgitating averages of training data, and may lack agency.

4. Design implications for each theme. Each of the eight themes is paired with an "Implications" discussion describing how an Experiential AI might embody that quality — for example, outside-in multimodal sensing (active listening), modeling temporality and endings (being in the time), and treating uncertainty as a learning signal rather than an error (embracing the unknown).

Main Findings

• Active listening comes before generation. Every interviewee stressed deep, active listening — being attuned to environment, partners, and self. P2 (jazz musician) described improvisation as "a conversation. If you're not truly listening, you're just playing pre-set ideas and the magic is lost." P5 described "constantly scanning the room — the hum of the speakers, the audience's energy — and also scanning myself. It's like a radar sweeping." The design implication is that AI must sense its umwelt in an outside-in, embodied way rather than operating as an inside-out next-token predictor.

• Ephemerality gives improvisation its shape. P6 (contact improviser) said, "Because it will end, it's beautiful — like life." P1 (dancer) noticed some of her most creative ideas in a jam session emerge in the final minutes, as if the "looming ending gives a sense of urgency and freedom." P5 said, "If we could improvise forever, maybe we'd take things for granted, but because time is limited, we dare to make bold moves." The paper notes that current AI lacks temporal embodiment — it does not live, tire, or die.

• Uncertainty and serendipity are creative resources. P2 described improvisation as "the art of not knowing what comes next, and being excited by that." P4 recounted accidentally discovering a new movement through a miscommunication that led to an unexpected lift which "felt like flying for a second." P1 said, "You have to jump off the cliff sometimes," describing a solo where she decided not to look at the piano keys at all. AI systems, by contrast, are described as optimizing within pre-trained data manifolds and minimizing uncertainty.

• Non-judgmental flow sustains continuity. P4: "When you're improvising, you can't be judging every move as 'good' or 'bad' — if you do, you hesitate and the flow is gone." The paper links this to the improv theatre "Yes, and" rule, to Csikszentmihalyi's concept of flow, and to neuroscience findings that self-monitoring brain regions deactivate during improvisation (Limb and Braun 2008). Current AI is characterized as overly judgmental, filtering outputs through rigid objectives and reward functions.

• The "sweet spot" lies between order and chaos. P4 said, "If it's too predictable, it gets boring; if it's too random, it falls apart." P2 described jazz musicians going "outside" the chord changes and resolving back "inside": "It's like we stretch the rubber band of the music but don't let it snap." P4 reframed two contact improvisation principles as self-sustain ("self-sustaining — like wu wei in Daoist philosophy") and reversibility ("all your actions should be reversible... Reversibility gives you the maximal possibility for evolution").

• Imagination and metaphor guide improvised action. P6: "Sometimes I imagine I'm water flowing, or a tree bending in the wind, and that shapes my movement." P5 reported an image of "a carnival at midnight" leading to unusual scales and rhythms. P6 stated, "Imagination is life. Only the one that can imagine the future can have that future." The paper argues today's AI lacks this cross-modal, metaphorical resonance.

• Trust, care, and boundaries are structural, not decorative. P6 noted that in contact improvisation "you have to trust your partner with your body." P4 described "an unspoken ethos of generosity. You let the other person shine sometimes, you take care of their ideas." P2 simplifies his playing when he senses another musician struggling, giving them space. Boundaries are explicitly taught — it is acceptable to say "no" or disengage — and respecting them builds deeper trust.

• Playfulness and intrinsic motivation maintain interestingness. P2: "Honestly, it's just fun." P1 described "following the sparkle" when something interesting emerges. P4 uses improv games in his classes "to remind adults how to play like kids again." P5: "In improv you get to assert your musical will — you're not following a score, you're writing it as you go, together." The paper contrasts this intrinsic drive with AI agents that pursue extrinsic objectives and optimize reward.

• No quantitative results are reported. The paper presents a qualitative thematic analysis only; it does not report benchmarks, dataset sizes, or model performance figures.

Methodology in Plain English

The authors conducted six one-on-one, semi-structured interviews with expert improvisers rather than running experiments or training models. Participants were recruited from distinct improvisation domains — contemporary dance, jazz music, live coding performance, and contact improvisation — and each had over 10 years of experience in their field. The interview protocol asked questions such as "What is a 'good' improvisation to you? What about a 'bad' improvisation?", "Can you describe what goes through your mind, moment-by-moment, when you improvise?", "How do you keep an improvisation interesting (and conversely, what makes it become boring)?", and "Have you ever improvised with AI or technology? How do you imagine improvising with an AI partner might compare to with a human?"

The researchers then used an open coding approach to identify recurrent themes in the transcripts and refined these into higher-level conceptual categories, arriving at eight themes. Each theme is presented in the Results section together with practitioner quotes and examples, and each is followed by an "Implications" subsection that connects the human practice to possible designs for AI agents, drawing on related work in complex systems, cognitive science, and HCI. The paper does not report a formal limitations section, saturation analysis, or inter-rater reliability procedure.

Why This Matters

For research, the paper argues that human improvisation offers a concrete, practice-derived vocabulary for the "last grand challenge" of open-endedness in artificial life and for computational creativity, described in the paper as AI's "final frontier." It situates its claims alongside recent arguments that AI is moving from an "era of human data" to an "era of experience" (Silver and Sutton 2025), and that open-endedness may be essential for any future artificial superintelligence (Hughes et al. 2024).

Real-world applications suggested by the paper's framing:

• Human-AI co-performance, building on systems such as LuminAI (Trajkova et al. 2024), Human–AI co-dancing (Pataranutaporn et al. 2024), and Improbotics (Mathewson and Mirowski 2018), where an AI partner must sustain interestingness rather than produce dull or chaotic output.
• Robotic and non-humanoid stage partners, extending work like Dancing with Drones (Dong et al. 2024) and Drone Chi (La Delfa et al. 2020), where an "intercorporeal understanding" must be developed between performer and machine.
• Musical collaboration tools, following the AI folk fiddler work of Benford et al. (2024), where performers negotiate autonomy and trust — for instance, by pre-setting the AI's "looseness of control" and using simple intensity cues during performance.
• Everyday interactive systems, given the paper's framing that improvisation extends beyond the arts to everyday conversation and that "life is improvisation."

Industry relevance: the paper suggests that agents built with these qualities would possess agency and playfulness, perceiving interestingness as an evolving internal drive rather than an imposed metric, which points toward interactive products that feel alive and co-creative rather than purely command-driven generators.

Future Directions

• How to operationalize the eight themes in real systems. The paper offers design implications for each theme — real-time multimodal sensing, modeling temporality and endings, treating serendipity as a learning signal, delaying evaluation, maintaining behavior near the edge of chaos, cross-modal metaphorical mapping, mutual theory of mind, and intrinsic playfulness — but does not specify how any of these would be implemented or evaluated.

• How to sustain "interestingness" over time. The authors identify as an open challenge preventing AI from generating dull or chaotic outputs and enabling it to meaningfully evolve its behavior over time, and ask whether AI can improvise in a genuinely open-ended, experientially creative way.

• What "good" improvisation means for AI. The paper's central question — "What makes a 'good' improvisation for AI?" — is posed rather than answered, leaving room for empirical work that would test these principles in human-AI or AI-AI improvisation.

• Building AI improvisers with their own intentions. The paper raises the need for each agent, human or AI, to have its own intentions and ability to act on them — an "assertion of musical will" — as distinct from being a passive follower of pre-set rules, without specifying how such agency would be engineered.

• Limitations not reported. The paper does not report limitations of the interview study, and its expansion to AI-AI improvisation remains speculative.

Target Audience

Researchers and practitioners in human-computer interaction, computational creativity, artificial life, and generative AI who are interested in open-endedness, interactive music and dance systems, or human-AI co-creativity. It is also relevant to artists and designers working with AI performance partners, and to readers who want a qualitative, practice-grounded counterpoint to purely optimization-driven accounts of machine creativity. Because the paper is conceptual and includes practitioner quotes with minimal technical formalism, it is accessible to newcomers to ALife and open-endedness, though some familiarity with those debates helps.

Authors’ abstract

Improvisation-the art of spontaneous creation that unfolds moment-to-moment without a scripted outcome-requires practitioners to continuously sense, adapt, and create anew. It is a fundamental mode of human creativity spanning music, dance, and everyday life. The open-ended nature of improvisation produces a stream of novel, unrepeatable moments-an aspect highly valued in artistic creativity. In parallel, open-endedness (OE)-a system's capacity for unbounded novelty and endless "interestingness"-is exemplified in natural or cultural evolution and has been considered "the last grand challenge" in artificial life (ALife). The rise of generative AI now raises the question in computational creativity (CC) research: What makes a "good" improvisation for AI? Can AI learn to improvise in a genuinely open-ended way? In this work-in-progress paper, we report insights from in-depth interviews with 6 experts in improvisation across dance, music, and contact improvisation. We draw systemic connections between human improvisational arts and the design of future experiential AI agents that could improvise alone or alongside humans-or even with other AI agents-embodying qualities of improvisation drawn from practice: active listening (umwelt and awareness), being in the time (mindfulness and ephemerality), embracing the unknown (source of randomness and serendipity), non-judgmental flow (acceptance and dynamical stability, balancing structure and surprise (unpredictable criticality at edge of chaos), imaginative metaphor (synaesthesia and planning), empathy, trust, boundary, and care (mutual theory of mind), and playfulness and intrinsic motivation (maintaining interestingness).