Sound Clouds: Exploring ambient intelligence in public spaces to elicit deep human experience of awe, wonder, and beauty

AI summary

Overview

Research area: Human-Computer Interaction, specifically ambient intelligence (AmI), interactive art installations, and experience-driven design in public spaces.

Technical level: Beginner-Friendly. The paper is a short demo/provocation paper rather than an algorithmic contribution; it describes a built installation, its sensing and sound-generation pipeline, and anecdotal audience response.

Scope: The paper introduces Sound Clouds, a public art installation of buoyant spheres that uses computer vision and rule-based music generation to turn full-body interaction into live sound, arguing that ambient intelligence can be designed for awe, wonder, and beauty rather than purely pragmatic ends.

What This Paper Is About

Most ambient intelligence systems people encounter are pragmatic: thermostats saving energy, ambient cues nudging behavior, or camera networks enhancing security. The authors ask how interactions with ambient AI in public spaces could instead elicit deep human feelings of awe, wonder, and beauty. Their answer is Sound Clouds, an immersive installation where participants move among floating human-height spheres that control live, algorithmically organized music, with no instructions given so that curiosity alone drives exploration.

Key Contributions

1. A designed artifact as provocation. The paper presents Sound Clouds as an exploratory ambient intelligence art installation, offering it as a concrete counterpoint to pragmatic AmI applications such as the Nest thermostat, ambient influence for promoting behavior change like taking the stairs, and LAICA for public security.

2. A complete sensing-to-sound architecture. It describes a pipeline in which a ceiling-mounted GoPro camera feeds video to a central-processing Mac running a fine-tuned YOLO12n object-detection model to obtain each sphere's location and diameter; sphere lights are driven via the ESP-NOW protocol to ESP32 boards embedded inside each sphere, and music is generated in Max/MSP.

3. A rule-based interaction design with three named scripts. The video feed is divided into nine zones, and music is produced by three scripts: cloudGrid (a sphere entering a new zone triggers a distinguished pre-recorded sound), orbProxi (two spheres close enough to each other, below a threshold, trigger a sound), and pitchShift (the depth of one sphere shifts the pitch of the background sound).

4. An account of a public exhibition and its experiential framing. The installation was experienced by more than 200 participants during a two-hour public exhibition, and the paper connects participant reactions to the literature on awe as a self-transcendent emotion, aligning the work with experience-prioritizing AmI research such as The Aware Home and Smart Kitchen.

Main Findings

• Installation form factor: Sound Clouds consists of several larger-than-human-height floating PVC spheres that were filled with helium and air to reach approximately neutral buoyancy. The abstract and introduction describe spheres at 8–12 feet in diameter, while Section 2 states diameters from 1.2 m to 3 m.
• Exhibition setting: It was exhibited inside a renovated factory building with interior dimensions of 18 m (width) × 42 m (length) × 9 m (height). Dry ice, fabric canopies hung from the ceiling, and projected "waves" of light on the ground created a serene, underwater-themed atmosphere.
• Scale of participation: More than 200 participants experienced the installation during a two-hour public exhibition, with at most 20 participants allowed to interact simultaneously.
• Division of AI and non-AI roles: Computer vision tracked the spheres, but music generation was rule-based through Max/MSP. Although the CV model tracked all spheres, only two spheres' parameters were used for music generation, and a looped ocean-like sound served as the background.
• Known technical caveat: Because of the GoPro's wide-angle lens and fish-eye view, the vision input was lightly distorted, as shown in the paper's top-down view of the exhibit space.
• Participant-reported experience: Post-interaction interviews yielded descriptions such as the ambiance and overall vibe being "peaceful and calming," an experience of "jaw on the floor" that "felt almost like dream-like," and a reflection that "sense of wonder and awe" differ in adulthood because the proportions are off.
• Evaluation scope: No formal quantitative evaluation, statistical analysis, or performance metrics for the computer vision model or music system are reported. The evidence for the experiential claims consists of participant quotations from post-interaction interviews.

Methodology in Plain English

The authors built and publicly exhibited an interactive art installation rather than running a controlled study. A ceiling-mounted GoPro camera looks down at the exhibition space and streams video over Wi-Fi to a central Mac. A fine-tuned YOLO12n object-detection model finds the floating spheres in that video and reports where they are and how big they appear, with the apparent diameter used to infer a sphere's height. That information drives two outputs. First, light signals are sent over the ESP-NOW protocol to ESP32 boards embedded inside each sphere so the spherical lights change according to position. Second, a Max/MSP patch generates live music using a small set of hand-authored rules: a nine-zone grid that triggers distinct pre-recorded sounds when a sphere enters a new zone, a proximity rule that fires a sound when two spheres come close enough, and a pitch-shifting rule tied to one sphere's depth. Only two spheres' parameters feed the music, and an ocean-like loop plays underneath. Crucially, participants receive no instructions or explanation of the system, so curiosity and embodied play drive their behavior. The authors then interpret interview responses through existing theory on awe and through prior AmI systems that prioritize experience and affect over efficiency.

Why This Matters

Impact on research. The paper argues for an expanded design space for ambient intelligence beyond efficiency, security, and behavior modification, and ties that argument to AmI literature on self-transcendent emotion: awe is described as rarely experienced and linked to feelings of interconnectedness, motivation to commit to social collectives, and improved perception of well-being. It positions Sound Clouds alongside experience-focused AmI lineages such as The Aware Home and Smart Kitchen, framing the installation as a provocation that enriches rather than replaces humanity.

Real-world applications:

• Public art and museum installations where movement and presence shape generative sound and light.
• Responsive architectural and civic spaces that use ambient sensing to create shared, contemplative experiences rather than surveillance.
• Interaction design for full-body, instruction-free interfaces in which curiosity, not onboarding, guides learning.
• Experience design education and prototyping, as a worked example of layering computer vision, embedded hardware, and rule-based audio.

Industry relevance. The work is relevant to companies building ambient and spatial computing products that need alternatives to purely utilitarian framing, and to entertainment, hospitality, retail-environment, and smart-building sectors interested in affect and atmosphere as design goals. It also illustrates an inexpensive, modular technical stack (commodity camera, Mac, ESP32 boards, Max/MSP) that practitioners can adapt.

Future Directions

• Whether the installation's experiential effects can be measured more rigorously than through participant quotations, since the paper reports no formal evaluation or metrics.
• How the system would behave at larger scale or over longer durations, given that the reported exhibition ran for two hours with more than 200 participants and a simultaneous-participant cap of 20.
• How music generation might move beyond the current rule-based Max/MSP scripts and the use of only two spheres' parameters, and how the lightly distorted wide-angle vision could be corrected.
• How the "provocation" framing generalizes: what design principles for awe-, wonder-, and beauty-oriented ambient intelligence can be abstracted from a single installation and applied to other public spaces.

Target Audience

This paper is most useful to HCI researchers and interaction designers working on ambient intelligence, public-space installations, and experience-driven or affective computing; to artists and creative technologists building immersive, sensing-driven environments; and to practitioners in spatial computing, smart buildings, and entertainment who want a concrete example of assembling computer vision, embedded lighting, and generative audio into an ambient experience. Its conceptual framing suits readers seeking design provocations; readers seeking quantitative evaluation or algorithmic novelty will not find it here.

Authors’ abstract

While the ambient intelligence (AmI) systems we encounter in our daily lives, including security monitoring and energy-saving systems, typically serve pragmatic purposes, we wonder how we can design and implement ambient artificial intelligence experiences in public spaces that elicit deep human feelings of awe, wonder, and beauty. As a manifestation, we introduce Sound Clouds, an immersive art installation that generates live music based on participants' interaction with several human-height spheres. Our installation serves as a provocation into future ambient intelligence that provokes, not limits, the future possibilities of AmI.