Sensorium Arc: AI Agent System for Oceanic Data Exploration and Interactive Eco-Art

AI summary

Overview

Research area: Artificial Intelligence — specifically conversational multi-agent LLM systems, retrieval-augmented generation (RAG), and immersive eco-art installation design for environmental data exploration.

Technical level: Intermediate. The paper assumes familiarity with LLM pipelines, RAG, embeddings, and vector search, but explains the artistic and interaction design motivations in accessible terms.

Scope: This paper describes the design, architecture, and deployment context of Sensorium Arc (AI reflects on climate), a real-time multimodal AI agent system that personifies the ocean as a poetic speaker and responds to spoken user questions with grounded narration, data visualizations, video, and synthesized speech.

What This Paper Is About

Massive public environmental datasets exist, but the paper argues the problem is not a lack of data — it is the absence of intuitive, affective, and situated ways to engage with it. Sensorium Arc addresses this by letting a participant whisper a question into a conch/Nautilus-shaped interface and receive an answer spoken in the voice of the Ocean, blending scientific grounding with ecological poetics. The goal is to turn oceanic data from a static dataset into a dialogic, co-authored experience rather than an abstract visualization.

Key Contributions

1. A modular multi-agent RAG architecture for a non-human persona. The system decomposes the task into three specialized LLM agents — a Visualization Decider Agent, a Retrieval with Query Rewriter Agent, and a Responder Agent — built by extensively modifying the open-source LLMUnity framework, rather than using a single model for all tasks.

2. Upstream, grammar-constrained visual selection. Instead of choosing visuals after generating a response, a small stateless decision agent runs before the main response step and outputs a visual selection token. The authors used few-shot examples, explicit reasoning, and GBNF grammar constraints with the Llama 3.2 3B model to restrict output to only the available options.

3. RAG grounding in an eco-art and marine science corpus. Unstructured texts — including The Time of the Force Majeure, "Apologia Mediterraneo" (created as part of Counter Extinction V), and a catalog for Peninsula Europe I from the Harrisons' archive — were pre-processed into paragraphs, split into sentences, embedded into a 384-dimensional vector space using all-MiniLM-L12-v2, and indexed for approximate nearest neighbor search via the usearch library, with a query rewriting agent based on Ma et al.'s approach.

4. An embodied interaction and audiovisual system. A Nautilus-motif interface built with Arduino-based control, a distance sensor, LED illumination, and a motor triggers recording when a participant comes within 50 cm. Responses are mapped through keyword matching to globe visualizations, pre-rendered video, and text-to-speech audio with synchronized subtitles.

Main Findings

• Latency: With CUDA acceleration offloading a configurable number of GPU layers from the CPU, average response latencies were under 4 seconds after user input.

• Single-model designs underperformed: Early trials revealed that a single model handling all tasks offered reduced fine control and suffered from prompt interference and opaque errors, motivating the switch to specialized role-driven agents.

• Retrieval depth: Initial testing led to a low value of k = 2, because retrieving more paragraphs often filled the final LLM step's context window and muddled response generation.

• Agent selection: Qwen 8B was chosen for the query rewriting agent after comparative testing on a set of representative queries, for its strong rule-following behavior, and basic chain-of-thought prompting improved output quality.

• Prompting trade-offs: A well-formed zero-shot prompt worked well for the Responder Agent in most cases, but did not guarantee consistent response characteristics such as length or format. Larger models improved quality over smaller models such as Llama 3.2 3B but increased latency.

• Prior iteration insight: Earlier Sensorium iterations in immersive installation and VR formats, presented at the AlloSphere exhibition as part of the Getty PST Art: Art and Science Collide initiative, showed high sensory engagement but revealed that audiences sought direct dialogue with the ocean — the insight that motivated the responsive AI narrator.

• Datasets visualized: Atmospheric CO2 emissions, chlorophyll concentration, sea surface temperature, ocean currents, and hyperspectral diffuse attenuation coefficients (Kd) from NASA Earthdata and earthaccess. Water clarity is rendered using 16 stacked layers of Kd data from NASA's PACE satellite (2024).

• Video content: Videos concerning plankton blooms, ocean acidification, plastic waste dispersion, and sea-level rise were pre-rendered in Unity for performance efficiency given available computing resources.

• Hardware and runtime: Development and testing used Unity version 6000.0.24f1 on Windows 11 with an Intel Core i9-13900HX CPU, 64 GB RAM, and an NVIDIA GeForce RTX 4090 Laptop GPU. Additional testing on macOS employed Vulkan for GPU acceleration.

• Not reported: The paper does not report results from controlled user studies on retention, empathy, or behavioral change — these are listed as planned future steps rather than completed findings.

Methodology in Plain English

A participant approaches a Nautilus-shell interface. A distance sensor detects whether they are within 50 cm; blue light ripples across the shell to signal readiness, and a microphone with active noise cancellation begins recording. Recording stops when the user moves beyond that range.

The captured audio becomes text and enters a pipeline of specialized, mostly stateless agents. First, a small decision agent looks at the query plus descriptions of every prepared visual and picks one visual by outputting a token, with grammar constraints forcing the output to match an available option. Second, a rewriting agent reformulates the user's question to better match the stored documents and strips out conversational filler, reasoning about the query before producing the rewrite. Third, a semantic search embeds the rewritten query with the same local embedding model used on the corpus and retrieves the k = 2 most similar sentences along with their source paragraphs.

The final Responder Agent receives three things: a description of the selected visual if one was chosen, two retrieved paragraphs, and the original user query. It produces the Ocean's reply. That reply then goes through a keyword and token matcher that fires corresponding audiovisual events — globe camera moves, data layer activation, video playback — while a text-to-speech model in Unity Sentis converts it into audio with synchronized subtitles. All of this runs inside the Unity engine so layers can be flexibly triggered and combined.

Why This Matters

The paper proposes that conversational AI agents can serve as mediators for affective, intuitive access to high-dimensional environmental data, and frames the result as a generalizable paradigm for human-machine-ecosystem interaction. It also argues that sustainability can be reframed as a shared cultural practice shaped through co-authorship with non-human agents, rather than only a technical or environmental metric.

Real-world applications:

• Museums and art-science exhibitions: The system is designed for a mobile form factor suitable for public exhibition, and the software installs on both Windows and macOS to maximize accessibility.

• Schools and public institutions: The authors state they are planning educational installations for schools and public institutions.

• Coastal community communication: Related tools cited by the paper (Rising Together, Dive, and the Sea Level Rise Explorer series) support coastal communities in understanding sea-level rise impacts and adaptation strategies; Sensorium Arc extends this space toward dialogue rather than static presentation.

• Accessible environmental data interfaces: Synchronized subtitles accompany the TTS audio, and the authors plan integration of more diverse and multilingual marine datasets via open API frameworks.

Industry relevance: The architecture is relevant to organizations building conversational interfaces over large scientific archives — particularly the pattern of using a lightweight, grammar-constrained agent for control decisions, a separate rewriting agent for retrieval quality, and a larger model only for the final grounded generation. The authors note that cloud offloading of inference could alleviate GPU contention between LLM inference and real-time rendering and support a wider range of devices.

Future Directions

• Improving RAG reliability: Extending the query-rewriting agent with query expansion to generate multiple semantically distinct rewrites per query, categorizing embedded documents by type (artistic media, scientific studies, activist narratives, data), adding an autonomous process to embed new documents as they become available, and exploring "Read the Doc before Rewriting" (R&R) approaches if current rewriting proves insufficient.

• Optimizing LLM agent performance: Offloading inference to the cloud to relieve GPU bottlenecks, and addressing open challenges in multi-agent LLM design such as clear role definitions, inter-agent alignment, and output verification.

• Artistic advancement: Sequences that open on a vast ocean horizon, draw participants in with layered marine soundscapes, and transition from macro to micro views such as an explorable coral reef. The authors also plan to replace pre-rendered video playback with real-time renders that allow user interaction with narratively-driven scenes.

• Expanding scientific and social impact: Controlled user studies on retention, empathy, and behavioral change; connecting emotional resonance with civic engagement; and integrating more diverse and multilingual marine datasets via open API frameworks.

Target Audience

Researchers and practitioners working at the intersection of conversational AI, RAG system design, and immersive media will find the most value here, especially those interested in architectural trade-offs between monolithic and multi-agent LLM pipelines. The paper also speaks directly to artists, curators, and museum technologists designing interactive climate exhibits, and to environmental communication researchers exploring how affective, dialogue-based interfaces differ from static data visualization. Educators considering installation-based science communication will find the hardware, latency, and deployment details practically useful.

Authors’ abstract

Sensorium Arc (AI reflects on climate) is a real-time multimodal interactive AI agent system that personifies the ocean as a poetic speaker and guides users through immersive explorations of complex marine data. Built on a modular multi-agent system and retrieval-augmented large language model (LLM) framework, Sensorium enables natural spoken conversations with AI agents that embodies the ocean's perspective, generating responses that blend scientific insight with ecological poetics. Through keyword detection and semantic parsing, the system dynamically triggers data visualizations and audiovisual playback based on time, location, and thematic cues drawn from the dialogue. Developed in collaboration with the Center for the Study of the Force Majeure and inspired by the eco-aesthetic philosophy of Newton Harrison, Sensorium Arc reimagines ocean data not as an abstract dataset but as a living narrative. The project demonstrates the potential of conversational AI agents to mediate affective, intuitive access to high-dimensional environmental data and proposes a new paradigm for human-machine-ecosystem.