ShadAR: LLM-driven shader generation to transform visual perception in Augmented Reality

AI summary

Overview

Research area: Human-Computer Interaction, specifically Augmented Reality (AR) display control and LLM-driven code generation.

Technical level: Intermediate. The paper assumes familiarity with AR headsets, shader programming (HLSL), and large language models, but explains the system pipeline without deep mathematical or graphics-theory detail.

Scope: The paper presents ShadAR, a demo system deployed on Meta Quest 3 that converts spoken natural-language descriptions of a desired visual effect into HLSL shader code that is compiled and applied to the AR passthrough viewport in real time.

What This Paper Is About

AR headsets can alter how the wearer sees the physical world, but today each altered-perception effect (such as a colorblindness simulation) must be hand-built by a developer ahead of time, for one specific condition. ShadAR lets a user simply say what they want to see — for example, "I want to see the world with heat vision" — and an LLM generates the shader code that produces that effect on the headset's camera feed. The goal is to make AR a flexible "visual experience machine" rather than a fixed set of pre-authored visual filters.

Key Contributions

1. An LLM-driven shader generation pipeline for AR. The system combines voice transcription, a prompt-engineered LLM (OpenAI o3-mini), HLSL shader generation, and deployment to a headset, spanning intent understanding, compilation, and viewport transformation.

2. Real-time deployment of generated shaders to a commercial AR headset. Shader code is compiled on a desktop using Unity in headless batch mode, packaged into an asset bundle, and pushed to the Meta Quest 3 over a local web server, where it is applied to the passthrough video stream accessed through the Meta Passthrough Camera API as a WebCamTexture.

3. A working demo interface for verbal visual-perception authoring. Users speak a desired effect while holding the Oculus B button; after approximately 30 to 45 seconds a custom shader is applied, and users can save the shader for reuse or record a new command.

4. Demonstrated example effects across three purposes. Accessibility (simulating red–green colorblindness), creative expression (an underwater look), and target highlighting (grayscale while preserving green), plus a "heat vision" example prompt.

Main Findings

• Natural language can drive shader authoring for AR. The system successfully interprets spoken user intent and produces compilable HLSL shaders that are applied to the live passthrough feed, removing the need for developers to predefine each effect.

• Generation takes roughly 30 to 45 seconds. The paper reports this as the delay between the user's recorded voice command and the shader being applied to the AR viewport.

• The pipeline has three stages. Intent understanding and shader generation (Whisper transcription plus a prompt-engineered OpenAI o3-mini), real-time shader compilation and deployment (Unity headless batch mode, asset bundle, local web server), and viewport access and transformation (Meta Passthrough Camera API, WebCamTexture, Unity3D).

• Prompt design is used to improve reliability. The LLM prompt incorporates domain knowledge (such as animal vision effects), example shader templates, and common pitfalls in shader programming.

• Effects are not always perceptually accurate. The authors state explicitly that the generated effects, though not always perceptually accurate, allow for rapid exploration of altered visual perceptions.

• The view is monoscopic, not stereoscopic. The AR viewport is rendered on a 2D quad showing the left-eye camera feed. By aligning the quad at an appropriate distance and scale to the user's eye, the authors attempt to achieve a visually coherent and sufficiently immersive experience.

• No user study, benchmark, or accuracy evaluation is reported. The paper is a demo contribution: it describes the architecture and illustrates example outputs, including a video at https://youtu.be/XS90nrJnel4, rather than reporting measured performance or user results.

Methodology in Plain English

The researchers built an end-to-end pipeline rather than training a new model. A user wears a Meta Quest 3 and speaks the visual effect they want while holding the Oculus B button to record. The audio is transcribed with Whisper and sent over HTTP to a prompt-engineered OpenAI o3-mini model, whose prompt includes domain knowledge, example shader templates, and known shader-programming pitfalls. The model returns HLSL shader code. That code is compiled on a desktop machine using Unity in headless batch mode, packed into an asset bundle, and delivered to the Quest device through a local web server. On the headset, the AR camera passthrough is accessed via the Meta Passthrough Camera API as a WebCamTexture, and the new shader is applied to the material of the quad that renders that video feed, changing what the user sees. The authors then demonstrate the approach with a set of example prompts and resulting effects, evaluated by inspection and demonstration rather than by formal measurement.

Why This Matters

Impact on research. The work shifts XR code generation from creating and manipulating virtual objects (prior systems in this space generated CSharp code) toward generating HLSL shaders that change the AR viewport itself. It reframes AR as an open-ended perceptual tool whose effects are authored on demand by the user rather than pre-programmed by developers.

Real-world applications:

• Accessibility and inclusive design — simulating colorblindness or other visual impairments so designers can better understand the challenges users face.
• Creative expression — producing artistic effects such as an underwater view or stylized environments.
• Productivity and search — prompts like "I want to see things in grayscale except green" to make a target object, such as a green folder, stand out in a cluttered scene.
• Novel visual experiences — for example, requesting heat vision to view a scene in an unfamiliar way.

Industry relevance. The system runs on commodity hardware (Meta Quest 3) and uses off-the-shelf components (Whisper, OpenAI o3-mini, Unity), suggesting that LLM-generated shader pipelines could be integrated into consumer AR headsets and development toolchains rather than requiring bespoke per-effect engineering.

Future Directions

• Improving perceptual accuracy. The authors acknowledge generated effects are not always perceptually accurate; validating whether a shader genuinely reproduces a condition such as colorblindness is an open question.
• Reducing generation latency. The reported 30 to 45 second wait between command and applied shader is a clear target for interactive use.
• Moving from monoscopic to stereoscopic rendering. The current 2D quad showing the left-eye feed is an approximation; supporting true stereoscopic views would improve immersion.
• Formal evaluation. No user study, task performance measure, or benchmark is reported, leaving room for studies of usability, prompt success rate, and real-world design or navigation tasks.

Target Audience

Researchers and practitioners in augmented and extended reality, human-computer interaction, and generative-AI tooling, particularly those interested in LLM-driven code generation, shader programming, accessible design tools, and real-time rendering on head-mounted displays. It is also useful to designers and artists exploring novel visual experiences, though it assumes some background in AR development and graphics programming.

Authors’ abstract

Augmented Reality (AR) can simulate various visual perceptions, such as how individuals with colorblindness see the world. However, these simulations require developers to predefine each visual effect, limiting flexibility. We present ShadAR, an AR application enabling real-time transformation of visual perception through shader generation using large language models (LLMs). ShadAR allows users to express their visual intent via natural language, which is interpreted by an LLM to generate corresponding shader code. This shader is then compiled real-time to modify the AR headset viewport. We present our LLM-driven shader generation pipeline and demonstrate its ability to transform visual perception for inclusiveness and creativity.