AI-Experiment: an augmented reality art project co-created with artificial intelligence

AI summary

Overview

Research area: Computer vision and generative AI applied to art practice — specifically latent diffusion model fine-tuning for artistic style transfer, plus generative video and browser-based augmented reality.

Technical level: Intermediate. The paper is written accessibly and reports no quantitative benchmarks, but it assumes some familiarity with diffusion models, fine-tuning, and prompting.

Scope: A practice-based, observational account of how one artist, one computer scientist, and several generative models co-created five physical paintings on canvas, extended by an augmented reality layer of generative videos.

What This Paper Is About

The authors ask whether modern latent diffusion models can generate abstract artwork that is actually interesting to humans, and they wanted to study this in a controlled way rather than by prompting models on styles already present in their training data. Their solution was to fine-tune a diffusion model on roughly 500 paintings by one living artist — co-author Hans Furer — generate synthetic works in his style, and then have that same artist paint selected generations back onto physical canvas. A second goal was to "bring the artwork to life" through an augmented reality web app that overlays AI-generated animations onto the finished paintings.

Key Contributions

1. A closed-loop co-creation pipeline. The model is trained on one specific artist's work, and the generated output is returned to canvas by the same artist who produced the training data — a loop the authors state they could not find in prior work.
2. Five physical artwork pairs. For each of five decades of the artist's output, one AI-generated piece was selected and painted in acrylic on a 100 x 100 cm canvas, paired with a training-set painting chosen by the artist, yielding five pairs of physical works.
3. An open release. Part of the training data (the 1981–1990 decade), the generation code, and the augmented reality experience are released at https://github.com/BernhardEgger/ARTimate under a permissive license for art, research, and education.
4. A browser-based AR art experience built with HTML5, JavaScript, and the MindAR library, which recognizes the paintings and plays one of several AI-generated videos, randomly selected on each visit.

Main Findings

• Model choice depends on the decade. Ten model variants were trained (five decades x Stable Diffusion 1.5 and 2.1). For the 1981–1990 decade, rich in faces and bodies, the newer model produced much more detail and less blur; for the more abstract 1991–2000 decade, Stable Diffusion 2.1 struggled more, showing texture-like patterns and possible hallucination, while Stable Diffusion 1.5 produced much less hallucinated content.
• The artist's "10 cm white border" objection was wrong. The artist said the model did not "understand" his work because a generated piece ran into the border he usually leaves blank — but inspection of the training data showed the model had learned this from the data rather than inventing it.
• Hallucination became creative input. The model produced crowded backgrounds where the artist typically painted uniform backgrounds. The artist found this interesting, and his subsequent work began experimenting with wild backgrounds.
• Video models only recently became usable. In 2023, tested video models produced simple translation or zooming, or inserted unrelated objects such as humans walking in front of the canvas. Animations of the abstract components only started working in summer 2025.
• Prompting dominated video results. The finally selected videos were generated at a mini-hackathon with eight master's and doctoral students engineering prompts; prompts were rewritten by a language model but the initial prompt still heavily steered the animation.
• Painted versions looked more vivid. The authors note the painted artwork appears more vivid than the generated source, while also reproducing model mistakes in text, signature, or dates.
• No quantitative evaluation was performed. The experiments are explicitly observational; the authors report no metrics for style, image quality, or video quality.

Methodology in Plain English

The team first considered the ArtBench dataset (10 artistic styles, 6'000 images per style) but avoided it because its copyright status was unclear. Instead they used a smaller set of roughly 500 photographs of Furer's paintings, taken by photographer Tobias R. Dürring for a 2013 catalogue raisonné. The work spans five decades (1971–1980, 1981–1990, 1991–2000, 2001–2010, 2011–2023), with 131 to 201 paintings per decade — roughly one per month — and a consistent 100 x 100 cm acrylic-on-canvas format since 1980.

They fine-tuned Stable Diffusion using DreamBooth, running a Jupyter notebook implementation on Google Colab. They made two changes: they removed prior preservation, which otherwise pushed outputs back toward the base model's style at the cost of losing text-prompt responsiveness (they treat this loss as a feature, since it removed human prompting bias), and they raised the maximum number of training steps from 800 to 10'000 for stronger style adoption. These settings were tuned by eye on a small set of generated images, not by systematic evaluation. For each of the ten model variants, 100 images were generated. The two authors then picked one generated piece per decade; Furer printed it on A4 paper, sketched proportions in pencil, and recreated it at 100 x 100 cm in acrylic, retaining freedom over exact colors and their application.

For the AR layer, commercially available and open-weight video models (Kling 2.1, Veo 3, Wan 2.2, Hailuo 2) were given either the painting as a starting frame, or the generated painting as a starting frame with the paired training painting as the end frame, plus a text prompt. The web app code was written by Gemini 2.5, and the hardest engineering problem was getting modern mobile browsers to autoplay video.

Why This Matters

Impact on research: The work demonstrates a reproducible protocol for studying style-level generation in a controlled setting — a single artist, a known five-decade corpus, and an artist in the loop — and it releases data and code so that newer models and fine-tuning techniques can be compared on the same footing. It also flags an open methodological gap: the authors note challenges in choosing or designing metrics for evaluating generated artwork in terms of style and content.

Real-world applications:

• Museum and gallery experiences that animate or extend static works, motivated by the authors' reference to The Borderline murals at MIT.
• Art education and workshops combining art and computer science, which the authors name as their intended future direction with teachers and museums.
• Artist tooling: fine-tuning on a personal archive to generate style-consistent studies or variants for the artist to react to.
• Browser-based AR delivery that avoids platform-specific apps, motivated by the observation that some existing apps (Artivive, Adobe Aero) were discontinued or not available on all mobile platforms.

Industry relevance: The paper is a practical data point on model selection (Stable Diffusion 1.5 vs 2.1 depending on the visual regime), on the trade-off between prior preservation and style fidelity, and on the state of generative video for non-photorealistic input. The authors note that all the methods used had successors within the three years since the project was performed, so results would likely differ today.

Future Directions

• Systematic evaluation. The released training data allows structured comparison of newer models and modern fine-tuning techniques; the authors explicitly leave this undone and call out the difficulty of defining style and content metrics.
• Newer video models. The successful animations came only in summer 2025 after repeated retesting of each new release, so continued tracking of video models against abstract input is an obvious thread.
• Education and museum partnerships. The stated next step is collaborating with teachers and museums to integrate the AR pipeline into teaching and workshops spanning art and computer science.
• Legal and conceptual questions. Intellectual property and copyright of AI-generated artwork remain open; the authors address these in a separate book rather than in this paper.

Target Audience

Most useful to artists working with generative models, computer vision researchers interested in style and creativity evaluation, HCI and digital-humanities scholars studying human–AI co-creation, and museum or art-education professionals considering AR extensions. Readers wanting quantitative benchmarks will not find them here; readers wanting a concrete, reusable, open-sourced workflow and an honest account of what failed and when it started working will.

Authors’ abstract

We describe the creation of a physical artwork that was performed in synergy between an artist, a computer scientist and generative AI tools. A latent diffusion model was finetuned on photographs of the artwork of the specific artist and the generative model then generated synthetic artworks in a similar style. From the generated artworks a subset was selected to be painted again with acrylic on canvas by the original artist. In addition to the physical artifact an augmented reality component was added which brings the artwork to life with a set of generative video models. Together with this paper we release part of the artwork (training data), the code that was deployed to generate the artwork as well as the augmented reality experience for arts, research and education under https://github.com/BernhardEgger/ARTimate