Art2Mus: Artwork-to-Music Generation via Visual Conditioning and Large-Scale Cross-Modal Alignment

AI summary

Overview

Research area: Multimodal generative deep learning, specifically cross-modal conditioning for music generation — at the intersection of computer vision, audio synthesis, and cultural heritage computing.

Technical level: Advanced. The paper assumes familiarity with latent diffusion models, variational autoencoders, transformer-based denoisers, CLIP/ImageBind embeddings, and multimodal captioning architectures.

One-sentence scope: The paper introduces ArtSound, a dataset of 105,884 artwork–music pairs, and Art2Mus, a framework that conditions a latent diffusion music generator directly on visual artwork embeddings without any image-to-text translation step.

What This Paper Is About

Existing systems that generate music from images have two weaknesses: they are usually trained on natural photographs rather than artworks (whose meaning is stylistic, symbolic, and culturally layered), and most of them convert the image into a text caption first, letting language act as a "semantic shortcut" that discards non-verbalizable visual cues. This paper tackles both problems at once by building a large artwork–music dataset and a framework that maps visual embeddings straight into the conditioning space of a pretrained audio diffusion model — no text intermediary at any point.

Key Contributions

1. ArtSound dataset. A large-scale multimodal resource of 105,884 artwork–music pairs, built by combining ArtGraph (116,475 digitized artworks across 18 genres and 32 styles) with the Large split of the Free Music Archive (105,884 retained tracks after removing corrupted audio). Every pair is enriched with both an image caption and an audio caption.

2. Two new alignment metrics for caption quality control. ICScore, a weighted blend of CLIP-Score and PAC-Score (γ = 0.35) for image captions, and ACScore, a weighted blend of ROUGE-1 and BERT-Score (α = 0.30) for audio captions. Captions scoring below 0.80 are regenerated with feedback-refined prompts for up to 3 iterations.

3. Art2Mus, the first framework designed explicitly for direct artwork-to-music generation. A Visual Conditioning Extractor and Image Aligner project CLIP (512-dimensional) or ImageBind (1024-dimensional) embeddings into GPT-2's 768-dimensional token space, which serves as a modality-agnostic bridge into AudioLDM 2's Language of Audio (LoA) embedding space.

4. A systematic study of artwork–music pairing strategies. Four embedding-based pairing approaches were evaluated using ImageBind, exposing a substantial modality gap and justifying the choice of raw Image–Audio pairing despite its lower average similarity.

Main Findings

• Text is the strongest shared space, not raw media. Pairing image captions with audio captions produced the highest mean cosine similarity (0.7081, std dev 0.3613). Raw Image–Audio pairing averaged 0.1864 (std dev 0.0697), Image–Audio Caption averaged 0.0644, and Image Caption–Audio averaged 0.0658. The authors interpret this as evidence of a substantial modality gap between images and audio.

• Qualitative inspection overrode the quantitative ranking. Although Image–Audio did not have the top average similarity, the authors selected it to build ArtSound because the resulting pairs appeared more semantically aligned than those from the other strategies.

• The greedy pairing strategy degrades toward the end. Because items are matched from highest similarity downward, the final pairings necessarily involve low-similarity leftovers. Similarity values were retained in the dataset so downstream users can filter or analyze this effect.

• Image captions were far richer than audio captions. Average image caption length was 950.18 characters (min 234.00, max 2302.00), while average audio caption length was 411.76 characters (min 82.00, max 1084.00). The authors attribute part of this gap to the choice of captioning models rather than to the modalities alone.

• Caption quality was high but not uniform. For image captions, mean ICScore was 0.8217 (CLIP-Score 0.7821, PAC-Score 0.8431); 78,845 captions exceeded the 0.80 threshold and 37,630 fell below it. For audio captions, mean ACScore was 0.8593 (BERT-Score 0.9321, ROUGE-1 0.6894); 95,837 exceeded the threshold and 10,047 fell below it. Captions that failed after all regeneration attempts were retained in the dataset along with their scores.

• Qwen3-14B was the best caption-fusion LLM tested. It scored ROUGE-1 0.6870, BERT-Score 0.9312, and ACScore 0.8579, ahead of Qwen3-8B (0.6301 / 0.9260 / 0.8372) and LLaMa3.1-8B-Instruct (0.5373 / 0.9056 / 0.7951).

• Art2Mus produces coherent, stylistically consistent music that reflects salient visual cues of the source artworks. Absolute alignment scores remain lower than those of text-conditioned systems — described as expected given the removal of linguistic supervision — but the framework achieves competitive perceptual quality and meaningful cross-modal correspondence. Numerical generation-quality results are not reported in the portion of the paper available here.

• Several alternative components were tested and discarded. CLIP, CLAP, RoBERTa, and Modern-BERT embeddings produced frequently unrelated image–audio pairs. For image captioning, some models were rejected as outdated, others produced overly long/unbalanced or generic captions without prompt-conditioning support. For audio captioning, one model was excluded for high VRAM requirements and low-quality captions, while others produced overly detailed outputs that repeated concepts, asked the user questions, or fabricated content. Pairing artworks with audio spectrograms or audio-generated images also gave poor overall results.

• A 1024-dimensional staging space helped. Empirically, projecting embeddings through a 1024-dimensional intermediate before mapping to GPT-2's token space yielded more faithful and stylistically coherent music than projecting lower-dimensional embeddings directly.

Methodology in Plain English

Step 1 — Assemble the raw material. Take 116,475 digitized artworks from ArtGraph (built from WikiArt and DBpedia, with metadata on artist, style, and genre) and 106,574 Creative Commons 30-second tracks from the Free Music Archive Large split. Removing corrupted audio leaves 105,884 tracks.

Step 2 — Caption everything. Each artwork is described by LLaVA (LLaVA1.6-Mistral-7B) using a structured prompt covering content, mood, stylistic cues, and possible artistic influences. Each track is described by LP-MusicCaps in 10-second segments, then those segments are fused into one coherent description by Qwen3-14B.

Step 3 — Score and regenerate captions. Image captions get an ICScore; audio captions get an ACScore. Anything below 0.80 triggers a new prompt that explicitly tells the model what went wrong, and the model tries again — up to 3 times, since further attempts showed diminishing returns.

Step 4 — Pair artworks with music. All items are embedded with ImageBind into a shared space. For each audio track, the most similar still-unpaired image is taken, and that image is removed from the pool. This continues greedily until all 105,884 tracks are paired.

Step 5 — Build the generator. The architecture reuses AudioLDM 2's latent diffusion backbone as a frozen generative prior: a VAE encodes mel-spectrograms into latents, a T-UNet denoiser reverses a noise process, and a HiFiGAN vocoder converts the decoded spectrogram to waveform. Conditioning normally flows through GPT-2 into the LoA embedding space, which is built with AudioMAE. Art2Mus inserts a new pathway: a pretrained vision encoder (CLIP or ImageBind) produces an embedding, an upscaler expands 512-dimensional CLIP embeddings to 1024 dimensions, and a projection layer maps them to N tokens in GPT-2's 768-dimensional space. GPT-2 then translates these into LoA embeddings. At inference, the visual conditioning is combined with the textual prompt "Music representing the content of this artwork" encoded by FLAN-T5, and the final conditioning vector drives the frozen diffusion backbone.

Why This Matters

The paper reframes artwork-to-music generation as a distinct, harder research problem rather than an application of text-to-music. By publishing both the dataset and the framework, it gives the field a shared benchmark for direct visual conditioning and a quantified picture of how large the image–audio modality gap actually is. It also supplies concrete evidence that captioning quality differs sharply by modality, which affects any pipeline built on captions.

Real-world applications:

• Multimedia art installations and performances where a piece of visual art dynamically generates an accompanying soundtrack.
• Cultural heritage experiences, generating music for museum or gallery exhibits from the artworks themselves without curatorial text input.
• AI-assisted creative practice, giving artists and composers a tool that responds to visual style rather than to a written prompt.
• Accessibility and cross-modal translation, offering a visual-to-audio pathway for audiences who engage with art primarily through sound.

Industry relevance: The framework reuses a frozen pretrained text-to-audio backbone with minimal architectural change, which means the approach is compute-efficient to adopt and can inherit improvements in the underlying audio diffusion model. The dataset's Creative Commons music licensing and open-access art sources also make it relatively deployment-friendly for commercial creative tooling.

Future Directions

• Closing the modality gap. Raw Image–Audio pairing averaged only 0.1864 cosine similarity, and Art2Mus's alignment scores remain below text-conditioned systems. Better visual–audio alignment objectives are the clear next target.
• Improving the pairing strategy. The greedy matching process forces low-similarity pairings at the tail end; optimal assignment methods or learned pairing could raise overall pair quality.
• Narrowing the caption quality gap between modalities. Image captions averaged 950.18 characters versus 411.76 for audio captions, and 37,630 image captions still fell below the 0.80 ICScore threshold after regeneration — better captioners and more varied refinement strategies are open problems.
• Learning non-verbalizable cues. The paper's core argument is that language discards stylistic and atmospheric information. A natural follow-up is measuring empirically how much visual information is lost when captions are used as an intermediary, which would directly test that hypothesis.

Target Audience

This paper is most useful to researchers working on multimodal generative models, particularly those focused on cross-modal conditioning and audio synthesis. It is also relevant to computer-vision researchers applying generative methods to cultural heritage and art analysis, and to practitioners building music-generation tools who need a large, captioned artwork–music resource. Readers will need a working understanding of diffusion models, multimodal embedding spaces, and captioning evaluation metrics; newcomers to deep generative modeling would find the architecture section difficult without background reading.

Authors’ abstract

Music generation has advanced markedly through multimodal deep learning, enabling models to synthesize audio from text and, more recently, from images. However, existing image-conditioned systems suffer from two fundamental limitations: (i) they are typically trained on natural photographs, limiting their ability to capture the richer semantic, stylistic, and cultural content of artworks; and (ii) most rely on an image-to-text conversion stage, using language as a semantic shortcut that simplifies conditioning but prevents direct visual-to-audio learning. Motivated by these gaps, we introduce ArtSound, a large-scale multimodal dataset of 105,884 artwork-music pairs enriched with dual-modality captions, obtained by extending ArtGraph and the Free Music Archive. We further propose ArtToMus, the first framework explicitly designed for direct artwork-to-music generation, which maps digitized artworks to music without image-to-text translation or language-based semantic supervision. The framework projects visual embeddings into the conditioning space of a latent diffusion model, enabling music synthesis guided solely by visual information. Experimental results show that ArtToMus generates musically coherent and stylistically consistent outputs that reflect salient visual cues of the source artworks. While absolute alignment scores remain lower than those of text-conditioned systems-as expected given the substantially increased difficulty of removing linguistic supervision-ArtToMus achieves competitive perceptual quality and meaningful cross-modal correspondence. This work establishes direct visual-to-music generation as a distinct and challenging research direction, and provides resources that support applications in multimedia art, cultural heritage, and AI-assisted creative practice. Code and dataset will be publicly released upon acceptance.