MPJudge: Towards Perceptual Assessment of Music-Induced Paintings

AI summary

Overview

Research area: Computer vision / cross-modal (audio-visual) representation learning, specifically the perceptual assessment of paintings created under musical influence.

Technical level: Advanced. The paper combines transformer-based vision encoders, a modulation-based cross-modal fusion mechanism (adapted from normalization techniques such as AdaIN), and Direct Preference Optimization, so it assumes familiarity with modern multimodal architectures and preference-based training.

Scope: The paper introduces a new assessment task, a human-annotated dataset (MPD) of music–painting pairs with scalar coherence scores and pairwise preferences, and a model (MPJudge) that injects music features into a painting encoder to predict perceptual coherence.

What This Paper Is About

The paper asks whether a painting perceptually "fits" the music that supposedly inspired it, and treats this as a learnable prediction problem rather than an emotion-matching problem. Existing work matches music and paintings indirectly, typically by running both through emotion recognition models and checking whether the predicted emotions agree. The authors argue that emotion models are noisy and ignore perceptual cues such as rhythm, timbre, texture, and composition, so they build a dataset of human perceptual judgments and a model that fuses music and painting features directly.

Key Contributions

1. A new task and dataset. The authors define the task of music-induced painting assessment and construct MPD, described as the first large-scale dataset with human perceptual annotations for this task: approximately 6,000 pieces of music and 11,000 paintings, yielding over 50,000 music–painting pairs, each with a scalar coherence score from domain experts.
2. Pairwise preference annotations. Because many scores cluster near 0.5 (ambiguous pairs), the authors collect pairwise preference judgments within the score range [0.4, 0.6], producing 10,428 preference tasks that pass quality control as 5,582 music-to-painting and 5,403 painting-to-music preference samples.
3. MPJudge, a music-conditioned visual encoder. The model uses an asymmetric dual-branch design in which music features modulate the painting encoder through Modality-Adaptive Normalization (MAN), and it is trained with a regression loss plus a Direct Preference Optimization (DPO) loss — described as the first use of DPO in cross-modal painting assessment.
4. Evaluation, ablations, and interpretability analysis. The paper reports comparisons against three baselines across three datasets, an ablation of the DPO loss and three fusion strategies, a user study with 20 participants, and layer-wise Modulation Intensity Maps (MIMs) showing where music influences visual encoding.

Main Findings

• MPJudge outperforms all baselines across datasets and metrics. Compared with L3-Net, ACP-Net, and CDCML, the model is best on IMAC dataset accuracy (0.75 versus 0.66 for CDCML, 0.62 for ACP-Net, and 0.57 for L3-Net), and the authors state it achieves an SRCC of 0.86 and an MAE of 0.04 on their own dataset.
• Human annotations are reliable. Across all samples the average standard deviation of annotator scores is 0.078, with 84.8% of samples below σ < 0.09 and 99.0% below 0.11; Krippendorff's Alpha is 0.86.
• Labels appear cleaner than emotion-derived labels. The authors observe that all methods perform better on their dataset, which they attribute to manual annotation avoiding noise from sentiment recognition models.
• DPO helps with ambiguous cases. Removing the DPO loss drops the ablation results from SRCC 0.68, PLCC 0.66, MAE 0.04, ACC 0.93 to SRCC 0.63, PLCC 0.62, MAE 0.08, ACC 0.89; without DPO the model struggles with samples whose scores cluster around 0.5.
• MAN beats other fusion strategies. A painting-only baseline reaches SRCC 0.34, PLCC 0.31, MAE 0.31, ACC 0.64; simple concatenation reaches SRCC 0.55, PLCC 0.54, MAE 0.12, ACC 0.83; cross-attention reaches SRCC 0.61, PLCC 0.60, MAE 0.09, ACC 0.90; MAN (the proposed method) reaches SRCC 0.68, PLCC 0.66, MAE 0.04, ACC 0.93 while being described as simpler and more efficient than cross-attention.
• Model decisions align with human judgment. In a user study with 20 participants (7 female, 13 male, aged 20–30), the model matched human judgments in a binary matching task (10 matched and 10 mismatched pairs per participant) and in a ranking task (ranking 5 paintings for a given music, across 5 music pieces), with error bars described as relatively small.
• Music modulation is hierarchical and interpretable. Visualizing MIMs from the first three and last three Transformer blocks shows early layers modulating low-level regions such as textures and localized color patterns, while deeper layers show more global, semantic-level modulation; for example, ground-truth scores of 0.2, 0.9, and 0.1 correspond to predicted scores of 0.17, 0.94, and 0.14.
• Computational cost. In the reported parameter and FLOP comparison, MPJudge's music encoder has 3.02 M parameters and 3.02 G FLOPs, while its painting encoder has 44.65 M parameters and 21.16 G FLOPs — larger than the painting encoders of L3-Net (4.83 M, 2.41 G), ACP-Net (7.37 M, 14.68 M), and CDCML (25.56 M, 4.09 G).

Methodology in Plain English

Building the data. The authors could not find enough genuine music-induced paintings, so they took paintings and music separately and paired them at random for experts to judge. Paintings come from WikiArt (9,885 works across Art Nouveau Modern, Abstract Expressionism, Baroque, Analytical Cubism, and Action Painting, plus about 1,200 children's paintings from the AACP dataset). Music comes from about 1,000 full-length DEAM tracks cut into non-overlapping 15-second clips, giving 6,127 segments, each converted to a Mel spectrogram (16,000 Hz sample rate, FFT size 1,024, 128 Mel bins, hop length 512).

Annotating coherence. Thirty-five domain experts — instructors or graduate students from art academies — were briefed to judge "perceptual consistency," defined as how well the experience evoked by the music matches that evoked by the painting. They were told this goes beyond simple emotion matching to include associations such as rhythm with brushstroke dynamics, timbre with color palette, and musical tension with visual composition. Each pair received five independent scores; the highest and lowest were discarded and the remaining three averaged.

Handling ambiguity with preferences. Since many scores sat near the middle of the scale, the authors asked annotators to pick, for a given painting or music clip, which of two candidates fits better. Only pairs where the annotators reached majority agreement were kept.

Model design. Music is processed by a lightweight convolutional encoder (four blocks of convolution, batch normalization, and SiLU), since spectrograms have simple, localized structure. The painting goes through a 12-block Transformer encoder with 8 attention heads, an embedding dimension of 512, and 16×16 patches on 256×256×3 images. Rather than concatenating the two modalities at the end, the music features generate scale and shift parameters that rescale the painting's internal features after every self-attention block (Modality-Adaptive Normalization). The authors note that MAN looks mathematically similar to AdaIN but serves a different purpose: transferring cross-modal conditioning rather than visual style.

Training. The fused representation predicts a coherence score trained with mean squared error against human ratings, plus a DPO objective that pushes the model to score preferred candidates above less preferred ones relative to a fixed reference model. The two losses are weighted by λ_reg = 1 and λ_DPO = 0.5; the regression loss applies to all scored data and DPO applies to the preference-labeled data. Training used eight NVIDIA H100 GPUs with PyTorch, the Adam optimizer, a learning rate of 1×10⁻⁵, weight decay of 0.05, and batch size 1024.

Evaluation. Baselines are L3-Net, ACP-Net, and CDCML. Datasets are IMAC (about 85,000 images and 3,812 songs with positive, neutral, and negative labels), IMEMNet (25,620 images and 1,802 pieces of music, giving 144,435 pairs with continuous scores), and the new MPD. IMAC is evaluated with precision, recall, and accuracy; IMEMNet and MPD with SRCC, PLCC, MAE, and accuracy at a fixed threshold.

Why This Matters

Research impact. The paper shifts music–painting matching away from proxy emotion labels toward direct human perceptual supervision, and it introduces preference-based optimization (DPO) into cross-modal aesthetic assessment. It also supplies a large annotated dataset and an interpretability tool (Modulation Intensity Maps) for studying how audio conditioning shapes visual representations layer by layer.

Potential applications (not claimed as tested in the paper):

• Curatorial and museum tools that suggest music pairings for an exhibition or gallery space.
• Music streaming or visual-album experiences that recommend artwork to accompany a track.
• Art and music education, giving students feedback on whether a painting communicates the music it responds to.
• Creative tools for synesthesia-inspired work, where artists and composers explore cross-sensory correspondences.

Industry relevance. The work connects to recommendation and content pairing in music and media platforms, to automated aesthetic evaluation pipelines used by art marketplaces or galleries, and to generative systems where a model must judge whether an output actually matches a conditioning input — a problem shared with text-to-image and audio-to-image generation.

Future Directions

• Extending assessment to broader visual content, such as sketches or abstract art, which the authors explicitly name as future work.
• Applying the model in generative or interactive settings, also named by the authors.
• Testing whether the same modulation-based fusion and DPO training transfer to other cross-modal pairs (for example, video with music) and to paintings genuinely created under musical influence rather than randomly paired examples.
• Investigating the ambiguity boundary more deeply — the paper notes that scores cluster near 0.5, and it is not reported how much of the remaining uncertainty is intrinsic to the artwork pair versus a limitation of the annotation protocol.

Target Audience

Researchers in computer vision and multimodal learning working on cross-modal alignment and aesthetic or perceptual assessment; affective computing researchers interested in moving beyond emotion recognition models; dataset builders and annotation-methodology specialists; and practitioners in music technology, cultural heritage, or creative AI who need a way to score how well visual and auditory content fit together. The paper assumes comfort with transformer architectures, normalization-based conditioning, and preference optimization, so it is best suited to readers with an intermediate-to-advanced machine learning background.

Authors’ abstract

Music induced painting is a unique artistic practice, where visual artworks are created under the influence of music. Evaluating whether a painting faithfully reflects the music that inspired it poses a challenging perceptual assessment task. Existing methods primarily rely on emotion recognition models to assess the similarity between music and painting, but such models introduce considerable noise and overlook broader perceptual cues beyond emotion. To address these limitations, we propose a novel framework for music induced painting assessment that directly models perceptual coherence between music and visual art. We introduce MPD, the first large scale dataset of music painting pairs annotated by domain experts based on perceptual coherence. To better handle ambiguous cases, we further collect pairwise preference annotations. Building on this dataset, we present MPJudge, a model that integrates music features into a visual encoder via a modulation based fusion mechanism. To effectively learn from ambiguous cases, we adopt Direct Preference Optimization for training. Extensive experiments demonstrate that our method outperforms existing approaches. Qualitative results further show that our model more accurately identifies music relevant regions in paintings.