Style or Signature? Artist-Disjoint Evaluation of Style Classification in Frozen Vision Embeddings

AI summary

Overview

Research area: Computer vision and computational art history, specifically the evaluation of style classification from frozen pretrained image embeddings.

Technical level: Intermediate. The paper uses nearest-neighbour classification and cosine similarity rather than any novel architecture, so the concepts are accessible, but it assumes familiarity with frozen feature extractors, CLIP-style image–text pretraining, and evaluation splits.

Scope: The paper re-evaluates painting-style classification from frozen vision embeddings under an artist-disjoint protocol on a balanced set of 320 paintings across four twentieth-century movements and four image encoders, showing that standard random-split accuracy partly measures recognition of individual artists rather than style.

What This Paper Is About

Frozen image embeddings such as CLIP are widely used to label paintings by art-historical movement, and they report high accuracy. The problem is that standard evaluations use random splits in which works by the same artist fall on both the reference and query sides, so a classifier can get the movement right simply by recognising the painter. The paper's goal is to measure whether style accuracy survives when no work is ever classified using other works by its own artist, and to explain where and why it does not.

Key Contributions

1. An artist-disjoint (leave-one-artist-out) evaluation protocol for painting-style classification in frozen vision embeddings, in which each of 32 artists is held out in turn and the remaining 310 works form the reference pool, so whole artists are removed from the neighbour pool. The authors state this control had not previously been applied to style classification in frozen vision embeddings.

2. Quantification of the gap between standard and artist-disjoint accuracy, showing it is sharply movement-dependent rather than uniform: accuracy falls from 0.869 to 0.766 overall, but Impressionism and Cubism barely move while Surrealism falls 20.0 points.

3. An artist-recognisability control that predicts the artist rather than the style with same-artist neighbours permitted, separating genuine shared visual form from artist memorisation, and further separating memorised artists from those the embedding encodes poorly.

4. Demonstration that the effect is encoder-general across four encoders differing in scale, architecture and training objective, including a vision-only self-supervised model, which rules out a text-side or language-based explanation.

Main Findings

• Standard evaluation overstates style accuracy. Full-pool 5-NN classification over all 320 works reaches 0.869 accuracy against 0.25 chance and majority-class rates, but under the artist-disjoint protocol it drops to 0.766, a gap of 0.103.

• The drop is sharply uneven across movements. Impressionism falls 3.8 points (0.938 to 0.900), Cubism 5.0 points (0.850 to 0.800), Abstract Expressionism 12.5 points (0.975 to 0.850), and Surrealism 20.0 points (0.713 to 0.513). Surrealism is the only movement for which the model misclassifies close to half of all held-out works.

• Robustness is not an artefact of a single artist. With eight artists per movement, a single held-out painter can move a per-style figure by as much as 0.125, so the paper reports the standard deviation across held-out artists. Impressionism has the smallest drop and smallest variance (0.132); Surrealism has the largest drop and high variance (0.226).

• Surrealism's collapse is concentrated in specific painters. Two automatist painters, Miró and Masson, fall to 0.1 and 0.2 when their own works are removed, while the other six Surrealists stay between 0.5 and 0.8. Gorky (0.2, Abstract Expressionism) is the only non-Surrealist among the lowest scorers.

• Artist identity is strongly encoded. 5-NN artist classification reaches 0.466 against a chance rate of 1/32 = 0.031, identifying the individual painter roughly fifteen times more often than chance.

• Recognisability and style robustness are not simply inversely related. Across the 32 artists the Pearson correlation is r = 0.06, and all four combinations of recognisable/unrecognisable and robust/collapsing occur.

• Three types of artist emerge. Memorised artists are recognisable but cannot be placed by style once held out: Gorky is recognised as himself at 0.90 but classified by style at 0.20, and Miró is recognised at 0.50 but classified by style at 0.10. Poorly encoded artists are weak on both measures: Masson is recognised at only 0.30 and classified by style at 0.20. The third type is the artist subsumed by a shared form, best seen at movement level.

• Movements pair recognisability and robustness in opposite ways. Cubism has the lowest mean artist-recognisability (0.238) yet a high artist-disjoint style accuracy of 0.800; Impressionism pairs 0.400 with 0.900. Surrealism is the mirror image, with higher recognisability (0.513) and the lowest robustness (0.513).

• Abstract Expressionism is an exception to both patterns. Its artists are the most individually recognisable of any movement (0.713) yet the movement stays fairly robust under hold-out (0.850), with Gorky's low style accuracy hidden at the movement level by his robust colleagues.

• The fragility is encoder-general. Surrealism is the lowest-accuracy movement under the artist-disjoint protocol for every encoder: 0.513 (CLIP ViT-B/32), 0.638 (CLIP ViT-L/14), 0.375 (DINOv2 ViT-B/14) and 0.500 (ResNet-50, ImageNet). Impressionism and Cubism remain robust in every row. Absolute accuracies are not comparable across encoders.

• The effect is visual, not linguistic. DINOv2 is trained by self-supervision on images alone with no text encoder, yet it reproduces the Surrealism collapse more sharply than any CLIP model, at 0.375.

• Errors concentrate in two destinations. Abstract Expressionism and Cubism each receive 34 misclassified works while emitting 12 and 16. Surrealism's 39 errors go 18 to Abstract Expressionism, 18 to Cubism and 3 to Impressionism.

• Silhouette scores are more cautious than accuracy. The overall mean silhouette is only weakly positive (0.066), and Surrealism is the sole movement with a negative mean (−0.039), meaning many Surrealist works lie closer to other movements than to their own.

Methodology in Plain English

Every painting is passed once through a frozen, pretrained image encoder with no fine-tuning, and the resulting feature vector is L2-normalised so that cosine similarity becomes an inner product. Unless stated otherwise, CLIP ViT-B/32 is the reference encoder, and the full pipeline is repeated with CLIP ViT-L/14, the self-supervised DINOv2 ViT-B/14 (CLS token) and a supervised ImageNet ResNet-50 (global-average-pooled). All nearest-neighbour computations use cosine distance.

The dataset is drawn from the Artificio/WikiArt release on Hugging Face, a static versioned snapshot, and is balanced by construction: four movements, eight artists per movement and ten works per artist, giving 320 images stored as 256 × 256 anisotropically resized thumbnails. The eight artists per movement are the ones most strongly associated with it in standard art-historical references, filtered to those with at least ten qualifying works; Cubism merges Cubism, Analytical Cubism and Synthetic Cubism, and Abstract Expressionism merges Abstract Expressionism, Action Painting and Color Field Painting.

As a baseline, the authors run 5-NN style classification over all 320 works, assigning each work the majority style among its five nearest neighbours excluding itself, with ties broken by the closest neighbour then alphabetically. Accuracy was stable across k in {3, 5, 7, 9}. They also report mean silhouette scores using style labels as cluster assignments.

The artist-disjoint protocol holds each of the 32 artists out in turn: that artist's ten works become queries and the remaining 310 works form the reference pool, so no work is ever classified using another by the same painter. The protocol is exhaustive and deterministic, and each movement yields eight accuracy figures, one per held-out artist, with the standard deviation across them reported. A separate control re-runs the same procedure but predicts the artist rather than the style, with same-artist neighbours permitted, measuring how recognisable each artist is individually.

A near-duplicate screen embedded all 320 images with CLIP ViT-B/32 and computed pairwise cosine similarity. No pair reached the 0.95 flagging threshold; the closest pair (two Miró works, cosine 0.949) and the next band (Gorky, Rothko and Pollock at 0.93 to 0.94) were inspected and confirmed distinct.

Why This Matters

Impact on research. A single aggregate style-classification accuracy can certify artist recognition as style understanding. The paper shows the overstatement is uneven across movements, reaching twenty points for Surrealism, so an aggregate figure hides where it falls. Artist-disjoint evaluation is inexpensive, needs no retraining, and converts a brittle aggregate into an interpretable per-movement and per-artist picture. The authors note the principle is not novel, since group-disjoint evaluation is standard in medical imaging and speaker-independent speech recognition, and that a recent survey observed it has been largely absent in style classification.

Real-world applications.

• Digital collection management and cataloguing, where movements and periods are assigned to works automatically and the label may need to generalise to artists not yet in the collection.
• Art retrieval and recommendation systems that group similar works, which may otherwise group by painter rather than by movement.
• Provenance and attribution support, where the paper's finding that Cubist artists are the hardest to tell apart and Impressionists cohere through shared form is directly relevant.
• Benchmarking and auditing of off-the-shelf vision models before they are deployed on cultural-heritage collections.

Industry relevance. Because frozen embeddings are used off the shelf rather than trained from scratch, the evaluation choice is the main lever available to practitioners. The paper argues artist-disjoint evaluation should be a default rather than a refinement when the question is genuinely about style, and the released manifest pins the exact 320 works by filename and SHA-256 hash at https://github.com/Rory-A/ml-art-representation.

Future Directions

1. Larger and less curated datasets spanning more movements and artists, since the present set is small (320 works, 32 artists) and a single held-out painter moves a per-style figure by as much as 0.125.

2. Classifiers beyond nearest neighbour, since the study deliberately uses fixed feature extractors with nearest-neighbour classification and does not fine-tune or probe intermediate layers.

3. Content-controlled subsets that match subject matter across movements, because the movements here are not matched for content and composition, leaving style entangled with what is depicted.

4. Probing the text encoders alongside the image ones, and asking whether fine-tuning on art-specific labels closes or merely relocates the artist dependence measured here.

Target Audience

Researchers and practitioners in computer vision and computational art history who use frozen pretrained embeddings for cultural-heritage tasks; benchmark designers and evaluation reviewers concerned with identity leakage in train/test splits; and digital collection or retrieval engineers who apply off-the-shelf image encoders to painting datasets.

Authors’ abstract

Frozen image embeddings from models such as CLIP are increasingly used to classify paintings by art-historical style, with high reported accuracy. We ask whether this accuracy reflects an understanding of style or the recognition of individual artists. Standard evaluation uses random splits in which works by the same artist appear on both sides, so a classifier can succeed by recognising the painter rather than the movement. We re-evaluate style classification under an artist-disjoint protocol, holding out every artist in turn so that no work is ever classified using other works by its own painter. On a balanced dataset of 320 paintings across four twentieth-century movements, 5-NN style accuracy falls from 0.87 to 0.77 under this protocol, and the drop is sharply uneven. Impressionism and Cubism barely move, while Surrealism falls twenty points. The pattern holds across four image encoders, including a vision-only self-supervised model, which places the effect in visual structure rather than language. Where an encoder captures genuine shared form, individual artists are barely recognisable yet style is robust, while Surrealism shows the opposite. We argue that artist-disjoint evaluation is necessary to measure stylistic understanding in frozen embeddings.