Learning visual representations for compositional analysis of artworks and photographs

AI summary

Overview

Research area: Computer vision and computational aesthetics, specifically compositional representation learning for photographs and artworks, combining object-centric learning (OCL), slot attention, and graph attention networks (GAT) against fine-tuned self-supervised foundation models (DINOv2).

Technical level: Advanced. The paper assumes familiarity with slot attention, graph attention networks, self-supervised vision transformers, and standard evaluation protocols such as PLCC, SRCC, mAP, and Precision@K.

Scope: A single study comparing a human-inspired, interpretable pipeline (object-centric region decomposition plus graph attention) with fine-tuned foundation models across composition category and score prediction, compositional retrieval, saliency detection, and attention faithfulness.

What This Paper Is About

Composition, the deliberate arrangement of visual elements, is central to how artworks convey meaning and aesthetic quality, yet it remains one of the least formalized dimensions of visual understanding. Prior work reports that existing models learn composition embeddings contaminated by semantic content and that annotated data is scarce, especially for artwork. This paper asks whether visual information must be processed in a human-like, region-based manner to learn composition, or whether a well-defined composition objective combined with sufficient training data is enough to unlock existing large-scale models.

Key Contributions

1. A human-inspired compositional pipeline. An object-centric model (FT-dinosaur with seven slots and a fixed/learned slot initialization) decomposes an image into region-level slot representations, and a graph attention network models relationships between those regions. The final representation concatenates slot features with the GAT output.

2. A direct head-to-head comparison of two paradigms. The structured pipeline is compared against frozen and fine-tuned DINOv2-small and DINOv2-base on the same tasks and datasets, isolating the effect of inductive bias versus scale and fine-tuning.

3. Evaluation beyond classification. The learned representations are tested on composition score/category prediction, compositional image retrieval, two semantic-bias probes (Composition Feature Distinction and Robustness to Semantic Interference), visual saliency detection, and an attention-faithfulness deletion-curve analysis.

4. A dataset split and an artwork fine-tuning study. Because PICD has no official split, the authors construct a stratified train/test split, and they fine-tune the OCL model on the BAID artwork dataset, evaluating the resulting segmentation on the DRAM artwork segmentation dataset.

Main Findings

• Frozen human-inspired features beat frozen DINOv2 on PICD. On the PICD test set with all encoders frozen, OCL + GAT reaches 84.14% precision, 76.89% recall, 80.35% F1, and 98.41% accuracy, versus 81.56/64.84/72.25/97.90 for frozen DINOv2-b and 82.41/61.08/70.16/97.81 for frozen DINOv2-s.

• Adding frozen DINOv2 to the structured pipeline helps on PICD. OCL + GAT + DINOv2-b (frozen) improves to 86.84% precision, 78.03% recall, 82.20% F1, and 98.57% accuracy, which the authors attribute to composition-relevant factors such as color and contrast that are not reducible to region relationships.

• Fine-tuned foundation models win on category prediction. Fine-tuned DINOv2-b achieves 88.66% precision, 88.21% recall, 88.44% F1, and 99.03% accuracy, and fine-tuned DINOv2-s achieves 86.34/86.31/86.33/98.85.

• Far fewer trainable parameters for the structured model. OCL + GAT uses 990K trainable parameters, compared with approximately 22M for DINOv2-s and 86M for DINOv2-b.

• On APDDv2 scores, the structured model does not lead. OCL + GAT reaches PLCC 54.45 and SRCC 53.27; fine-tuning OCL on BAID (OCL-BAID + GAT) raises this to 59.21 and 57.71, surpassing frozen DINOv2-s (56.25/55.40) but not frozen DINOv2-b (63.78/63.49). Fine-tuned DINOv2-b reaches 68.94/67.48 and fine-tuned DINOv2-s reaches 64.50/62.69, with the highest accuracy of 85.08 for fine-tuned DINOv2-s.

• The learned graph is uninformative on APDDv2. Analysis of the GAT graph on APDDv2 shows near-uniform attention across edges. The authors suggest this is because composition scores are global, whereas discrete composition categories encourage attention to objects and their spatial relationships.

• Semantic-bias probes favor the structured model when frozen. On the Composition Feature Distinction and Semantic Interference tasks, OCL + GAT scores 86.9 and 70.0, versus 68.8/28.2 for frozen DINOv2-b and 67.5/31.3 for frozen DINOv2-s. Fine-tuned DINOv2 reverses this, reaching 94.3/94.7 (DINOv2-b) and 93.7/91.0 (DINOv2-s), which the authors note contradicts prior characterizations of such models as inherently semantically biased.

• Retrieval results mirror the same pattern. Using the full PICD test set of 9,600 images, OCL + GAT achieves mAP 32.08, Precision@1 69.64, Precision@5 65.06, and Precision@10 62.44, while frozen DINOv2-b gets 18.60, 55.14, 49.26, and 45.99. Fine-tuned DINOv2-b reaches mAP 82.18 and 86.00 on all Precision@K values; fine-tuned DINOv2-s reaches 79.20 and 83.73/83.77/83.84. Retrieval is strongly driven by the predicted composition class.

• Saliency can be read off the graph. Region importance is computed as the sum of outgoing edge weights, with top-k selection or a dynamic gap threshold of 0.2 to decide how many salient regions to keep. Compared against eye-tracking data, the model consistently identifies the regions humans fixate on most and reproduces the ordering of visual attention, though OCL segments at a coarse level (whole objects rather than parts such as a face).

• GAT attention is faithful. In a deletion-curve analysis on the full PICD test set, the area between the high-attention and random curves is 0.1449, while the area between the random and low-attention curves is 0.1040. Removing the highest-attention node produces a prediction-confidence drop approximately five times larger than random removal. Keeping only the top-3 slots yields an F1 of 73.83%, versus 36.87% for only the bottom-3 slots.

• Artwork fine-tuning gives a small segmentation gain and a larger downstream gain. On DRAM, FT-dinosaur improves from mBO 50.3, FG-ARI 21.1, mIoU 50.1 to 51.1, 22.6, 50.9 after fine-tuning on BAID.

• OCL was chosen over SAM deliberately. The authors argue SAM produces masks without associated region-level feature representations, only sparsely covers scenes in automatic mode, requires per-image parameter tuning, and yields a variable number of segments incompatible with a fixed-node graph.

Methodology in Plain English

The study sets up two competing ways to build a representation of image composition.

The first is human-inspired. An object-centric model called slot attention takes features from a pretrained backbone and clusters them, in a differentiable attention-based process conceptually similar to K-means, into a fixed number of groups called slots. Each slot corresponds to a region of the image. The authors use FT-dinosaur with seven slots. Because slot initialization by random sampling is inconsistent across runs, they switch to learned slot initializations so that slots are deterministic. Those slots become nodes in a graph attention network, where edges represent relationships between regions and the attention mechanism lets some regions matter more than others. The final representation concatenates the slot features with the GAT output. Crucially, the graph structure makes it possible to read off which regions matter for the prediction, and to convert that into a saliency estimate.

The second paradigm is to fine-tune a self-supervised foundation model (DINOv2-small and DINOv2-base) on composition labels, treating it as a black box.

Both are trained for 20 epochs with AdamW, cosine annealing, and a two-epoch linear warmup, with the minimum learning rate at one-tenth of the maximum. Frozen-encoder runs use a maximum learning rate of 1×10⁻³; DINOv2 fine-tuning uses 1×10⁻⁵. PICD multi-label classification uses Binary Cross-Entropy with Logits loss and APDDv2 score prediction uses Mean Squared Error. Experiments use batch size 32 on a single NVIDIA RTX A4500 GPU.

Evaluation spans two main datasets: PICD (49k photographs over 24 composition categories, with the authors' own stratified split reserving 20% of each category for testing) and APDDv2 (10k paintings with attribute-level scores, using the composition and layout attribute rated 0 to 10). OCL is additionally fine-tuned on BAID and its segmentation checked on DRAM. Downstream probes include 1,000 triplets for each of the two semantic-bias tasks, retrieval on 9,600 test images, a deletion-curve faithfulness analysis, and qualitative comparison against eye-tracking fixations.

Why This Matters

Impact on research. The paper reframes the debate about semantic bias in composition models. Rather than treating semantic bias as an inherent property of large models, it argues that prior weak results reflect training objectives rather than architectural limits: models trained for aesthetic assessment or image cropping were not trained for composition. It also supplies an interpretable, parameter-efficient baseline and an attention-faithfulness evaluation that is uncommon in computational aesthetics.

Real-world applications

• Aesthetic assessment and curation tools for photography platforms and galleries.
• Composition-aware image retrieval for stock libraries, archives, and cultural heritage collections.
• Generative art and image cropping or reframing tools that need an explicit notion of layout.
• Saliency estimation for display, advertising, and accessibility without collecting costly eye-tracking data.

Industry relevance. The structured pipeline delivers competitive composition category prediction with 990K trainable parameters instead of roughly 22M or 86M, and its saliency output comes free from the graph rather than from a dedicated model or eye-tracking annotations. For products that must run under compute constraints, require frozen feature extractors, or need to explain their predictions, this is a meaningful trade-off against raw accuracy.

Future Directions

• Richer artwork composition datasets. Most artwork composition datasets provide only a single global score, which cannot capture inter-region relationships; the few composition-specific ones are under 200 images. The authors propose compositional descriptions, noting that no dataset systematically captures compositional structure in artworks through dedicated descriptions, and that Wikipedia's artwork descriptions plus LLMs could generate such annotations automatically.

• Hierarchical relational modeling. The current model lacks hierarchical attention, so saliency is coarse, identifying a person but not the person's face. Adding hierarchy is proposed as a fix.

• Cross-domain transfer and label definitions. Photographic composition concepts are often too rigid for the more complex, dynamic structures in artwork, and categories such as symmetry are absent from PICD. The authors also flag that composition label definitions are critical because models learn those labels, and question whether semantic information should be fully removed or partially retained.

• Reconciling the two paradigms. An open question is whether the interpretability and cross-domain generalization of the structured approach can be combined with the raw performance of fine-tuned foundation models, and whether the structured approach can be improved for global composition score prediction, where overall scores depend on many factors beyond inter-region relationships.

Target Audience

Researchers and graduate students in computer vision, computational aesthetics, and cultural heritage computing; practitioners building aesthetic assessment, image retrieval, saliency, or generative art systems; and anyone interested in interpretable, parameter-efficient architectures as an alternative to fine-tuning large foundation models. Readers need a working understanding of attention mechanisms and representation learning to follow the quantitative comparisons.

Funding (as reported): European Union (ERC AdG, GRAPPA, 101053925, awarded to Johan Wagemans) and the Research Foundation-Flanders (FWO, 1159925N, awarded to Fatemeh Behrad). Code and pre-trained models are available on GitHub.

Authors’ abstract

Composition, the deliberate arrangement of visual elements, is central to how meaning, emotion, and aesthetic quality are conveyed in artwork, yet it remains among the least formalized dimensions of visual understanding. Prior work highlights a persistent gap in learning meaningful compositional representations, attributing it to semantic bias and suggesting that human-inspired approaches may be key. We compare two parallel paradigms for composition analysis: a human-inspired method grounded in perceptual grouping, and fine-tuned foundation models enabled by recent large-scale compositional datasets. The human-inspired approach uses object-centric models for region-level decomposition and a graph attention network to capture spatial relationships between elements. Both paradigms are evaluated on composition score/category prediction, compositional image retrieval, and visual saliency detection. With frozen encoders, the human-inspired method achieves competitive performance while remaining interpretable. When sufficient data enables fine-tuning, large self-supervised models outperform significantly, but at the cost of interpretability and cross-domain generalization.