Visual Interestingness Decoded: How GPT-4o Mirrors Human Interests

AI summary

Overview

Research area: Computer vision and multimodal foundation models, specifically the subjective concept of visual interestingness and how well Large Multimodal Models (LMMs) capture it.

Technical level: Intermediate. The core ideas are accessible, but the paper uses ranking metrics (agreement rates, Spearman rank correlation), prompt-based annotation protocols, and a learning-to-rank distillation setup.

Scope (one sentence): The paper compares human and LMM judgments of image interestingness in both single-image and pairwise settings, finds that GPT-4o outperforms prior computational models at predicting human interest, and distills that knowledge into a lightweight ranking model.

What This Paper Is About

Deciding whether an image is "interesting" is a highly subjective judgment that depends on context, observer, and background, yet attracting attention matters enormously in online media. The authors ask whether the implicit knowledge encoded in Large Multimodal Models, especially GPT-4o, can approximate human judgments of visual interestingness well enough to be useful. They do this by collecting verified human annotations alongside model annotations for the same images and comparing how closely the two align.

Key Contributions

1. A new annotated dataset: 1,000 images (2,500 image pairs) with metadata and multiple verified labels from human annotators (via Amazon Mechanical Turk) and from three state-of-the-art LMMs: GPT-4o ('gpt-4o-2024-11-20'), Llama 3.2 ('Llama-3.2-11B-Vision-Instruct'), and DeepSeek's Vision-Language VL2 ('DeepSeek-VL2-tiny').
2. A new approach to estimating image interestingness without direct user input, which the authors report outperforms state-of-the-art models for everyday images.
3. A comparative analysis of human and LMM annotations, identifying where they agree and diverge, using both the "why" explanations and image appearance descriptions to characterize what is considered interesting.
4. A distillation pipeline: pairwise labels from GPT-4o are used as training data for a simple learning-to-rank model, showing knowledge can be transferred to a small computational model.

Main Findings

• Single-image questions are nearly useless. Asking "Is this image interesting?" produced almost uniformly positive answers. Human annotations were 91.9% consistent and 99.9% of those consistent images were labeled interesting. GPT-4o was 93.9% consistent with 95.3% positive; Llama was 93.1% consistent with 99.8% positive; DeepSeek was 76.2% consistent with 81.4% positive. Human–GPT-4o agreement was 92.9% overall and 93.6% on the human consensus set.
• Pairwise comparison is where the signal is. Human annotators reached consensus on 56.3% of image pairs, while GPT-4o was consistent on 95.5% of pairs. Overall human–GPT-4o agreement was 66.2%, rising to 73.8% when humans agreed and dropping to 56.5% on the human dissent set.
• GPT-4o has a systematic position bias. When image pairs were shown twice with the images swapped, GPT-4o reported the second image as more interesting in 36% of cases; only 64% (1,599 out of 2,500) of annotations stayed the same regardless of order. This error was not strongly correlated with human consensus (56.3% of pairs in the human consensus set and 47.6% in the dissent set were error-free).
• Demographic prompting had no effect. Using a system prompt assigning gender (male or female), continent (North America or Africa), and age range (25 to 34 or 45 to 54) across all eight combinations on 500 random image pairs, after filtering out pairs with systematic error, the remaining 116 image pairs produced identical results. The authors note a difference only appeared for highly stereotyped images such as a blue car or a pink flower.
• Other LMMs failed the pairwise task. Llama 3.2 and DeepSeek models either do not support multiple image inputs or, when images were combined into a single input, selected one image while explaining the other. Because of this, further pairwise analysis was limited to GPT-4o.
• GPT-4o beats prior models at predicting human interest. On agreement with human annotations the ordering was GPT-4o 73.8%, Commonly Interesting Images (CI) 69.6%, Aesthetics 68.3%, #Comments 68.0%, #Favorites 66.4%, #Views 61.7%, CuPL 60.3%, and Memorability 35.5%. Against GPT-4o annotations as ground truth, the Aesthetic model reached 75.1%, #Comments 74.8%, #Favorites 74.0%, CI 67.6%, #Views 63.9%, CuPL 60.9%, and Memorability 39.1%.
• Distillation works. A learning-to-rank model trained on GPT-4o labels reached 84.8 ± 2.5% accuracy on GPT-4o-labeled pairs, while a model trained on human labels reached 77.5 ± 2.5% accuracy on human-labeled pairs. Cross-source models scored lower (GPT-4o-trained model on human data: 73.4 ± 3.4%; human-trained model on GPT-4o data: 72.0 ± 3.4%). The correlation between human and GPT-4o rankings was 0.59 ± 0.06.
• Ranking generalizes beyond pairs. Spearman rank correlations on the global image ranking were consistent with the pairwise accuracies, and memorability was the only concept negatively correlated with human judgments (-0.42 ± 0.08 for the model trained on humans).
• Domain transfer to video-summarization data drops sharply. On the Int10k dataset, the human-annotation model reached about 59.3 ± 2.5% accuracy and GPT-4o reached a comparable 59.2 ± 2.1%.
• Shared interests and gaps. Hierarchical clustering of GPT-4o explanations surfaced shared clusters for "cute," "emotional," and "uniqueness," while clusters around vibrant color and dynamic action were considered interesting only by GPT-4o. Human explanations were predominantly short and repetitive words such as "nice," "beautiful," or "good," so the qualitative analysis focused on GPT-4o text embeddings.
• Content-level agreement patterns. Examples from the appearance clusters: "perched, bird, branch, flower" showed 82% agreement with 97% positive judgments and mean ranks of 53 (human) and 80 (GPT-4o); "train, track, station, railway" showed 94% agreement with 60% positive and ranks of 143 and 184; "people, group, front, standing" showed 86% agreement with only 23% positive and ranks of 246 and 236; "person, sitting, room, window" showed 81% agreement with 24% positive and ranks of 217 and 207.

Methodology in Plain English

The authors started from 1,000 everyday images sampled from the Flickr-User dataset, equally sampled according to a previously proposed commonly interestingness score, and designed two user studies on Amazon Mechanical Turk.

In the first study, a random image was shown with the question "Is this image interesting to you?" plus a request for a short explanation. Five Human Intelligence Tasks were created per image, workers were paid $0.01 per task, and 258 unique workers participated, never seeing the same image twice. The same prompt was given to each LMM five times per image.

Because nearly everything was labeled interesting, the authors moved to a second, relative study: 2,500 image pairs (each of the 1,000 images used in five random pairs), asking workers "Which image is more interesting to you?" Again five HITs per pair and $0.01 per task, this time with 553 unique workers. Each image was compared in deliberately swapped order to detect model position bias, and only order-stable pairs were retained.

They then compared GPT-4o against alternative signals: an aesthetics model trained on user comments (VILA), a memorability model, a commonly-interesting-images model trained from Flickr favorites, social metrics (number of views, favorites, comments), and a zero-shot approach (CuPL) that generates 250 unique prompts describing an interesting image and scores images by CLIP cosine similarity.

Finally, they distilled the pairwise labels into a Siamese network with shared weights: each image is embedded with CLIP, passed through a single linear neuron, and the score difference between two images is squashed through a sigmoid with binary cross-entropy loss. After training, a single image can be scored with the same neuron, allowing global rankings. Models were trained for 25 epochs, no overfitting was observed, and each experiment was repeated 50 times with different train/test splits. They also report experiments using DINOv2 embeddings (67.1% for humans and 68.3% for GPT-4o).

To interpret the results, they embedded the "why" explanations using OpenAI's 'text-embedding-3-small' (1536-dimensional vectors) and performed hierarchical clustering, and separately clustered GPT-4o-generated image appearance descriptions (threshold d < 2) with a UMAP 2D visualization.

Why This Matters

Impact on research: The paper shows that a general-purpose LMM can act as a stronger proxy for a subjective human concept than purpose-built models, while also documenting concrete failure modes (uniformly positive single-image answers, position bias, unreliable multi-image handling in other LMMs). It reframes subjective labeling as a relative-ranking problem and provides data and code for follow-up work.

Real-world applications:

• Content curation and recommendation: ranking candidate images or thumbnails by likely human interest.
• Large-scale dataset labeling: using LMM pairwise judgments to reduce costly manual annotation effort, then distilling into a lightweight deployable ranker.
• Media editing and advertising: automatically preselecting the most engaging image from a set of candidates.
• Accessibility and assistive tools: describing or prioritizing visual content by likely appeal.

Industry relevance: The distillation result matters practically, since the trained ranker is a small CLIP-based model rather than an expensive LMM call. The finding that GPT-4o is order-biased in 36% of comparisons is a direct warning for anyone building pairwise-preference pipelines or using LMMs as automatic evaluators.

Future Directions

• Investigate the remaining gap between GPT-4o and human annotation, including demographic factors, which the paper's synthetic demographic prompts did not resolve.
• Extend the study beyond Flickr images to other datasets, since the authors note their Int10k results dropped to roughly 59% accuracy and attribute this to a domain gap between everyday images and cinematic sequences.
• Improve or fix multi-image handling in other LMMs such as Llama 3.2 and DeepSeek VL2, which could not be fairly compared to GPT-4o in the pairwise setting.
• Explore what GPT-4o finds interesting that humans do not (for example vibrant color and dynamic action scenes) to better characterize machine versus human interest.

Target Audience

Researchers and practitioners in computer vision, multimodal learning, and human-computer interaction who work on subjective visual attributes, preference modeling, or learning-to-rank from model-generated labels. It is also useful for applied teams using LMMs as automatic annotators or evaluators who need to know the concrete failure modes, and for anyone building recommendation or content-curation systems over image collections. Readers without a machine learning background can follow the study design and headline results, but the ranking and embedding methodology assumes some familiarity with standard evaluation metrics.

Authors’ abstract

Our daily life is highly influenced by what we consume and see. Attracting and holding one's attention -- the definition of (visual) interestingness -- is essential. The rise of Large Multimodal Models (LMMs) trained on large-scale visual and textual data has demonstrated impressive capabilities. We explore these models' potential to understand to what extent the concepts of visual interestingness are captured and examine the alignment between human assessments and GPT-4o's, a leading LMM, predictions through comparative analysis. Our studies reveal partial alignment between humans and GPT-4o. It already captures the concept as best compared to state-of-the-art methods. Hence, this allows for the effective labeling of image pairs according to their (commonly) interestingness, which are used as training data to distill the knowledge into a learning-to-rank model. The insights pave the way for a deeper understanding of human interest.