A Multimodal Benchmark for Framing of Oil & Gas Advertising and Potential Greenwashing Detection

AI summary

Overview

• Research area: Multimodal natural language processing and computer vision applied to climate communication — specifically framing analysis of oil and gas (O&G) advertising video, with greenwashing detection as the motivating application.
• Technical level: Intermediate. The task framing is intuitive, but the paper assumes familiarity with multi-label classification, F-scores, in-context (few-shot) prompting, and vision-language models (VLMs).
• Scope: The paper introduces and releases the first expert-annotated multimodal (video + transcript) benchmark for detecting framing techniques in O&G advertising across Facebook and YouTube, and benchmarks six recent VLMs on it.

What This Paper Is About

Oil and gas companies spend heavily on public relations that portrays them as climate-friendly, and researchers and journalists suspect these campaigns of "greenwashing" — making people believe a company does more for the environment than it really does. Prior computational work on framing only used ad text, but greenwashing often lives in imagery (wind turbines, smiling workers, research labs) and around 30% of the videos in this dataset contain no spoken language at all. The goal is to build a video-based benchmark that lets researchers measure whether modern vision-language models can identify these framings at scale, and to test whether that capability can support systematic greenwashing detection.

Key Contributions

1. A multimodal framing benchmark for O&G video ads. The dataset covers 706 videos, 35,476 seconds of footage, 5,607 transcript segments, and 1,183 annotations, spanning more than 50 entities across 20 countries, with videos published from 2010 to 2025. It provides 13 framing types in total and is explicitly designed to evaluate VLMs rather than text-only models.
2. Two complementary domains with different label schemas. The Facebook subset (320 videos, 17 entities, 1 country, 381 labels) uses seven fine-grained "climate obstruction" framing labels derived from prior work; the YouTube subset (386 videos, 42 entities, 20 countries, 802 labels) uses six impression-based framing labels developed by the authors.
3. Benchmark evaluation of six VLMs under zero-shot and 1-shot settings, including a custom entity-aware 1-shot retrieval mechanism, plus ablation studies and error analyses.
4. A pilot framing of the benchmark as a greenwashing-detection tool, with temporal trend analysis and company-level framing profiles, plus publicly released code and data.

Main Findings

• Environmental messages are detectable, green innovation is not. GPT-4.1 detects environmental messages with 79% F1 score, while the best model reaches only 46% F1 on framing around green innovation.
• GPT-4.1 is the strongest overall model. It achieves the highest micro-averaged "All" score on YouTube zero-shot at 71.0 and on Facebook 1-shot at 72.6. GPT-4.1's YouTube zero-shot score (71.0) actually exceeded its 1-shot score (69.3), which the authors read as evidence of its capability in this domain.
• 1-shot prompting helps most models, but not uniformly. The 1-shot gains are large for open-weight models such as Qwen2.5-VL, and on Facebook the 1-shot setting outperformed zero-shot for all models. The exception is GPT-4.1 on YouTube.
• Transcript inputs matter, especially on Facebook. Ablating the transcript with Qwen2.5-VL 32B dropped YouTube from 66.2 to 61.2 and Facebook from 70.5 to 60.6. The authors note that for smaller models such as DeepSeekVL2 on YouTube, adding transcripts sometimes degraded performance.
• Entity-aware retrieval helps. Removing embedding-based search dropped Qwen2.5-VL 32B to 64.0 (YouTube) and 59.1 (Facebook); removing entity restriction dropped it to 65.6 and 68.1. Entity restriction's effect was mixed across all models, likely because some entities have few training videos.
• Label difficulty tracks the availability of concrete cues. Labels such as "Community and Life", "Work", and "Environment" scored higher, being tied to visible families, workers, or environmental commitments. "Green Innovation", "Patriotism", "Economy and Business", "PB", "GA", and "SA" are described as low-resource and harder.
• Models over-label hard labels; experts under-label them. For high-scoring labels, precision tends to exceed recall; for difficult labels such as "Economy and Business", "Green Innovation", and "Patriotism", recall exceeds precision. Qwen2.5-VL 7B was by far the most under-labeling model (171 labels output, versus a mean of 349), while GPT-4o-mini over-labeled (409 labels).
• Models miss the gold label co-occurrence structure. DeepSeekVL2 produced the pair "Community and Life" and "Economy and Business" 115 times versus 63 times in the gold data, while under-producing the gold's most common pair, "Community and Life" and "Work" (17 versus 131).
• Video length is a challenge. Shorter videos (0–60 seconds) were harder for the model, which the authors attribute to their being more contextualized and vague.
• Cultural context varies by model. GPT-4o-mini and Qwen2.5-VL performed well on videos from European and North American companies, while DeepSeekVL2 and InternVL2 outperformed GPT-4o-mini on videos from Asia.
• Temporal trends are recoverable. The predicted ratio of "Environment"-labeled YouTube videos closely followed the gold trend, including an increasing trend after 2020, which the authors note correlates with (but does not establish causality for) the Biden administration period.
• Annotation agreement is moderate. The YouTube annotation reached a final Fleiss' Kappa of 0.61, with 0.46 under an alternative calculation method. The Facebook inter-annotator agreement is not reported in this paper; the authors point to the original text-based studies. Facebook labels are described as "distant" because they were originally based on ad text rather than the videos.

Methodology in Plain English

The authors built a dataset rather than a new model. For Facebook, they started from a prior dataset of US ads (2020–2021) tied to climate obstruction messaging, collected 320 videos that were still available, transcribed them with Whisper-1, and reused the seven fine-grained labels from that earlier work — accepting that these labels are "distant" since they came from ad text, not the video. For YouTube, they used LobbyMap to pick target entities, pulled up to 30 ad-like videos per company channel (720 total), randomly sampled 500 for annotation, and kept 386 after removing deleted or clearly non-ad videos (for example, earnings calls). The YouTube labels were redesigned through multiple annotation rounds because the Facebook schema produced poor agreement on the more implicit YouTube content. Both subsets were split 50:50 into train and test.

For the benchmark, each video is split into frames. Transcript segments are used to select frames dynamically: for each segment, the frame at the midpoint timestamp is taken, up to a model-dependent limit (10 frames for GPT-4.1, GPT-4o-mini, and Qwen2.5-VL; 3 frames for InternVL2 and DeepSeek-VL2). The prompt contains annotation instructions, the sampled frames with their transcript segments, and optionally one training example. For 1-shot prompting, the authors add an entity-aware retrieval step: candidates are restricted to the same entity, then ranked by cosine similarity over CLIP embeddings that average frame embeddings and transcript embeddings, combined with weights of 0.5 and 0.5.

Why This Matters

• Research impact: It moves framing and greenwashing research from text-only benchmarks to video, which the authors argue is necessary for a holistic view of corporate messaging. It also gives the climate-NLP community a cross-domain, entity-level, and temporal evaluation resource with a moderate-but-honest annotation ceiling, so improvements are measurable.
• Real-world applications:
  • Computer-assisted screening of ad libraries and corporate YouTube channels for greenwashing risk, reducing the manual research burden the authors describe.
  • Company-level framing profiles that analysts or researchers can use to compare a firm's messaging against industry norms.
  • Temporal monitoring of how environmental messaging shifts over time, including the post-2020 increase the pilot study captures.
  • Selection and stress-testing of VLMs for video-framing tasks, using the paper's per-label, per-region, and per-length error breakdowns.
• Industry relevance: The work speaks directly to advertising and public relations accountability, ESG-adjacent corporate analysis, and the practical question of whether automated tools can be trusted to interpret strategic communication. The performance gap between "Environment" (79% F1) and "Green Innovation" (46% F1) shows that the subtler the framing, the less any current model can be relied on.

Future Directions

• More granular annotation. Extending labels to capture whether a message is implicit or explicit, and whether it is conveyed visually, through spoken language, or as captions.
• Coverage of more domains with high greenwashing risk, beyond Facebook and YouTube advertising.
• Improving model handling of the identified failure modes: implicit framing, videos of varying length (especially short ones), imbalanced labels such as "Patriotism" and "Green Innovation", over- versus under-labeling, and differing cultural backgrounds.
• Verifying the greenwashing interpretation. The pilot temporal trend and company-level profiles are presented as promising candidates for follow-up manual evaluation, and the paper explicitly does not verify causality for the post-2020 increase.

Target Audience

Researchers in climate NLP, computational social science, and multimodal machine learning; practitioners building or evaluating vision-language systems for video classification; and analysts, journalists, or policy-facing researchers who want to monitor corporate climate messaging or greenwashing signals at scale. Readers need only a basic grasp of classification metrics and prompt-based model evaluation.

Authors’ abstract

Companies spend large amounts of money on public relations campaigns to project a positive brand image. However, sometimes there is a mismatch between what they say and what they do. Oil & gas companies, for example, are accused of "greenwashing" with imagery of climate-friendly initiatives. Understanding the framing, and changes in framing, at scale can help better understand the goals and nature of public relations campaigns. To address this, we introduce a benchmark dataset of expert-annotated video ads obtained from Facebook and YouTube. The dataset provides annotations for 13 framing types for more than 50 companies or advocacy groups across 20 countries. Our dataset is especially designed for the evaluation of vision-language models (VLMs), distinguishing it from past text-only framing datasets. Baseline experiments show some promising results, while leaving room for improvement for future work: GPT-4.1 can detect environmental messages with 79% F1 score, while our best model only achieves 46% F1 score on identifying framing around green innovation. We also identify challenges that VLMs must address, such as implicit framing, handling videos of various lengths, or implicit cultural backgrounds. Our dataset contributes to research in multimodal analysis of strategic communication in the energy sector.