EviScene: Beyond "Is It AI?" for Agentic Image Factuality Assessment

AI summary

Overview

Research area: Computer vision and multimodal fact-checking, specifically agentic (tool-using) verification of image content rather than AI-image detection.

Technical level: Advanced. The paper assumes familiarity with ReAct-style agents, supervised fine-tuning, and knowledge distillation; the benchmark itself is accessible, but the training method is not beginner material.

One-sentence scope: The paper defines an "Image Factuality Assessment" task, releases a 10,174-image benchmark (EviLens) with twelve factual error types, and trains a compact 9B-parameter ReAct agent (EviScene) that gathers web and image evidence to decide whether a depicted situation is supported or refuted.

What This Paper Is About

Most image-authenticity research asks whether a picture was AI-generated, but that answer says nothing about whether the scene it shows actually happened, or happened the way it appears. This paper instead asks a system to look at a single image, with no accompanying text claim, and decide whether the depicted situation — the people, actions, relationships, time, and place — is factually supported or refuted, with an explanation that cites real evidence. The authors build both the benchmark needed to measure this and an agent that performs the investigation.

Key Contributions

1. A new task formulation. "Image Factuality Assessment" is defined as: given one image and a generic verification instruction (no supplied claim, caption, or prompt), return a supported/refuted verdict plus an evidence-grounded explanation. The label concerns the depicted situation, not how the image was produced.

2. The EviLens benchmark. 10,174 images (3,114 supported, 7,060 refuted) built through controlled construction and web collection, spanning twelve factual error types grouped into three families: event and relation, attribute and detail, and context. Every example passed manual screening by two domain experts with a third resolving disagreements, and the dataset is split into 8,647 training and 1,527 test images. Evaluation covers both verdict accuracy and evidence sufficiency.

3. The EviScene agent. A multimodal ReAct agent that interleaves visual inspection, web search, text-based image search, reverse-image search, source-page reading, and reference-image comparison, using new observations to refine its factual questions and revisit relevant image regions. Three tool groups (visual, retrieval, evidence) are available at every turn, with up to 24 actions per investigation.

4. A two-stage training recipe for a compact policy. Full-parameter supervised fine-tuning of Qwen3.5-9B on 4,872 reviewed teacher trajectories from Qwen3.5-397B-A17B, followed by privileged on-policy self-distillation (PSD) on verified decisions drawn from the policy's own investigations, plus ablations across tools, image sources, and generators.

Main Findings

• Investigation lifts evidence sufficiency on every backbone. Across the five shared backbones, EviScene improves the Strict Evidence Sufficiency Rate (SESR — the share of cases with a correct verdict and sufficient grounds for the decisive fact) over direct judgment by 27.11 to 46.82 percentage points. Balanced accuracy also improves for four of the five.

• The trained 9B policy is the strongest on evidence. Qwen3.5-9B after SFT+PSD ("Full") reaches 79.05% balanced accuracy and 52.26% SESR on the 1,527-image test set, versus 72.34% and 42.89% for the base agent with the same backbone and tools — gains of 6.71 and 9.37 points. Its SESR is the highest among all evaluated agents.

• It beats a far larger model. The Full policy outperforms the Qwen3.5-397B-A17B agent under the same action limit by 7.33 points in balanced accuracy and 8.97 points in SESR.

• The best raw verdict accuracy belongs to a frontier agent. Gemini 3.7 Flash with EviScene reaches 81.57% balanced accuracy — the top score — but its SESR is 42.89%, well below the trained 9B policy's 52.26%.

• Direct judgment produces lopsided verdicts. Gemini 3.1 Pro recalls 96.52% of refuted cases but only 28.65% of supported ones; GLM-4.6V shows the mirror pattern at 95.49% supported and 24.43% refuted recall. Direct models also score near-zero on evidence sufficiency (GPT-5.5 highest at 3.67%, Qwen3.5-9B lowest at 1.70%).

• One backbone does not improve in accuracy. Gemini 3.1 Pro moves from 62.58% to 62.42% balanced accuracy with investigation, even though its SESR rises from 5.57% to 32.68%.

• SFT mainly fixes the supported/refuted imbalance. The base agent recalls 93.48% of refuted but only 51.19% of supported cases. SFT moves supported recall to 76.39% and refuted to 81.22%, narrowing the gap from 42.29 to 4.83 points and raising balanced accuracy from 72.34% to 78.81%. PSD narrows the gap further to 2.11 points.

• PSD adds a small amount. Relative to SFT, the Full policy gains 0.24 points in balanced accuracy and 1.77 points in SESR; macro F1 is slightly lower for Full (74.67%) than SFT (75.69%). SFT alone accounts for most of the improvement over the base agent.

• Web search is the most valuable tool family. Removing it from the Full policy drops balanced accuracy by 4.68 and SESR by 16.37 points (to 74.37 / 35.89). Removing image retrieval costs 4.53 / 12.25, and removing evidence inspection costs 2.50 / 6.22. SESR falls more than accuracy in every ablation.

• Gains hold on both acquisition routes. From direct judgment to Full, SESR rises from 1.29% to 52.75% on constructed images and from 3.46% to 50.17% on web-collected images; balanced accuracy is similar across the two at 76.75% and 77.21%.

• Gains transfer to unseen generators. The Full policy reaches 46.98% SESR on Nano Banana Pro, 58.39% on GPT Image 2, and 54.64% on Seedream 5.0 Lite. GPT Image 2 and Seedream 5.0 Lite were not used to build training images, so the improvement is not tied to the training generator.

Methodology in Plain English

The authors first had to build the test material. They used a "curation agent" that planned searches, read sources, and proposed candidate images. For constructed examples, the agent started from a documented fact and either rendered it faithfully (supported) or deliberately altered a detail or relation (refuted); a subset of refuted images fabricated an event involving real people or places in conflict with records. For web-collected examples, the agent started from circulating images — both photographs and synthetic ones — and recovered the source information needed to judge them. Automated checks screened for clarity, evidence alignment, duplication, and leakage, after which experts reviewed every candidate and either accepted, edited, or rejected it. Each retained example carries an image, label, explanation, evidence, and provenance record; those records are kept private and used only for scoring.

At inference, the agent works in a loop. It sees the image and its own history of prior reasoning steps, tool calls, and observations. It reasons briefly, picks one tool action — crop and zoom on a region, read text, count objects, search the web, reverse-image search, open a source page, compare against a retrieved reference image — and receives an observation. That observation can change what it decides to check next: an external identification of a person may send it back to inspect a specific region, or a date found online may prompt a search for a different year. The loop runs until the agent finishes or hits a 24-action budget, then it writes a verdict, an explanation, and a list of the specific observations it is citing as evidence.

For training, the authors generated teacher trajectories using Qwen3.5-397B-A17B through the same interface the student would use, keeping only structurally complete trajectories whose recorded observations actually supported the final report; 4,872 survived review. Qwen3.5-9B was then fully fine-tuned for three epochs (3,084 optimizer steps, learning rate 10⁻⁵), with loss weights of 2 on tool-call and final-report tokens, 1 on reasoning tokens, and 0 on instruction and observation tokens — so the model is rewarded for correct actions and reports more than for narrating.

The second stage, PSD, tries to fix mistakes the model makes in its own investigations. A frozen copy of the current policy runs new investigations. A separate training-only model, Gemini 3.7 Flash, identifies the earliest recoverable faulty decision and writes a short procedural hint — guidance about procedure, not the answer, not a specific query, not a case-specific solution. The hint is inserted into the original context, the frozen policy continues from there, and the continuation is kept only if the decision gets corrected, the final report is right and evidence-backed, and the trajectory passes an audit. The student is then trained to match the frozen teacher's renormalized top-K token probabilities on that selected completion — but with the hint removed from its context, so it must learn the corrected behavior without the crutch. Successful investigations are also used as "preservation" targets so the model does not lose behavior that was already working.

Scoring uses Gemini 3.7 Flash as a judge against private reference material, checking verdict correctness, factual alignment, and whether the stated grounds are sufficient. Fact-checking websites are blocked during investigation so agents cannot look up the answer directly.

Why This Matters

Impact on research. The paper separates image authenticity from image factuality — a distinction the benchmark comparison table makes explicit, since adjacent resources like GenImage (2.68M images), NewsCLIPpings (988,283 pairs), and MOCHEG (15,601 claims) each miss at least one of the properties EviLens combines: no supplied claim, a world-fact verdict, web evidence, evidence-bearing output, and an origin-independent label. It also supplies a concrete result that a trained 9B open-weight policy can beat a 397B model on evidence sufficiency for this task, which matters for anyone weighing agent training against model scale.

Real-world applications.

• Newsroom verification: a reporter receives a viral photo with no caption and needs to know whether the event, date, and location it depicts are real, plus citable sources for the answer.
• Platform trust and safety: moderation queues currently flag images as "AI-generated or not," which does not address whether the depicted claim is false; this framing does.
• Crisis and disaster response: images circulating during emergencies often reuse real footage from a different year or place, and temporal and geographic displacement are two of the twelve error types explicitly modeled.
• Insurance and legal documentation: claims supported by photos can be checked against records for identity substitution, numerical alteration, or source impersonation.

Industry relevance. The tool ablation shows that evidence sufficiency depends heavily on retrieval infrastructure — dropping web search costs 16.37 SESR points — so the practical value sits at the intersection of model training and search/OCR pipeline engineering. The reported runs use Serper for web and image search, Serper Lens for reverse-image search, Jina Reader for page content, and the Baidu OCR API, all wrapped in a 24-action budget, which is a realistic deployment shape. The finding that a 9B model can be trained to this level also implies a cost story: multi-step evidence gathering is affordable without a frontier model at inference time.

Future Directions

• Closing the evidence-sufficiency gap. SESR tops out at 52.26%, meaning roughly half of even the best agent's correct verdicts lack sufficient stated grounds. What training signal or reward would push this higher is unresolved — PSD contributed only 1.77 SESR points over SFT.

• Making PSD worth its complexity. The distillation stage improved balanced accuracy by 0.24 points and lowered macro F1 from 75.69% to 74.67%. Whether a different decision-localization strategy, a larger set of verified repairs, or more rounds would yield a clearer return is an open question.

• Reducing dependence on a proprietary judge and intervention model. Gemini 3.7 Flash serves both as the evaluation judge and as the training-only model that localizes errors and writes hints. How sensitive the reported numbers are to that choice, and whether an open substitute exists, is not established in the paper.

• Extending beyond static images and a fixed action budget. The agent runs under a 24-action cap and handles single images; the paper reports no results on video, image sets, or longer investigations, and does not analyze how performance scales with the action budget.

Target Audience

Researchers and engineers working on multimodal agents, fact-checking, and misinformation detection will get the most from this paper, particularly those interested in training small open-weight models to perform multi-step tool use. It is also relevant to benchmark designers, since the twelve-type taxonomy and the joint accuracy-plus-evidence scoring protocol are reusable, and to trust-and-safety teams evaluating whether authenticity detection alone is sufficient for their moderation pipeline. Readers without background in agent training can still use the task definition, taxonomy, and main results tables, but the PSD section requires familiarity with distillation.

Authors’ abstract

Determining whether an image is AI-generated does not establish whether the situation it depicts is factually supported. We formulate Image Factuality Assessment: given an image and a generic verification instruction, a system returns a supported or refuted verdict with an evidence-grounded explanation. Without a supplied textual claim, the system must progressively refine factual questions during investigation and align external evidence with the specific situation depicted. We introduce EviLens, an expert-reviewed benchmark of 10,174 supported and refuted images from controlled construction and web collection, covering twelve factual error types and evaluating both verdict accuracy and evidence sufficiency. We develop EviScene, a multimodal ReAct agent that interleaves visual inspection, external retrieval, and evidence analysis, using new findings to refine its questions and revisit relevant image details. We train a compact policy with supervised fine-tuning on reviewed teacher trajectories, followed by privileged on-policy self-distillation applied to verified decisions from the policy's own investigations. On the 1,527-image test set, the trained Qwen3.5-9B agent achieves 79.05% balanced accuracy and a Strict Evidence Sufficiency Rate (SESR) of 52.26%, improving over the same-backbone base agent by 6.71 and 9.37 percentage points, respectively. Its SESR is the highest among the evaluated agents.Code, models, and data will be released at https://github.com/wangzian828/EVISCENE and https://huggingface.co/ZiAnwang/EviScene.