Skip to content
AI.info

The Pulse

Study finds AI safety guards can give unsupported explanations

Researchers found that explanations accompanying multimodal safety decisions often contained unsupported visual facts or incorrect safety interpretations. They propose a training method that rewards visual tool use when it improves grounding.

Study finds AI safety guards can give unsupported explanations

AI systems that judge the safety of image-and-text content can produce explanations containing unsupported visual details or incorrect interpretations, even when their final classification is right, a preprint found.

Source arxiv.org

The study examined eight rationale-generating safety guard methods across five vision-language backbones and four safety datasets. The authors, who are affiliated with Harbin Institute of Technology, submitted the preprint to arXiv on Nov. 19.

Two kinds of error

Multimodal safety guards classify image-and-text inputs as harmful or benign. Some also produce a rationale explaining the basis for the decision, including where a risk may arise from an image, text or their relationship.

The researchers describe two types of hallucination in those explanations. A visual hallucination introduces facts that the image does not support; a safety attribution hallucination gives observed content an incorrect safety interpretation relative to the dataset's annotations.

The preprint identifies over-flagging, in which a rationale assigns an unsupported harmful attribution to benign content, and under-flagging, in which it omits the harmful attribution from harmful content. It also identifies mis-attribution, where an explanation assigns harmfulness to the wrong target despite correctly identifying the content as harmful.

How the explanations were checked

To assess visual grounding, the researchers used a language model to extract descriptive clauses from each rationale, excluding subjective judgments and hypothetical statements. They then broke the clauses into individual visual, attribute and relation facts. A verifier determined whether each fact was supported by the corresponding image, producing a visual evidence score.

For safety attribution, a judge extracted the attribution from the rationale within the relevant dataset's annotation space. The researchers compared it with the ground-truth attribution. This assessment addresses a different problem from visual grounding: an explanation can describe an image accurately but still give an incorrect account of its safety implications.

The researchers found that neither language-workflow methods nor methods using visual inspection tools consistently eliminated hallucinations. They also found that higher classification accuracy did not necessarily mean the supporting explanations were better grounded.

Visual tools and training

The preprint examines a method known as Think-with-Image, in which models can inspect visual information using operations such as cropping, rotation and optical character recognition before generating a rationale. In the researchers' analysis, safety-critical attributes and relationships were more error-prone than basic visual facts, while the methods used tools infrequently and often made coarse crops.

The authors propose a training approach called Tool-Contrastive Grounding Reinforcement Learning, or TCG-RL. It compares rationales generated with and without visual tools and rewards tool use when it produces a better-grounded explanation.

The authors report that the approach improved visual rationale grounding by approximately 20% over strong baselines while maintaining comparable safety prediction accuracy. They also report that it produced more concise and safety-relevant rationales.

Sources

  1. Original source · Preprint When Safety Rationales Hallucinate: Diagnosing and Grounding Multimodal Safety Guards arXiv:2611.25793

Related stories

Explore

More articles