Revis: Sparse Latent Steering to Mitigate Object Hallucination in Large Vision-Language Models

AI summary

Overview

Research area: Large Vision-Language Models (LVLMs), mechanistic interpretability, hallucination mitigation, inference-time model steering.

Technical level: Advanced. The paper assumes familiarity with transformer hidden states, latent-space geometry, orthogonal projection (Gram-Schmidt), and benchmark metrics such as CHAIR and POPE.

Scope: The paper proposes Revis, a training-free inference-time framework that uses orthogonal projection and sparse, risk-gated activation steering at a single calibrated layer to reduce object hallucination in LVLMs without degrading general reasoning.

Publication details: arXiv:2602.11824v2 [cs.AI], 11 May 2026. All authors are affiliated with Ant Group. Code is released at https://github.com/antgroup/Revis.

What This Paper Is About

Large Vision-Language Models sometimes describe objects that are not in the image — a failure called object hallucination. The authors argue this happens because visual information gets suppressed by the model's "language prior" (its tendency to guess from text patterns when visual evidence is weak), and because visual and textual representations become entangled in the deeper network layers. Revis aims to re-activate the suppressed visual signal directly inside the model's hidden states, rather than correcting the output text after the fact.

Key Contributions

1. Mechanistic diagnosis. The authors identify feature entanglement in the latent space — where raw visual vectors overlap heavily with language-prior vectors in deep layers — as a primary cause of hallucination, and validate through causal probing that purified visual vectors can re-activate visual awareness.

2. The Revis framework. A training-free method that mathematically decouples visual information from language priors via orthogonal projection (Gram-Schmidt), selects a single optimal intervention layer through calibration, and applies a dynamic, risk-aware intervention only when the model drifts away from the visual information subspace.

3. Broad empirical validation. Experiments on five standard benchmarks (POPE, CHAIR, MME, MM-Vet, MMMU-Pro) and seven models (Qwen2.5-VL-7B-Instruct, Qwen2.5-VL-32B-Instruct, Qwen3-VL-8B-Instruct, LLaVA-1.5-7B, LLaVA-NeXT-7B (Mistral-based), InternVL3-8B, InternVL3.5-8B) show superior mitigation and efficiency compared with five baselines (VCD, M3ID, ONLY, AGLA, VTI).

4. Minimal inference overhead. Because the intervention acts on internal activations rather than requiring parallel forward passes, Revis avoids the latency penalty that contrastive decoding methods incur.

Main Findings

• Hallucination reduction of approximately 19% over state-of-the-art baselines. On Qwen2.5-VL, Revis reduced the CHAIR_S score of the Regular (greedy decoding) baseline from 31.00% to 25.00%, described by the authors as a 19.35% reduction in hallucination rate, outperforming AGLA (29.00%) and ONLY (33.00%).

• POPE results. Revis achieved the highest F1 scores across all POPE subsets on Qwen2.5-VL. In the Adversarial setting it surpassed the strongest baseline (VCD) by 0.77% in Accuracy.

• Reasoning capability preserved or improved. MME Overall rose to 2345.30 versus the Regular baseline's 2327.52, and MM-Vet reached 72.16 versus 70.18. On MMMU-Pro, Revis improved Qwen2.5-VL's average score from 0.348 to 0.357.

• The entanglement problem is real and severe. Applying the prior method VTI (which steers with an entangled, averaged difference vector across all layers) reduced CHAIR_S from 14.00% to 9.00% but collapsed MM-Vet from 70.18 to 56.38 (CHAIR result reported with max token = 64). Naive steering with the raw visual vector also caused model collapse — infinite repetition and empty outputs — as intensity α exceeded 0.5 and approached roughly 0.7.

• Orthogonalization is what makes high-intensity steering safe. The purified vector allowed strong intervention without degeneration, whereas the raw vector did not.

• Layer choice matters, and it is model-specific. Ablating fixed layers on Qwen2.5-VL showed that intervening at Layer 23 was ineffective (CHAIR_S 37.00%) and harmful to utility (MM-Vet 60.37), while performance peaked at Layer 27, which also gave the highest MM-Vet (72.16). The authors report that the optimal layer L* differs across models, motivating the calibration-based backward search.

• A dynamic risk threshold is necessary. Removing the threshold τ caused model collapse in LLaVA-1.5 (infinite repetition loops, no valid outputs) and degraded CHAIR_S from 34.00% to 26.00% on LLaVA-NeXT. On Qwen2.5-VL, removing τ gave nearly identical numbers (90.02 vs. 90.00 POPE average, 25.0 CHAIR_S in both cases, 72.57 vs. 72.16 MM-Vet).

• Generalization across architectures. On LLaVA-1.5, Revis reduced CHAIR_S from 54.00% to 30.00% (a 44.4% reduction), and improved MME from 1754.33 to 1787.27 while maintaining MM-Vet at 33.53. On LLaVA-NeXT it recorded the lowest CHAIR_S (26.00%) and highest POPE-Adversarial F1 (86.73%), increased MME Overall to 1817.32 (baseline 1803.98), and held MM-Vet at 47.48.

• Scales to a larger backbone. On Qwen2.5-VL-32B-Instruct, Revis reduced CHAIR_S across the 64, 128, and 512 maximum-token budgets (9.00/22.00/52.00 to 8.00/18.00/46.00), preserved MM-Vet (74.31 vs. 74.13), and improved MMMU-Pro from 0.422 to 0.435.

• Negligible latency cost. Revis achieved a mean Time Per Token of 0.025s, statistically indistinguishable from Regular decoding (approximately 0.025s), whereas VCD and M3ID approximately doubled inference cost (approximately 0.05s) due to parallel forward passes.

• Hyperparameter sensitivity. Hallucination mitigation followed a convex trajectory with respect to steering intensity α, peaking at α = 1.6. Beyond that, CHAIR_S degraded (25.00% to 29.00%). Default settings were k = 0.8 and α = 1.6 for Qwen2.5-VL-7B-Instruct.

Methodology in Plain English

The work starts with a diagnostic experiment. The authors sample 1,000 instances from the Nullu dataset, each containing an image, a ground-truth caption, and a hallucinated caption. For every image-text pair they extract the hidden state of the last token ([EOS]) under five counterfactual input conditions, forming a 5-cluster semantic state space that includes visual input with ground-truth text, visual input with hallucinated text, no visual input with each type of text, and a no-visual-input refusal response. Visualizing these states with t-SNE shows that factual and hallucinated states overlap in shallow layers but separate in deep layers (for example, Layer 27 of Qwen2.5-VL). This means hallucination can be treated as a state-selection failure — the model drifting from the factual state toward the hallucinated one.

Next, the authors define two vector quantities from reference data. A "raw visual vector" is the average difference between the hidden state with visual input and ground-truth text and the state with no visual input but ground-truth text — intended to represent the net contribution of vision. A "language prior vector" is the average difference between the state with no visual input and hallucinated text and the state with no visual input and a refusal response — representing the direction of ungrounded guessing. They find high cosine similarity between these two vectors in deep layers, meaning the raw visual vector is not purely visual; it carries language priors along with it.

To fix this, Revis applies Gram-Schmidt orthogonalization: it projects the raw visual vector onto the orthogonal complement of the language prior vector, producing a purified visual steering vector that is guaranteed to be perpendicular to the language prior. Steering along this direction adds visual grounding without amplifying the model's tendency to fabricate.

Because separability only emerges in deep layers, dense multi-layer intervention is unnecessary. Revis defines a hallucination risk score as the negative cosine similarity between a hidden state and the purified visual vector, then searches backward from the last layer to the first, choosing the deepest layer where the average risk on hallucinated states exceeds the average risk on factual states. Using 100 images from COCO train 2014 with a POPE-like binary existence protocol, it then sets a threshold τ at a percentile k of the factual risk distribution.

At inference, at each decoding step the model computes the instantaneous risk at the selected layer. If the risk exceeds τ, it adds α times the purified visual vector to the hidden state; otherwise it does nothing. This gating keeps the model's normal behavior intact when it is already visually grounded.

Why This Matters

Impact on research. The paper reframes hallucination mitigation as a latent-space geometry problem rather than a decoding-time filtering problem. It provides causal evidence that feature entanglement between vision and language prior — not merely weak visual features — underlies the failure, and demonstrates that orthogonalization makes high-intensity steering stable where naive steering collapses the model. This connects mechanistic interpretability tooling (activation steering, causal probing) with a practical reliability fix for multimodal systems, and offers an inference-time alternative to expensive RLHF/DPO alignment pipelines such as LLaVA-RLHF and HA-DPO.

Real-world applications (implied by the results, not evaluated in the paper):

• Accessibility tools that describe images for blind or low-vision users, where fabricated objects are a safety issue.
• Medical or scientific image reporting assistants, where describing non-existent findings has direct consequences.
• E-commerce and catalog automation, where product attributes must match the actual photograph.
• Latency-constrained deployments such as on-device or real-time multimodal assistants, given the reported 0.025s Time Per Token.

Industry relevance. The method is training-free, requires only small calibration sets (N = 100 paired examples from Nullu for vector extraction, and 100 COCO train 2014 images for calibration), and adds negligible inference latency, so it can be layered onto existing deployed LVLMs without retraining. The authors are all affiliated with Ant Group, and the code is publicly released, signaling direct industrial intent.

Future Directions

• Extending beyond base-model limits. The authors state that Revis relies on the base model's representations and cannot rectify absolute perceptual blindness — cases where the model simply never perceived the object. Whether this can be addressed by combining latent steering with perceptual-level interventions is not resolved.

• Scaling validation. The current evaluation covers 7B/8B and 32B architectures; verifying scalability on larger models (for example, 70B+) is explicitly left for future work.

• Reducing calibration dependence. The optimal layer L* is model-specific and requires a calibration search, and the threshold depends on a model-specific hyperparameter k. Automating or generalizing this calibration across model families is an open problem.

• Broadening hallucination categories. The paper's headline results center on object hallucination. Its Appendix E.1 illustrates mitigation of attribute errors, but the extent to which the orthogonalization approach transfers to relational, temporal, or counting hallucinations is not reported.

Target Audience

Researchers and engineers working on multimodal foundation models, model reliability, and hallucination mitigation; practitioners deploying LVLMs in latency-sensitive or safety-sensitive products; and mechanistic interpretability researchers interested in activation steering and causal probing of vision-language representations. Readers should be comfortable with linear algebra and transformer internals, as the core method hinges on orthogonal projection and hidden-state geometry.

Authors’ abstract

Despite the advanced capabilities of Large Vision-Language Models (LVLMs), they frequently suffer from object hallucination. One reason is that visual features and pretrained textual representations often become intertwined in the deeper network layers. To address this, we propose REVIS, a training-free framework designed to explicitly re-activate this suppressed visual information. Rooted in latent space geometry, REVIS extracts the pure visual information vector via orthogonal projection and employs a calibrated strategy to perform sparse intervention only at the precise depth where suppression occurs. This surgical approach effectively restores visual information with minimal computational cost. Empirical evaluations on standard benchmarks demonstrate that REVIS reduces object hallucination rates by approximately 19% compared to state-of-the-art baselines, while preserving general reasoning capabilities.