FlexAC: Towards Flexible Control of Associative Reasoning in Multimodal Large Language Models

AI summary

Overview

• Research area: Computer Vision / Multimodal Large Language Models (MLLMs) — specifically internal interpretability and inference-time control of model behavior.
• Technical level: Advanced. The paper assumes familiarity with transformer hidden states, activation steering/steering vectors, contrastive decoding, and multimodal benchmarks.
• Scope: The paper identifies where "associative reasoning" is encoded inside MLLMs and introduces a training-free framework, FlexAC, that can turn that reasoning up or down at inference time to make a model either more faithful or more creative.

What This Paper Is About

Multimodal large language models face a trade-off: factual tasks (like image captioning) need low-association, grounded output, while creative tasks (like storytelling or event planning) benefit from high-association, imaginative output. Existing methods specialize in one direction only — hallucination-mitigation techniques such as Contrastive Decoding and Direct Preference Optimization improve faithfulness but suppress associative ability, while controllable creativity enhancement remains largely unexplored. This paper's goal is to give MLLMs a single, flexible dial that can move them toward either faithfulness or creativity depending on the task.

Key Contributions

1. A unified framing of hallucination and creativity. The authors link both phenomena to a shared "associative reasoning" mechanism and identify middle-layer representations as the key control point inside the model.
2. FlexAC, a lightweight and training-free framework. It modulates associative strength at inference time by injecting hallucination-derived steering vectors into middle-layer features, enabling task-aware switching between hallucination suppression and creativity enhancement.
3. VDAT (Visual Divergent Association Test). A new diagnostic benchmark that measures associative reasoning strength by prompting a model to generate nouns unrelated to the input image, scored with CLIP ViT-L/14 embeddings.
4. Comprehensive experiments. Evaluations across hallucination benchmarks (CHAIR, POPE), creativity benchmarks (VDAT, Creation-MMBench), and general-purpose benchmarks (MME, MMMU, MMStar) show improvements in both low- and high-association tasks without sacrificing general capability.

Main Findings

• Middle layers govern associative behavior. In LLaVA-1.5-7b, cosine distance between associative and non-associative features peaks in middle layers (layers 10–15), while shallow layers (0–9) show consistently low distance and Euclidean distance keeps growing across middle and deep layers (10–31).
• Layer intervention confirms the middle layers as the source, not just a relay. Replacing associative features with non-associative ones in layers 10–15 substantially reduced divergence in later layers; replacements in shallow layers (0–9) or deep layers (16–31) had limited downstream impact, suggesting deep layers propagate rather than generate associative features.
• A single coefficient controls both hallucination and creativity. Sweeping the steering coefficient α from −1.5 to 1.5 raised CHAIR from approximately 38.8 to 53.6 and VDAT from around 83 to 87.9, showing that higher α produces both more hallucination and stronger associative ability.
• Hallucinated responses yield usable control directions. The difference between hallucinated and grounded features provides the steering vector that guides associative reasoning.
• Hallucination results. On CHAIR_S, FlexAC reduced hallucination to 19.2 on Qwen-VL, 36.6 on LLaVA-1.5, and 28.6 on Deepseek-VL2 (versus 40.6, 50.8, and 32.6 for the regular models). On CHAIR_I it reached 5.4, 10.4, and 8.1 respectively, and achieved the highest POPE F1-score on LLaVA-1.5 (87.9). The abstract reports a 29% reduction in hallucination rate on CHAIR overall.
• Creativity results. FlexAC scored 86.58 (Qwen-VL), 88.49 (LLaVA-1.5), and 84.76 (DeepSeek-VL2) on VDAT. On Creation-MMBench it achieved the highest overall Reward (10.92) against VCD (−3.86) and VAF (−1.63), while maintaining competitive VFS (6.25 overall). The abstract reports up to a 5.8× improvement in creativity on Creation-MMBench.
• Baselines in the other direction lose creativity. Ha-DPO lowered hallucinations but reduced VDAT to 85.11 versus 86.89 for the regular LLaVA-1.5 model; Figure 1 cites a 14.0 improvement in CHAIR paired with a 1.78 drop in VDAT for existing methods.
• Components each matter. Ablating Instance Selection (IS) and Steering Intensity Calibration (SIC) worsened CHAIR_S from 19.2 to 30.4 and dropped VDAT from 86.58 to 85.05; removing Directional Integration (DI) produced a slight VDAT improvement, but DI is described as important for creativity.
• Middle-layer control beats shallow or deep control. In the layer-wise ablation, controlling middle layers gave the best CHAIR results for FlexAC-P and the best VDAT scores for FlexAC-C; layers 15, 16, and 17 were selected for Qwen-VL.
• General capabilities are preserved. Both FlexAC-P and FlexAC-C performed comparably to the vanilla model on MME, MMMU, and MMStar; FlexAC-C outperformed the baseline on MME's OCR task.
• VDAT aligns with human judgment. In a user study with 15 raters on 30 randomly selected Qwen-VL image-response pairs, over 70% of responses favored FlexAC and fewer than 6% favored the baseline.
• White-box requirement. FlexAC requires access to hidden states and is explicitly not applicable to black-box models such as ChatGPT.

Methodology in Plain English

The authors start by probing what happens inside a model when it hallucinates. Using 1,000 COCO2024 images and LLaVA-1.5-7b, they collect two kinds of responses per image: a grounded response from the model's default output and a hallucinated response induced with blurred images and specific prompts. Treating the hallucinated response as a proxy for high-association behavior, they extract hidden features layer by layer and measure cosine and Euclidean distances between the two kinds of features.

To check whether the observed late-layer differences are generated or merely inherited, they run a replacement experiment: they swap in a non-associative feature at a given layer and see how much the downstream layers change. The middle layers turn out to be the source.

They then confirm that the difference vector between hallucinated and grounded features can act as a steering signal. Adding a scaled version of this vector to middle-layer activations during inference moves both hallucination and creativity measures in a controllable way.

FlexAC builds on this in two phases. In the offline phase, the authors collect grounded-hallucinated feature pairs, keep the top-K pairs with the largest cosine distance, and average their differences into a general associative vector. They also generate a few high-association, task-aligned examples with GPT-4o to build task-specific vectors for multi-dimensional creative needs. In the inference phase, they add a weighted combination of the general and task-specific vectors to middle-layer hidden states. A Steering Intensity Calibration module sets the strength per sample using a sigmoid of the (clipped at zero) negative cosine between the current feature and the steering vector — stronger steering when the representation is misaligned with the associative direction, weaker when it is already aligned. The modulated feature is then rescaled to preserve its original norm.

Experiments use LLaVA-1.5, Qwen-VL, and Deepseek-VL2-tiny against Ha-DPO, VCD, and VAF. General vectors were built from 2,000 randomly selected COCO2014 images, narrowed down by Instance Selection to 50 images. Steering layers were Qwen-VL (15, 16, 17), LLaVA-1.5 (11, 12, 13), and Deepseek-VL (4, 5, 6). FlexAC-P (faithfulness) used α = −1 and FlexAC-C (creativity) used α = 1. All experiments ran on 8×RTX 4090 GPUs.

Why This Matters

The work reframes hallucination and creativity not as separate problems to be solved by separate tools, but as two settings of one underlying mechanism. If that framing holds, it means safeguards against hallucination do not have to come at the cost of imaginative capability, and vice versa — a single inference-time method can do both.

Real-world applications:

• Education and medical support, where factual precision is essential and associative drift is harmful.
• Storytelling and creative writing assistants, where image-grounded generation benefits from drawing distant, non-obvious associations.
• Event planning and ideation tools, cited in the paper as an example of a high-association task that existing hallucination-reduction methods degrade.
• Advertising and artistic image captioning, where metaphor and abstract themes are valued but output must still remain tied to the visual input.

Industry relevance: Because FlexAC is training-free and requires only an offline vector-construction step, it can be layered onto already-deployed open-weight MLLMs without retraining or fine-tuning. The ability to tune a single coefficient per request — rather than ship two separate models — is attractive for products that must serve both factual and creative use cases.

Future Directions

• Extending beyond white-box models. The authors state FlexAC cannot be applied to black-box models like ChatGPT; finding ways to approximate hidden-state steering without access to internal activations is an open problem.
• Reducing dependence on GPT-4o for task-specific vectors. The directional-integration step relies on GPT-4o to generate high-association, instruction-aligned examples, which raises questions about whether the technique can be fully self-contained.
• Scaling the validation. The paper evaluates on three MLLMs, with layer selection tested per model; whether the same middle-layer pattern and layer choices generalize to larger or differently architected models is not reported.
• Establishing the causal link between hallucination and creativity more broadly. The paper's shared-mechanism hypothesis is supported by its own experiments, but the paper does not report a wider study across architectures or association types, leaving room for follow-up work.

Target Audience

This paper is most useful to researchers and engineers working on multimodal large language model interpretability, activation steering, and inference-time intervention. It is also relevant to practitioners building applications that need controllable output behavior — either strictly factual or deliberately imaginative — from open-weight MLLMs, and to benchmark designers interested in measuring associative or divergent thinking, since VDAT is introduced here as a new diagnostic.

Authors’ abstract

Multimodal large language models (MLLMs) face an inherent trade-off between faithfulness and creativity, as different tasks require varying degrees of associative reasoning. However, existing methods lack the flexibility to modulate this reasoning strength, limiting MLLMs' adaptability across factual and creative scenarios. To bridge this gap, we propose equipping MLLMs with mechanisms that enable flexible control over associative reasoning. We begin by investigating the internal mechanisms underlying associative behavior in MLLMs and find that: (1) middle layers play a pivotal role in shaping model's associative tendencies, (2) modifying representations in these layers effectively regulates associative reasoning strength, and (3) hallucinations can be exploited to derive steering vectors that guide this modulation. Building on these findings, we introduce Flexible Association Control (FlexAC), a lightweight and training-free framework for modulating associative behavior in MLLMs. FlexAC first induces hallucination-guided intermediate representations to encode associative directions. Then, it selects high-association instances to construct effective associative steering vectors, whose strengths are adaptively calibrated to balance creative guidance with output stability. Finally, recognizing the multi-dimensional nature of associative reasoning, FlexAC incorporates task-specific associative vectors derived from a forward pass on a few target-domain samples, enabling models to follow diverse associative directions and better adapt to creative tasks. Notably, our method achieves up to a 5.8x improvement in creativity on Creation-MMBench and a 29% reduction in hallucination rate on CHAIR, surpassing existing baselines and demonstrating its effectiveness in enabling flexible control over associative reasoning in MLLMs. Our code is available at https://github.com/ylhz/FlexAC.