InfoCIR: Multimedia Analysis for Composed Image Retrieval

AI summary

Overview

Research area: Human-Computer Interaction / Visual Analytics, applied to multimodal (image + text) information retrieval.

Technical level: Intermediate. Readers benefit from familiarity with CLIP-style vision-language embeddings, dimensionality reduction (UMAP), and retrieval metrics, but the system design and user-study findings are described at an accessible level.

Scope: The paper presents InfoCIR, an interactive six-panel visual analytics dashboard that unifies composed image retrieval, explainability, and LLM-driven prompt enhancement into a single workflow, and reports a controlled user study comparing it against a stripped-down baseline interface.

What This Paper Is About

Composed Image Retrieval (CIR) lets a user search for images by pairing a reference image with a text prompt describing a desired change (for example, an image of a forest plus the prompt "in autumn colors"). While models such as CLIP make this possible, developers currently diagnose failures by blind trial-and-error: they see a low Recall@K and tweak prompts without knowing why the model ignored the text or returned the wrong images. The goal of InfoCIR is to close that gap by giving users one dashboard where they can compose queries, see how results sit in the embedding space, inspect which image regions and prompt words drove the ranking, and test LLM-generated prompt variants with immediate visual feedback.

Key Contributions

1. A unified visual analytics architecture integrating retrieval, attribution, and prompt engineering into a "diagnose-compare-enhance" feedback loop, replacing isolated blind trial-and-error workflows.
2. An interactive relevance-feedback workflow that couples LLM-generated prompt variants (using mistralai/Mistral-7B-Instruct-v0.2) with visual verification via saliency maps, token-level attribution, and a novel Ideal-anchored Rank-Δ heatmap.
3. A supervised metric projection pipeline using a debiased UMAP approach — the Class Projection Preservation Pipeline with style debiasing (PCA-based filtering), contrastive debiasing, and Independent Component Analysis (ICA) — designed to overcome style bias in CLIP-based embedding spaces.
4. A controlled user study with eight Master's students in AI showing that InfoCIR significantly improves retrieval efficiency and success rates compared to a baseline CIR interface.

Main Findings

• Success rate nearly doubled: Participants using the full InfoCIR interface placed the target image in the top-3 in 87.50% of cases, versus 37.50% with the baseline interface.
• Faster completion: Average time on task fell from 277 seconds (baseline) to 133 seconds (InfoCIR).
• Fewer query attempts: The average number of unique queries submitted dropped from 7.4 to 3.3.
• One participant failed in both conditions: P6 did not succeed in either condition because the initial provided prompts were overly generic (the paper's text is truncated at this point).
• Style bias is a real obstacle: The authors observe that standard UMAP on CLIP embeddings can cluster images by artistic style rather than semantically relevant categories, which is why they introduce the Class Projection Preservation Pipeline.
• Textual conditions can be ignored: The authors report that standard saliency maps fail to reveal when a retrieval model ignores the text condition in favor of visual dominance — for example, retrieving a "green apple" despite the prompt specifying "red."
• Weak token influence is diagnosable: In the usage scenario, the attribution panel showed that the term "red" contributed weakly to the ranking; after switching to the variant "crimson apple," the selected image moved from rank 12 to rank 2.
• Generic embedding tools are insufficient: Existing embedding projectors treat queries as static points and fail to visualize the vector shift induced by the textual modifier, which the authors identify as the core mechanism of CIR.

Methodology in Plain English

The researchers built a system rather than a new model. InfoCIR wraps an existing zero-shot CIR back-end, SEARLE, which converts a reference image into a pseudo-token inside CLIP's text embedding space, fuses it with the text prompt into one joint query vector, and then ranks database images by cosine similarity.

On top of that retrieval engine, they built a six-panel interface using Plotly and Dash:

• Panel A for inputting a reference image, a text prompt, and the number of results k.
• Panel B for viewing top-k results in a similarity-sorted gallery, where any image can be clicked to mark it as an ideal target.
• Panel C for a class-frequency histogram (C1) and a word cloud (C2) summarizing labels in the current top-k.
• Panel D, the central Embedding View, showing a 2D UMAP projection of the dataset with the reference image, composed query embedding, and top-k results highlighted.
• Panel E for LLM-generated alternative prompts conditioned on the user-selected ideal images.
• Panel F for explanations: a saliency map (F1), a token attribution bar chart (F2), and a Rank-Δ heatmap (F3).

All panels are linked, so clicking an image in one view updates the others.

For UMAP, the authors chose it over t-SNE for three reasons: parametric stability (it can transform unseen data without re-running the projection), better global structure preservation, and denser, more compact clusters that support rapid semantic verification. Before projection, they apply the Class Projection Preservation Pipeline to reduce style-driven overlap between classes.

For explanations, they compute the gradient of the similarity score with respect to image pixels (an adaptation of Grad-ECLIP implemented from scratch for SEARLE's CLIP-based encoder) and with respect to each prompt token. The token gradient magnitudes become the attribution bar chart.

For evaluation, eight Master's students in AI from the University of Amsterdam completed two retrieval tasks on ImageNet-R: retrieving a stylized (cartoon-like) version of a realistic reference image. Task A used a porcupine and Task B a Boston Terrier, chosen for comparable visual complexity. Participants had a 5-minute limit per task and each query returned ten images. Two conditions were compared: a Baseline using only Panels A and B, and the full InfoCIR interface. Groups completed the tasks in reversed order to avoid order bias (Group A: Task A Baseline then Task B InfoCIR; Group B: Task B Baseline then Task A InfoCIR).

Why This Matters

Impact on research. The paper argues that InfoCIR is the first system to jointly support analysis of embedding space, query dynamics, and model behavior for CIR. It shows that token-level attribution plus rank-change visualization can close the causality loop between which result moved and which word caused the move — something earlier rank-visualization tools like RankExplorer and Colorslope did not do.

Real-world applications.

• E-commerce and fashion search: Shoppers searching for "this jacket but in navy" benefit from tools that reveal why a text modifier was ignored by the retrieval model.
• Digital asset and stock photo management: Editors refining searches for a scene with specific attributes (season, time of day, mood) can iteratively verify prompt impact rather than guessing.
• Model debugging during development: Researchers diagnosing low Recall@K metrics can see structural outliers, over-represented classes, and weak token contributions instead of randomly tweaking inputs.
• Human-in-the-loop content moderation or curation: Reviewers needing precise semantic control over retrieved images can use the ideal-image anchoring and Rank-Δ feedback.

Industry relevance. The modular Plotly-Dash architecture lets new retrieval models, datasets, and attribution methods be plugged in as long as they expose embeddings in CLIP's space, which lowers the integration cost for teams already using CLIP-based pipelines. The reported efficiency gains — roughly halving completion time and reducing query attempts from 7.4 to 3.3 — speak directly to the productivity of practitioners working with these models. Source code for a reproducible demo is available at https://github.com/giannhskp/InfoCIR.

Future Directions

• Broader and larger user studies. The quantitative evaluation used eight Master's students in AI on two tasks; whether these gains hold for larger, more diverse participant pools and for non-expert users is not established.
• Generalizing beyond SEARLE and ImageNet-R. The paper demonstrates the system with one CIR back-end and one dataset. The modular design invites testing with other CIR models and domains, but that remains future work.
• Formal validation of the debiasing pipeline. The Class Projection Preservation Pipeline is explicitly framed as a heuristic, task-driven semantic pre-alignment strategy rather than model training or adaptation, leaving open how robustly it removes style bias across datasets.
• Addressing the cold-start problem systematically. The paper handles zero-hit scenarios by having users manually adjust the initial modifier in Panel A, which suggests an opportunity for more automated or guided recovery when the initial query returns nothing relevant.

Target Audience

This paper is most useful for HCI and visual analytics researchers building interactive tools for model inspection, machine learning engineers and practitioners working with CLIP-based retrieval systems who need to debug prompt sensitivity, and developers of composed image retrieval applications who want a working template. Researchers interested in explainable AI for multimodal systems, and anyone designing counterfactual or relevance-feedback interfaces for search, will also find the Rank-Δ heatmap and token attribution design directly applicable.

Authors’ abstract

Composed Image Retrieval (CIR) allows users to search for images by combining a reference image with a text prompt that describes desired modifications. While vision-language models like CLIP have popularized this task by embedding multiple modalities into a joint space, developers still lack tools that reveal how these multimodal prompts interact with embedding spaces and why small wording changes can dramatically alter the results. We present InfoCIR, a visual analytics system that closes this gap by coupling retrieval, explainability, and prompt engineering in a single, interactive dashboard. InfoCIR integrates a state-of-the-art CIR back-end (SEARLE arXiv:2303.15247) with a six-panel interface that (i) lets users compose image + text queries, (ii) projects the top-k results into a low-dimensional space using Uniform Manifold Approximation and Projection (UMAP) for spatial reasoning, (iii) overlays similarity-based saliency maps and gradient-derived token-attribution bars for local explanation, and (iv) employs an LLM-powered prompt enhancer that generates counterfactual variants and visualizes how these changes affect the ranking of user-selected target images. A modular architecture built on Plotly-Dash allows new models, datasets, and attribution methods to be plugged in with minimal effort. We argue that InfoCIR helps diagnose retrieval failures, guides prompt enhancement, and accelerates insight generation during model development. All source code allowing for a reproducible demo is available at https://github.com/giannhskp/InfoCIR.