Text-Only Training for Image Captioning with Retrieval Augmentation and Modality Gap Correction

AI summary

Overview

Research area: Image captioning and vision-language modeling, specifically zero-shot and text-only training methods, retrieval-augmented generation, and cross-modal (CLIP-style) embedding alignment.

Technical level: Advanced. The paper assumes familiarity with CLIP/SigLIP contrastive encoders, cross-attention decoding, LoRA, FAISS nearest-neighbor search, and captioning metrics such as CIDEr and METEOR.

Scope: The paper introduces and empirically evaluates TOMCap, a text-only captioning framework that trains a language model decoder without any paired image-caption data, using retrieval-augmented prompting plus a mean-and-standard-deviation correction to reduce the CLIP modality gap.

What This Paper Is About

Image captioning models normally need large sets of human-written image-caption pairs, which are expensive to collect. Text-only methods sidestep this by training a decoder on captions alone, then swapping in image embeddings at inference — but this works poorly because image and text embeddings from contrastive encoders like CLIP occupy different regions of the same vector space (the "modality gap"). This paper proposes TOMCap, which combines retrieved similar captions, latent embedding guidance through cross-attention, and a correction that re-aligns the means and standard deviations of image and text embeddings.

Key Contributions

1. A unified text-only training framework (TOMCap) that combines retrieval-augmented generation, cross-attention over CLIP/SigLIP latent embeddings, and an explicit modality gap correction, using Low-Rank Adaptation (LoRA) so that the encoder and decoder stay frozen.
2. An extended modality gap correction that goes beyond prior mean-subtraction approaches by also rescaling with per-dimension standard deviations of the image and text embedding distributions.
3. A training target substitution strategy where the closest retrieved caption (rather than a ground-truth caption) is used as the supervision target, with retrieval performed over a datastore of approximately 16 million captions from MSCOCO, CC3M, and CC12M.
4. Extensive ablations on the retrieval and correction components, covering the number of retrieved captions (K), two separate noise injection magnitudes (B and L), caption ordering in the prompt, and diversity-based re-ranking.

Main Findings

• TOMCap beats prior training-free and most text-only methods: With a GPT2-base decoder, TOMCap reaches BLEU@1 72.7, BLEU@4 28.4, METEOR 25.8, and CIDEr 103.4 on MSCOCO. On NoCaps validation, it reports 76.3 (In), 75.5 (Near), 75.7 (Out), and 76.2 (Overall) CIDEr, outperforming compared text-only methods such as CapDec (CIDEr 91.8), ViECap (92.9), EntroCap (94.3), and CLOSE (97.8).
• A larger decoder improves results: TOMCap with GPT2-large reaches BLEU@1 73.9, BLEU@4 30.2, METEOR 26.6, CIDEr 108.3 on MSCOCO, and 76.4 overall CIDEr on NoCaps.
• Retrieval is the most impactful component: The ablated "embedding only" variant (cross-attention but no retrieved captions in the prompt) scores CIDEr 76.6, versus 103.4 for the full model. The "retrieval only" variant (no cross-attention) scores CIDEr 101.6, and the no-training retrieval version scores CIDEr 15.2, BLEU@1 30.4, BLEU@4 8.6, METEOR 8.4. The paper notes the BLEU@1 value for the "embedding only" variant is not clearly reported in the extracted table.
• Modality gap correction is critical: Without any correction, CIDEr is 79.3 (BLEU@1 67.1, BLEU@4 19.8, METEOR 22.4). Mean-only correction raises this to 102.2 CIDEr, and joint mean plus standard deviation correction to 103.4.
• Retrieval noise level L = 0.1 is best but noise is optional: CIDEr is 103.4 at L = 0.1, 103.0 at L = 0 (no noise), 101.5 at L = 0.125, 93.7 at L = 0.5, and 93.4 at L = 1.
• Decoder noise magnitude B has minor impact: CIDEr ranges from 101.9 (B = 0) to 103.4 (B = 0.125), a spread of 1.5 points across the tested values.
• More retrieved captions give diminishing returns: Moving from K = 4 (CIDEr 103.4) to K = 8 (104.2) yields an improvement of only 0.8 CIDEr. Training with K = 4 and inferring with K = 6 gives CIDEr 105.6, and training with K = 6 while inferring with K = 8 gives 105.4.
• Caption ordering in the prompt has negligible effect: CIDEr is 103.4 for decreasing similarity order, 103.2 for increasing order, and 103.9 for random order, suggesting GPT-2's positional embeddings do not introduce meaningful bias here.
• Re-ranking for diversity hurts: Applying Maximal Marginal Relevance with λ = 0.5, 0.2, −0.2, and −0.5 produces CIDEr scores of 90.3, 92.7, 92.6, and 93.5, all below the default λ = 0 setting of 103.4.
• KNOR analysis supports the correction: Using 2,500 MSCOCO validation captions and k ∈ {5, 10, 15, 50, 100}, the correction performs similarly to no correction at small k but surpasses it as k grows, indicating improved local geometry preservation.
• TOMCap still trails supervised captioning: The authors state that the model's scores "significantly lag behind those of specialized models that require image-caption pairs for supervised training," though no supervised baseline numbers appear in the reported table.

Methodology in Plain English

The system has three parts. First, an encoder: a frozen SigLIP2 L/16 model turns both captions and images into 1024-dimensional vectors. Second, a correction step: because image vectors and text vectors from this kind of encoder sit in slightly different regions of the space, the method rescales text vectors so their per-dimension mean and standard deviation match those of image vectors, then injects small amounts of Gaussian noise to soften the remaining mismatch. Third, a decoder: a GPT2-base (or GPT2-large) language model is given a prompt listing captions of similar items — "Similar images have the following captions: {caption1} ... {captionk}. Write a caption for this image:" — and is extended with cross-attention layers that read the corrected embedding vectors directly.

During training, the model works entirely from text. Captions are encoded and indexed in a FAISS datastore of roughly 16 million captions; for each training caption, the system retrieves the K+1 most similar datastore captions, uses the top one as the target to predict, and places the other K in the prompt. Only the new cross-attention layers and LoRA adapters (rank 32, α = 32, rsLoRA variant) on the decoder's attention projections are trained; CLIP and GPT2 weights stay frozen. Optimization uses AdamW with learning rate 1e-4 and batch size 32, running up to 10 epochs with early stopping over a 5% MSCOCO training split used for validation, evaluating every 2048 steps. Training takes up to 6 hours on a single NVIDIA RTX 6000 GPU with up to 32 GB of memory. At inference, an image is encoded, corrected, used to retrieve K captions, and the decoder generates a caption via deterministic beam search with beam size five. The image-side statistics for the correction are computed from Flickr30k, and the text-side statistics from the datastore itself, so no paired image-caption data is needed anywhere.

Why This Matters

Research impact: The paper shows that the CLIP modality gap is not just a nuisance but a measurable bottleneck for text-only captioning, and that a simple mean-and-standard-deviation correction plus retrieval recovers most of the lost performance. It also gives a clean set of negative results — diversity re-ranking and prompt ordering barely help, and extra retrieval noise actively hurts — that future work can build on.

Real-world applications:

• Automatic alt-text for accessibility, where paired training data for a specific domain or language may not exist.
• Metadata and catalog generation for e-commerce, stock photography, and media archives, where captions must be produced at scale.
• Domain adaptation for specialized imagery (medical, industrial, remote sensing) where annotated caption pairs are scarce but unpaired text descriptions exist.
• Multilingual captioning for less-resourced languages, where the method's independence from image-text pairs makes it possible to retrain only the text side.

Industry relevance: The approach avoids the most expensive input in captioning pipelines — human-written image-caption pairs — while keeping the encoder and decoder frozen and training only small LoRA and cross-attention modules. This lowers compute and data-annotation costs, allows swapping in newer frozen encoders or decoders without full retraining, and reduces the risk of catastrophic forgetting of the language model's general linguistic ability.

Future Directions

1. Relax the independence assumption. The correction treats each embedding dimension as an independent Gaussian, which the authors acknowledge does not fully hold. Modeling cross-dimension covariance could close more of the modality gap.
2. Move beyond English. All experiments use English captions; the authors plan to train TOMCap in other languages and hypothesize it is particularly promising for less-resourced languages, possibly by translating existing English datasets such as MSCOCO or the Conceptual Captions datasets.
3. Expand the datastore to reduce bias. Broader datastore coverage is proposed as a way to mitigate dataset-specific biases without needing exact image-caption pairs.
4. Evaluate on the NoCaps test set. Results are reported only on the NoCaps validation set because the online benchmark was unavailable at the time of writing; the 10,600-image test split remains unevaluated.

Target Audience

Researchers and graduate students working on image captioning, vision-language pretraining, or zero-shot multimodal generation; practitioners who need captioning in domains or languages without paired image-text data; and engineers interested in parameter-efficient fine-tuning (LoRA, cross-attention adapters) and retrieval-augmented generation applied to multimodal tasks.

Authors’ abstract

Image captioning has drawn considerable attention from the natural language processing and computer vision fields. Aiming to reduce the reliance on curated data, several studies have explored image captioning without any humanly-annotated image-text pairs for training, although existing methods are still outperformed by fully supervised approaches. This paper proposes TOMCap, i.e., an improved text-only training method that performs captioning without the need for aligned image-caption pairs. The method is based on prompting a pre-trained language model decoder with information derived from a CLIP representation, after undergoing a process to reduce the modality gap. We specifically tested the combined use of retrieved examples of captions, and latent vector representations, to guide the generation process. Through extensive experiments, we show that TOMCap outperforms other training-free and text-only methods. We also analyze the impact of different choices regarding the configuration of the retrieval-augmentation and modality gap reduction components.