VLF-MSC: Vision-Language Feature-Based Multimodal Semantic Communication System

AI summary

Overview

Research area: Semantic communication (SC) for wireless networks, combining computer vision, vision-language models (VLMs), large language models (LLMs), and diffusion-based generative models.

Technical level: Advanced. The paper assumes familiarity with transformer architectures (ViT, Q-Former, U-Net), latent diffusion models, channel models (Rayleigh fading, AWGN), and semantic communication metrics (BLEU, BERT score, LPIPS, CLIP score).

Scope: The paper proposes and evaluates VLF-MSC, a system that transmits one compact vision-language feature over a noisy wireless channel and uses it to generate both a caption and an image at the receiver, comparing it against three modality-specific baselines.

What This Paper Is About

Conventional semantic communication systems handle one modality at a time: text-based systems miss fine-grained visual detail, image-based systems need high bandwidth, and any change in what the receiver wants forces a separate transmission. This paper asks whether a single, modality-agnostic feature extracted by a pre-trained vision-language model can serve as the sole context for both text and image generation at the receiver, eliminating modality-specific streams and retransmissions while remaining robust to channel noise.

Key Contributions

1. A single transmitted feature, two decoders. The transmitter extracts one Vision-Language Feature (VLF) from the source image using a vision-language model, transmits it over the channel, and uses the received noisy feature as the only context for both a decoder LLM (text) and a diffusion-based image decoder (image), removing the need for modality-specific streams or retransmissions.

2. A resolution-invariant transmit budget. The transmitter maps the VLF directly onto transmitted complex symbols, with no autoregressive token generation and no pixel- or patch-domain source-channel coding, and the receiver performs prompt-free generation. Because the number of channel uses is fixed by the feature dimensions, it does not grow with pixel count.

3. Demonstrated multimodal superiority over modality-specific baselines. VLF-MSC is reported to achieve higher perceptual fidelity and stronger image-text semantic alignment than Img2Img-SC, and to preserve textual meaning more faithfully than text-centric baselines at low SNR.

4. A communication-centric design that avoids auxiliary prompts. Unlike BLIP-Diffusion, which can use a subject text to improve generation, VLF-MSC deliberately omits text prompts at inference because transmitting an extra subject description would add bandwidth overhead and latency.

Main Findings

• Image transmission beats the image baseline at low SNR: VLF-MSC achieves lower LPIPS (better perceptual similarity) and higher CLIP score (better semantic alignment) than Img2Img-SC across SNRs from −5 dB to 5 dB. The paper attributes this to converting the image into a VLF and transmitting it through the text modality rather than feeding image latents directly to a decoder.

• Qualitative image robustness: In the qualitative comparison, main objects such as an airplane and a boat remain faithfully conveyed at low SNRs.

• Text: better semantics, weaker lexical overlap: DeepSC performs better on BLEU, which measures n-gram overlap, because it uses a token-level reconstruction strategy. VLF-MSC consistently outperforms both DeepSC and the ASCII + (7,4) Hamming code baseline on BERT score, which measures semantic similarity, across all tested SNRs.

• Joint modality behavior across noise levels: In the appendix results, a single VLF transmitted once conditions both decoders, and as SNR increases from 0 dB to 10 dB (columns at 0, 2.5, 5, and 10 dB) perceptual fidelity improves while object identity and scene semantics remain stable even at low SNR.

• Bandwidth compression: Under the paper's BCR definition, BCR = N·d / (2C·h·w), so with N = 32 and d = 768 the BCR for a 256×256 RGB image is 1/16, decreasing as resolution increases. The experimental-details section separately states a BCR of 1/8 for a 256×256 input. Img2Img-SC's combined BCR is about 1/11 and remains roughly constant across image resolutions.

• No numeric metric values are reported in text: LPIPS, CLIP score, BLEU and BERT score results appear only in Figures 2 and 3 as curves; no tabulated numbers are given for any method.

Methodology in Plain English

The system has three parts: transmitter, channel, and receiver.

Transmitter. An input image of size 3 × H × W is passed through a frozen Vision Transformer image encoder (EVA-CLIP ViT-g/14) that splits the image into patches and produces patch-level feature vectors. A Q-Former — the same design used in BLIP-2 — uses 32 learnable query tokens of dimension 768 to probe those image features through cross-attention, producing 32 refined query embeddings. This set of embeddings is the VLF, and it is much smaller than the raw image or the full set of patch embeddings. The transmitter writes the VLF directly onto analog complex symbols.

Channel. The VLF is modeled as a sequence of N·d/2 complex symbols. The channel is Rayleigh fading with additive white Gaussian noise, so the received feature is h·y + n, where h is an element-wise channel gain and n is circularly symmetric Gaussian noise. The effective SNR is P̄·E[|h|²]/σ², which simplifies to P̄/σ² when E[|h_i|²] = 1.

Receiver. The noisy VLF drives two decoders. The text decoder average-pools the received queries, projects them into the LLM embedding space as e₀ = W_proj((1/N) Σ ỹ_i), and prepends that vector as a soft prompt to a decoder-based LLM (OPT-6.7B), which then autoregressively generates tokens until an end-of-sequence token. The image decoder is a VLF-conditioned latent diffusion model built on Stable Diffusion v1.5, inspired by BLIP-Diffusion but without text prompts at inference: the received VLF is projected into the CLIP text-encoder embedding space, and the resulting sequence conditions a U-Net through cross-attention at each denoising step.

Training. The diffusion model is trained to minimize the mean squared error between the noise predicted by the conditional U-Net and the true noise, using the simplified latent diffusion objective, and is conditioned on both the VLF and a subject label that is tokenized and processed by the CLIP text encoder. Fine-tuning randomly samples DDPM training steps from [0, 100], and inference uses 50 DDIM steps.

Evaluation setup. Experiments use the Open Image V6 dataset with all images resized to 256×256, over an AWGN channel with SNR from −5 dB to 10 dB in 2.5 dB increments. Baselines are Img2Img-SC (image), DeepSC (text), and ASCII + (7,4) Hamming code with 16-QAM modulation (text); both text baselines receive BLIP-2-generated captions as their input. Text is scored with BLEU and BERT score (both 0 to 1, higher is better), and images with LPIPS (lower is better) and CLIP score (higher is better).

Why This Matters

Impact on research. The paper argues for shifting semantic communication from modality-specific pipelines toward a unified vision-language representation, showing that one transmitted feature can serve two generative decoders. It also reframes semantic communication as task-oriented compression in which the receiver regenerates the desired modality rather than reconstructing bits, and it positions pre-trained foundation models as the mechanism for noise robustness.

Real-world applications:

• Augmented and virtual reality, where a receiver may need either a caption or a rendered image from the same transmission.
• Autonomous driving, where bandwidth is limited and channel conditions vary sharply.
• Massive Internet of Things networks, where many devices must send visual information under tight spectral budgets.
• Any deployment where the receiver's required output modality changes over time and retransmission overhead is unacceptable.

Industry relevance. The design keeps the transmitter lightweight by avoiding autoregressive token generation and by using frozen encoders, while relying on receiver-side foundation-model priors for reconstruction. Because the per-image transmit budget does not scale with pixel count, the framework claims higher spectral efficiency for high-resolution content — attractive to operators and device makers who must trade bandwidth against perceptual quality.

Future Directions

• More fine-grained ablation studies on the number of transmitted queries and feature dimensions, since these directly determine both bandwidth consumption and semantic quality.
• Exploring bandwidth-quality trade-offs across varying image resolutions and message complexities to characterize scalability.
• Extending the reported performance gains to higher-SNR regimes.
• Optimizing the framework for real-time operation and multi-user scenarios to advance practical deployment.

Target Audience

Researchers and graduate students working on semantic communication, joint source-channel coding, and generative wireless systems; engineers building bandwidth-constrained multimodal transmission pipelines; and practitioners interested in applying pre-trained vision-language, language, and diffusion foundation models to communication problems. Readers without a background in channel modeling, transformer architectures, or diffusion models will find the technical sections demanding.

Authors’ abstract

We propose Vision-Language Feature-based Multimodal Semantic Communication (VLF-MSC), a unified system that transmits a single compact vision-language representation to support both image and text generation at the receiver. Unlike existing semantic communication techniques that process each modality separately, VLF-MSC employs a pre-trained vision-language model (VLM) to encode the source image into a vision-language semantic feature (VLF), which is transmitted over the wireless channel. At the receiver, a decoder-based language model and a diffusion-based image generator are both conditioned on the VLF to produce a descriptive text and a semantically aligned image. This unified representation eliminates the need for modality-specific streams or retransmissions, improving spectral efficiency and adaptability. By leveraging foundation models, the system achieves robustness to channel noise while preserving semantic fidelity. Experiments demonstrate that VLF-MSC outperforms text-only and image-only baselines, achieving higher semantic accuracy for both modalities under low SNR with significantly reduced bandwidth.