BitMar: Low-Bit Multimodal Fusion with Episodic Memory for Edge Devices

AI summary

Overview

Research area: Efficient multimodal vision-language modeling for edge devices, combining low-bit quantization, cross-modal fusion, and memory-augmented decoding.

Technical level: Intermediate. The paper assumes familiarity with transformer attention, quantization (BitNet-style ternary weights), and key-value memory networks, but explains each component clearly.

Scope: The paper introduces and evaluates BitMar, a 14M-parameter, 1.58-bit multimodal transformer that fuses quantized text and vision encoders with an external episodic memory and an attention-sink decoder, benchmarking it against larger low-bit language models and a suite of BabyLM evaluation tasks.

What This Paper Is About

Multimodal vision-language models such as BLIP-2, Flamingo, and Kosmos-2 produce strong grounded language, but their full-precision backbones are too heavy to run on edge hardware. Separately, memory-augmented transformers retrieve past context well but are rarely combined with aggressive quantization. BitMar's goal is to unify 1.58-bit multimodal encoding with an episodic memory system in a single small model that can generate image-text output under tight latency, energy, and memory budgets.

Key Contributions

1. Low-bit multimodal encoding framework. A unified architecture pairs a 1.58-bit quantized BitNet text encoder with a quantized ViT-based vision encoder (built on frozen DiNOv2 features), producing compact multimodal embeddings.

2. Memory-augmented decoding mechanism. A lightweight episodic memory module (K=512 slots, C=128 dimensions) retrieves contextual vectors and injects them into each transformer layer via per-layer conditioning, rather than only at the input.

3. Edge-efficient multimodal reasoning. The authors show the model achieves competitive results on binary reasoning and coreference while running at low latency and minimal memory footprint, with an attention-sink sliding-window mechanism (S=4 sink tokens, W=1020 window) for long or streaming inputs.

4. Joint training scheme with adaptive stabilization. A combined loss (language modeling, InfoNCE cross-modal at weight 1.5, memory consistency at weight 0.1) plus an adaptive controller that freezes one encoder or upweights the cross-modal loss when cross-modal cosine similarity drops by more than 0.12 from its recent maximum over a 200-step EMA.

Main Findings

• Model size and quantization. BitMar is a 14M-parameter model. Quantization effectiveness — the zero-weight fraction in ternary weights across BitNet-quantized layers — rises during training and stabilizes at 42.8%, indicating effective compression without degrading downstream performance.

• Language benchmarks (Table 1). BitMar scores 28.32 on ARC-Easy, 42.83 on BoolQ, 30.04 on HellaSwag, 54.57 on WinoGrande, 24.57 on CommonsenseQA, and 27.90 on MMLU. It is strongest relative to its size on BoolQ and WinoGrande, and lags larger low-bit models on ARC-Easy and HellaSwag. For comparison, BitNet b1.58 2B reaches 74.79, 80.18, 68.44, 71.90, 71.58, and 53.17 on the same six tasks.

• BabyLM evaluation (Table 2). BitMar averages 60.5% across finetuned NLP benchmarks, with the strongest results on QQP (70.2%), MRPC (69.1%), and BoolQ (66.5%), and weaker ones on MNLI (42.3%) and RTE (54.0%). Multimodal tasks score modestly (21–25%), with the best on EWoK (24.9%). BLIMP reaches 48.7% and compositional reasoning 51.5%.

• Memory activation evolution (Figure 2). Early in training, episodic memory slot activations are weak and scattered with little specialization; late in training they strengthen and differentiate, indicating selective storage of contextual features.

• Memory-on vs. memory-off efficiency (Table 3). With memory enabled, throughput is 57.3 tok/s versus 7.7 tok/s without; latency per token is 17.3 ms versus 129.8 ms; energy is 1.90 J versus 9.17 J; RAM is 956 MB versus 1,076 MB.

• Memory-on vs. memory-off task effects (Table 4). Gains appear on Entity Tracking (Split 1: +2.9; Split 2: +4.1), COMPS (+3.4), BLiMP (+0.6), and VQA (+3.4). Regressions appear on EWoK Split 1 (−1.6) and Winoground (−1.6); EWoK Split 2 improves by +1.0, and DevBench shows no effect.

• Reported regressions. WUG morphology correlations are negative under extreme quantization (−0.36 for adjectives and −0.16 for past tense, as stated in the ablation discussion). Reading alignment is lower with memory (0.44/0.11) than without (1.11/0.66), suggesting episodic conditioning can dampen psycholinguistic alignment.

• Fine-tuning impact. No significant changes appear on BoolQ, MNLI, MRPC, MultiRC, QQP, RTE, or WSC, suggesting the memory mainly affects generation rather than supervised heads.

• Ablation summary. The episodic memory is approximately 7.5× faster, uses 79% less energy, and 11% less VRAM in the authors' tests, delivering 3–4 percentage point gains on entity/property reasoning and multimodal QA.

Methodology in Plain English

The authors build a four-stage pipeline. First, two encoders convert inputs into compact 128-dimensional embeddings: a 4-layer, 128-dimension BitNet-style transformer for text (up to 256 tokens, ternary weights of {-1, 0, +1} with learned per-layer scales, and 8-bit activations scaled per token to [-127, 127]), and a vision path that reuses frozen DiNOv2 768-D patch features, performs 2×2 average pooling to cut patch count by 4×, then compresses to 128 dimensions through a two-layer MLP bottleneck. Second, a cross-modal fusion module uses standard cross-attention with 1.58-bit ternary projections (softmax, residual, and layer norm kept in FP32) to align text and vision into a shared latent space, then pools the result into a single query vector. Third, that query addresses a fixed-size episodic memory matrix (512 slots × 128 dimensions) using learned write weights, soft multi-slot writes at rate α=0.2, and content-based reading via softmax(M·q). A Frobenius penalty on changes to the store and usage-based forgetting prevent memory thrashing. Fourth, a 4-layer causal BitNet decoder conditions each layer on the retrieved memory vector (by concatenation or residual addition) and produces logits over the GPT-2 vocabulary (128 → 50,257). The decoder uses attention sinks with a sliding window so long or streaming inputs fit a fixed memory budget. Training ran on an NVIDIA A6000 with FP16 and gradient checkpointing, 64 sequences per step plus two-step accumulation (effective batch 128), AdamW8bit at 2×10⁻⁴ with cosine restarts (T₀=1000, T_mult=2, η_min=0.1·lr) for 10 epochs over 100M tokens — 50M multimodal from CC3M and Localized Narratives, and 50M text-only from BabyLM's six domains — with a 1M-token hold-out for tracking alignment and perplexity.

Why This Matters

Impact on research: The paper tests whether aggressive 1.58-bit quantization can be combined with memory augmentation and cross-modal fusion in a single sub-100M-parameter model. Its mixed results — competitive binary reasoning and coreference but weak knowledge-heavy and morphological performance — give a concrete data point about where extreme compression holds up and where it breaks, and the memory ablation quantifies the speed, energy, and VRAM savings that episodic conditioning can unlock.

Real-world applications:

• On-device image captioning and visual question answering where connectivity, latency, or privacy rules out cloud inference.
• Assistive and accessibility tools running on phones or wearables that need to describe scenes locally.
• Industrial or field robotics using small compute modules for real-time scene understanding.
• Streaming or long-context interfaces on edge hardware, where the attention-sink window plus optional SD-card offloading of memory keeps memory use bounded.

Industry relevance: The results speak directly to chipmakers, embedded-AI vendors, and product teams deciding whether sub-billion-parameter or even sub-100M-parameter multimodal models can replace larger cloud models for narrow tasks. The energy and VRAM numbers (79% less energy, 11% less VRAM from the memory ablation) are the kind of figures that drive deployment decisions for battery- and memory-constrained devices.

Future Directions

• Tuning memory capacity or the injection strategy to address the reported regressions in WUG morphology and psycholinguistic reading alignment.
• Closing the gap on knowledge-heavy benchmarks such as MMLU (27.90) and CommonsenseQA (24.57), where BitMar trails larger low-bit models by wide margins.
• Reconciling the architecture's stated dimensional scheme: the paper describes all modules operating in a "consistent 768-dimensional space" while also specifying that fusion, memory, and decoder paths operate in 128-D after compression.
• Investigating why episodic memory helps some multimodal and entity-reasoning tasks but hurts EWoK Split 1 and Winoground, and whether the per-layer conditioning method (concatenation versus residual addition) changes that balance.

Target Audience

Researchers and engineers working on edge AI, tiny vision-language models, and quantization-aware training will get the most from this paper, particularly those interested in how far 1.58-bit precision can be pushed when combined with external memory. It is also useful for practitioners evaluating whether a 14M-parameter multimodal model can substitute for larger backbones in constrained deployments, and for students studying memory-augmented neural networks applied to multimodal generation.

Authors’ abstract

Cross-attention transformers and other multimodal vision-language models excel at grounding and generation; however, their extensive, full-precision backbones make it challenging to deploy them on edge devices. Memory-augmented architectures enhance the utilization of past context; however, most works rarely pair them with aggressive edge-oriented quantization. We introduce BitMar, a quantized multimodal transformer that proposes an external human-like episodic memory for effective image-text generation on hardware with limited resources. BitMar utilizes 1.58-bit encoders, one for text (BitNet-style) and one for vision (DiNOv2-based), to create compact embeddings that are combined and used to query a fixed-size key-value episodic memory. During vector retrieval, the BitNet decoder applies per-layer conditioning, which increases the contextual relevance of generated content. The decoder also employs attention sinks with a sliding-window mechanism to process long or streaming inputs under tight memory budgets. The combination of per-layer conditioning and sliding-window attention achieves a strong quality-speed trade-off, delivering competitive captioning and multimodal understanding at low latency with a small model footprint. These characteristics make BitMar well-suited for edge deployment.