MacVQA: Adaptive Memory Allocation and Global Noise Filtering for Continual Visual Question Answering

AI summary

Overview

• Research area: Multimodal machine learning — specifically continual learning for Visual Question Answering (VQA), combining computer vision, natural language processing, and lifelong learning.
• Technical level: Advanced. The paper assumes familiarity with transformer-based multimodal encoders (VL-T5), Faster R-CNN region features, denoising autoencoders, prototype memory banks, and continual-learning metrics such as Average Performance (AP) and Average Forgetting (AF).
• Scope: The paper proposes MacVQA, a framework combining Global Noise Filtering and Adaptive Memory Allocation, and evaluates it on ten sequential VQA tasks built from the VQA v2 benchmark.

What This Paper Is About

Continual VQA systems must learn new question types one after another without forgetting earlier ones, and existing methods struggle to simultaneously retain old knowledge, adapt to new tasks, and keep feature representations clean. The authors build a framework that both denoises fused visual-textual features and manages a prototype memory pool adaptively, aiming to balance knowledge acquisition, retention, and compositional generalization. They evaluate the result across ten continual VQA tasks derived from VQA v2.

Key Contributions

1. Global Noise Filtering (GonF): A module that scores image regions, builds a global feature vector by weighted summation, denoises region features with a Denoising Autoencoder (DAE), and fuses the global and denoised features into enhanced visual representations.
2. Adaptive Memory Allocation (AMA): A module that projects fused features into visual and textual subspaces, retrieves top-k prototypes by cosine similarity, fuses them via a gating mechanism, and updates the memory pool using a temporal interpolation rule.
3. A joint training objective: A combined loss weighting GonF, AMA, and decoder losses (with φ₁ + φ₂ + φ₃ = 1), plus a decoder that fuses modality-specific embeddings through cross-attention to generate answers.
4. A ten-task evaluation on VQA v2: Reporting 43.38% AP and 2.32% AF on standard tasks, and 42.53% AP and 3.60% AF on novel composition tasks, along with ablations, prototype-selection comparisons, memory-size sweeps, and hyperparameter sensitivity analysis.

Main Findings

• Standard test performance: MacVQA reaches 43.38% AP and 2.32% AF, outperforming all baselines including Vanilla, EWC, MAS, ER, DER, VQACL, QUAD, and ProtoGroup. It achieves the highest accuracy on judgment (66.64%), action (58.13%), and color (56.95%) tasks.
• Novel composition performance: 42.53% AP and 3.60% AF. Top results include commonsense reasoning at 67.05%, subcategory at 47.21%, and causal reasoning at 17.00%.
• Ablation results: Starting from the VQACL baseline (38.77% AP, 2.90% AF), GonF alone yields 41.75% AP and 2.14% AF, AMA alone yields 40.97% AP and 2.34% AF, and the full combination yields 43.38% AP (+3.57) and 2.32% AF (−0.58). The paper states GonF enhances perception tasks (recognition, location) while AMA supports memory-sensitive reasoning (color, typicality).
• Prototype selection matters: Max-Similarity selection outperforms Random selection. MacVQA with Max-Similarity reaches 43.38% AP / 2.32% AF (standard) and 42.53% AP / 3.60% AF (novel), versus 40.29% AP / 3.46% AF and 41.25% AP / 3.76% AF with Random. ProtoGroup with Random scores 37.22% AP / 3.89% AF (standard) and 35.78% AP / 4.54% AF (novel); with Max-Similarity, 39.81% AP / 2.87% AF and 36.81% AP / 4.09% AF. Top-3 prototype retrieval by similarity is credited with retrieving semantically aligned exemplars.
• Memory capacity: At the largest buffer size tested (5000), MacVQA achieves 43.38% AP and 2.32% AF on the standard test and 42.53% AP with 3.60% AF on the novel test, which the paper describes as outperforming the best baseline by +3.57% AP and −0.52% AF. Note that the paper reports the forgetting improvement as −0.52% AF in the memory-capacity section while the ablation table reports −0.58; the text is internally inconsistent on this figure. VQACL is described as showing rapid AP gains at smaller memory sizes but fluctuating AF, QUAD as consistently exhibiting high AF, and ProtoGroup as yielding modest gains.
• Hyperparameter sensitivity: Performance preferences are task-specific. Visual-dominant tasks (recognition, location) peak at α = 1.0, β = 0.4 with 43.53% AP, while question-dominant tasks (commonsense, judgment) peak at 48.5% with α = 0.4, β = 1.0.
• Qualitative robustness: In the location task, MacVQA answers "in the chair" for a standard image and "in the bed" for a novel object-reasoning combination, while baseline methods are reported to fail in both settings.

Methodology in Plain English

The framework processes every image-question pair through three stages. First, a pre-trained Faster R-CNN extracts n region features per image (embedding dimension d, typically 2048), while the question is encoded into word embeddings. A scoring and softmax step assigns an attention weight to each region; these weights are used both to build one global summary vector and to guide a Denoising Autoencoder that cleans up the individual region features. The cleaned regions are combined with the global vector and passed with the question embeddings into a Transformer-based multimodal encoder. The training loss for this stage combines reconstruction error with an entropy term on the attention weights, so the model learns to denoise while keeping attention spread out.

Second, the fused feature is split into visual and textual subspaces through learned projection matrices. Each is compared by cosine similarity against stored visual and textual prototypes, and the top-k matches are pulled out. A gating vector decides how much weight to give the current input versus the retrieved prototypes, and the memory pool is updated by interpolating between the old memory state and the current feature (controlled by λ). Regularization terms keep the gate weights summing to one and limit how far the fused feature drifts from the original.

Third, the fused representation is projected into visual and question embeddings, concatenated, and passed through a cross-attention decoder trained with negative log-likelihood over the ground-truth answer tokens. All three module losses are combined with weights φ₁, φ₂, φ₃ that sum to 1. Experiments use a shared pretrained VL-T5 backbone, Adam with a 3e-5 learning rate, gradient clipping of 5, and a 0.1 warmup ratio.

Why This Matters

Impact on research: The paper targets a specific tension in multimodal continual learning — that regularization methods struggle with fine-grained cross-modal interactions while rehearsal methods depend on stored data and scale poorly. By pairing feature-level denoising with prototype-based memory, it proposes a middle path and provides a full ablation isolating the contribution of each component, plus a systematic comparison of prototype selection strategies that prior VQA continual-learning work (VQACL, QUAD, ProtoGroup) does not foreground.

Real-world applications cited by the paper:

• Autonomous driving, where vehicles analyze scenes through visual queries.
• Medical diagnosis, assisting professionals in interpreting imaging by answering questions about abnormalities.
• Accessibility tools for visually impaired individuals.
• Intelligent human-computer interaction systems.

Industry relevance: Systems that must absorb new visual concepts and question types over time without retraining from scratch benefit directly. The memory-size sensitivity analysis is practically relevant because it shows how accuracy and forgetting trade off as buffer capacity grows — a key constraint for deployed systems with limited storage. The released code (https://github.com/HubuKG/MacVQA) supports reproduction and adoption.

Future Directions

1. Extending to broader modalities. The authors state that future work will extend the framework beyond vision-and-language to broader modalities.
2. Dynamic memory strategies for large-scale scenarios. The conclusion calls for exploring dynamic memory strategies at larger scale than the buffer sizes tested here.
3. Resolving the reported-metric inconsistency. The forgetting improvement is stated as −0.52% AF in the memory-capacity analysis and −0.58% AF in the ablation table; clarifying which comparison basis is correct would strengthen the claims.
4. Reducing dependence on task-specific hyperparameter tuning. Because visual-dominant and question-dominant tasks prefer opposite α/β settings, an open question is how to select modality weights automatically rather than by validation per task type.

Target Audience

Researchers and graduate students working on continual learning, multimodal learning, or VQA; engineers building question-answering systems that must adapt to evolving data without full retraining; and practitioners interested in prototype-based memory mechanisms or denoising techniques for cross-modal feature fusion. Readers should be comfortable with transformer architectures and standard continual-learning evaluation metrics.

Authors’ abstract

Visual Question Answering (VQA) requires models to reason over multimodal information, combining visual and textual data. With the development of continual learning, significant progress has been made in retaining knowledge and adapting to new information in the VQA domain. However, current methods often struggle with balancing knowledge retention, adaptation, and robust feature representation. To address these challenges, we propose a novel framework with adaptive memory allocation and global noise filtering called MacVQA for visual question answering. MacVQA fuses visual and question information while filtering noise to ensure robust representations, and employs prototype-based memory allocation to optimize feature quality and memory usage. These designs enable MacVQA to balance knowledge acquisition, retention, and compositional generalization in continual VQA learning. Experiments on ten continual VQA tasks show that MacVQA outperforms existing baselines, achieving 43.38% average accuracy and 2.32% average forgetting on standard tasks, and 42.53% average accuracy and 3.60% average forgetting on novel composition tasks.