CogVis: Must Open-Vocabulary Change Detection Perceive the Scene Anew for Every Query?

AI summary

Overview

• Research area: Open-Vocabulary Change Detection (OVCD) in remote sensing — identifying what changed between two satellite or aerial images when the categories to look for are given as free-form text rather than a fixed training label set.
• Technical level: Advanced. The paper assumes familiarity with vision-language models, frozen encoders, feature pyramids, threshold calibration, and standard remote-sensing change detection benchmarks.
• Scope: The paper proposes CogVis, a three-stage framework (Scene Change Perceptron, Semantic Memory Calibrator, Adaptive Region Filter) that separates query-shared temporal perception from query-specific semantic calibration and region verification, and evaluates it on seven benchmarks.

What This Paper Is About

Existing open-vocabulary change detection methods fold three different decisions into one fixed pipeline: whether a region changed at all (temporal), which text-specified category explains the change (semantic), and whether a detected region is trustworthy (regional). The paper argues that this entanglement causes unstable predictions and forces the same scene to be re-parsed from scratch for every new query. The goal is a framework that computes change evidence once per image pair, reuses it across all queries, and calibrates the final decision boundary per image-query pair using a memory of previously seen cases.

Key Contributions

1. A reformulation of OVCD as perception-memory-verification. Inspired by human visual change perception, the authors reorganize the task so that temporal evidence, semantic category decisions, and regional reliability are handled at separate stages instead of inside the fixed Mask-Compare-Identify (M-C-I) and Identify-Mask-Compare (I-M-C) cascades described by DynamicEarth.
2. The CogVis framework itself. Three components instantiate the paradigm: the Scene Change Perceptron (SCP) extracts a reusable category-agnostic change prior from frozen bi-temporal features; the Semantic Memory Calibrator (SMC) retrieves a memory-anchored decision threshold and corrects it for the current image-query pair; the Adaptive Region Filter (ARF) accepts or rejects connected candidate regions using semantic, temporal, and structural evidence.
3. Targeted fixes for three specific failure modes. Semantic-temporal-structural entanglement (SCP decouples temporal evidence from category decisions), image-query score shifts (SMC replaces a fixed threshold with an adaptive one), and unreliable region predictions (ARF verifies candidates before output).
4. Validation on seven benchmarks plus an efficiency analysis. CogVis reports the best metric on all seven evaluated datasets and reports a 28.50% improvement in inference throughput by sharing scene-level temporal perception across queries.

Main Findings

• State-of-the-art on all seven benchmarks. CogVis achieves 32.48% mIoU on SECOND, 25.36% mIoU on SCSCD, 33.67% IoU on CLCD, 48.50% IoU on DSIFN, 72.55% IoU on LEVIR-CD, 75.86% IoU on WHU-CD, and 32.68% mIoU on xBD. Margins over the strongest prior result range from 0.05 to 8.17 percentage points.
• Largest gain is on building-damage assessment. On xBD, mIoU rises from 24.51% to 32.68%, the largest single improvement reported (8.17 points).
• Smaller margins on cleaner binary datasets. LEVIR-CD improves by 0.05 points and WHU-CD by 0.14 points; the paper attributes this to existing methods already performing strongly on those datasets. Larger gains on CLCD (2.19) and DSIFN (0.95) are linked to suppression of appearance-induced temporal nuisances.
• Semantic change detection improves as well. SECOND gains 0.81 points over the previous best, and SCSCD gains 2.14 points.
• Efficiency comes from shared computation, not from a smaller model. CogVis reaches 0.230 FPS with 4.82 GiB peak memory and 28.50% higher throughput than the next-fastest method, despite using 990.29M parameters. At K=10 queries, CogVis requires 10.03 s versus 12.56–956.60 s for category-wise baselines.
• The learned change prior is a better change signal than raw pixel differences. SCP raises ROC AUC from 0.79 (raw RGB differences) to 0.96.
• Component ablation confirms complementarity. The full model beats the Vanilla variant by 13.12 mIoU on SECOND, 21.57 IoU on CLCD, and 4.25 mIoU on xBD. SCP alone gives the largest single-component gains on SECOND (2.95) and CLCD (8.13), while SMC gives the largest gain on xBD (1.85).
• ARF depends on upstream quality. Used alone, ARF slightly reduces SECOND performance while improving CLCD by 3.22 and xBD by 0.17 points. Combined with SCP and SMC, ARF consistently improves SECOND by 4.44, CLCD by 4.76, and xBD by 1.82 points.
• Memory gains are transferable rather than dataset-specific. Memory-only exclusion reduces SECOND/CLCD/xBD by 0.38/0.35/0.02 points; excluding the CA-CDD subset from both adapter optimization and memory construction reduces them by 1.06/0.50/0.22. Disabling the remaining cross-source memory costs 8.03, 7.47, and 1.60 points.
• Semantic memory can be compressed substantially. A 10% memory bank retains 94.9%/97.7%/95.6% of full-memory gains on SECOND/CLCD/xBD while cutting storage from 525.2 MiB to 52.6 MiB. A 25% bank uses 131.4 MiB and a 50% bank 262.6 MiB.
• Retrieval settings are not sensitive. Varying K_mem from 8 to 32 or T_m from 0.05 to 0.10 causes only marginal changes on all benchmarks. Region-level parameters (τ_r and η_a) have larger effects, particularly on CLCD.
• Qualitative robustness to non-semantic disturbances. The paper presents error maps for shadows, radiometric or textural variation, seasonal shifts, and residual misregistration, reporting sharper boundaries and closer alignment with annotations.
• Latency still grows with query count. Because SMC and ARF remain query-specific, increasing the number of queried categories still increases latency, though one-time temporal perception reduces the per-query cost.

Methodology in Plain English

CogVis keeps the heavy visual and language models frozen and trains only three lightweight adapters, each responsible for one decision.

1. Perceive the scene once (SCP). A frozen SAM3 feature pyramid is extracted for both timestamps. The second-level features from the two dates are combined into a single descriptor that captures appearance at each date, the absolute difference between them, and their element-wise product. A Change Adapter turns this descriptor into a residual, a temporal gate caps that residual's root-mean-square magnitude at 25% of the frozen-feature magnitude so it cannot dominate the original representation, and a change head predicts a single category-agnostic change map. This map is computed once per image pair and shared by every query. The original feature pyramid is left untouched so semantic decoding is unaffected.
2. Calibrate the decision boundary per query (SMC). For a given text query, the frozen decoder produces a target response, and CLIP-selected semantic neighbors provide the strongest competing response. The two are compared so ambiguous activations are suppressed. Four cues are then derived — semantic transition, support from the change prior, suppression of persistent semantics in stable regions, and change in query presence — and multiplied into a single semantic-change response. Separately, a memory bank stores, for each source image-query pair, a 1556-dimensional key (512-dimensional scene, temporal, and query embeddings plus 20 response statistics) alongside the oracle threshold that would have maximized IoU. At inference, cosine similarity retrieves the top 16 entries and their thresholds are combined into a temperature-weighted anchor (T_m = 0.07). A small Score Adapter predicts a bounded correction to that anchor, and the final threshold is clipped between a lower bound of 0.75 times a label-free adaptive threshold and an upper limit. Thresholding the response produces a candidate mask.
3. Verify regions (ARF). Connected components of the candidate mask are labeled and each is described by a 19-dimensional descriptor covering geometry, semantic confidence, temporal support, calibrated margin, and semantic-temporal agreement. A Gate Adapter predicts each component's reliability. A component is kept if either its area ratio is at least 0.04 (large-region protection) or its gate score is at least 0.60.
4. Training. The visual encoder, semantic decoder, and CLIP encoders stay frozen. The Change, Score, and Gate Adapters are optimized sequentially for 100 epochs at batch size 16 on CA-CDD, which the paper states does not overlap with any test set. The Change Adapter uses equally weighted balanced BCE-with-logits and soft Dice losses. The Score Adapter is trained on memory entries using leave-one-out retrieval and Smooth-L1 regression toward the oracle threshold. The Gate Adapter uses binary cross-entropy against a source-overlap label. Efficiency was profiled on an RTX 3090 GPU with 512×512 inputs, Python 3.10.20, PyTorch 2.5.1, CUDA 12.1, in FP32 with TF32 disabled.

Why This Matters

Impact on research. The paper reframes open-vocabulary change detection as a question of decision scope rather than architecture choice, arguing that query-shared, query-dependent, and region-level decisions should be separated. It also gives memory a different role than prior work: entries store a calibrated operating threshold rather than a feature prototype, so retrieval transfers a decision boundary instead of a representation. Because the encoders stay frozen and only adapters are trained, the approach is training-light and compatible with future foundation models.

Real-world applications (drawn from the applications the paper cites for change detection):

• Land-cover monitoring, tracking conversions between categories such as water, vegetation, bare land, and built structures.
• Disaster assessment, including the four post-event building damage states evaluated on xBD (no damage, minor damage, major damage, destroyed).
• Urban planning and built-environment monitoring, using the binary building-change benchmarks (LEVIR-CD, WHU-CD, DSIFN, CLCD).
• Ecological observation and environmental monitoring of Earth's surface, where acquisition conditions vary and users may query categories absent from a task-specific classifier.

Industry relevance. Satellite and aerial imagery providers, geospatial analytics platforms, and disaster-response organizations need models that answer arbitrary text queries without retraining. The reported throughput gain (28.50%) and the finding that a 10% memory bank retains most of the accuracy at 52.6 MiB of storage both matter for deployment, where latency and memory budgets are constrained. The 990.29M parameter count is a practical consideration for edge or on-board deployment.

Future Directions

• Recovering missed regions. The paper states explicitly that ARF removes unreliable proposals but cannot recover regions missing from the candidate mask, leaving candidate recall as an open problem.
• Reducing per-query cost further. Latency still increases with the number of queries because SMC and ARF remain query-specific, so only the temporal perception stage is currently shared.
• Scaling and generalizing the memory. The memory-bank analysis shows a 10% bank retains most of the gains and that gains come from cross-source transfer rather than same-dataset memorization; how far this coverage can be compressed or extended to new domains is not resolved.
• Broadening evaluation. All results use fixed, benchmark-stationary prompt vocabularies and all adapter training and memory construction draw from CA-CDD alone; behavior under free-form rephrasing, multi-lingual prompts, or sensor types outside the seven benchmarks is not reported.

Target Audience

Remote-sensing and geospatial researchers working on change detection; computer-vision researchers interested in open-vocabulary segmentation, vision-language model adaptation, and retrieval-augmented or memory-based inference; and practitioners building monitoring or disaster-response systems that must answer text-specified queries without retraining a model per category. Readers need a working knowledge of foundation models, dense prediction, and IoU-style evaluation to follow the ablations and benchmark tables.

Authors’ abstract

Earth-surface monitoring requires change detection models capable of recognizing arbitrary semantic categories. Open-Vocabulary Change Detection (OVCD) addresses this need. However, existing methods often entangle temporal perception, semantic discrimination, and region verification, causing unstable results and redundant computation. Inspired by human visual change perception, we propose CogVis, a cognitive memory-guided framework that reformulates OVCD as a perception-memory-verification paradigm. CogVis first employs a Scene Change Perceptron (SCP) to extract a reusable, category-agnostic change prior from frozen bi-temporal features, thereby decoupling temporal evidence from semantic category decisions. A Semantic Memory Calibrator (SMC) then compensates for category-dependent score shifts by dynamically estimating an image-query-specific decision threshold. Finally, an Adaptive Region Filter (ARF) filters connected candidates using learned semantic, temporal, and structural reliability. Experiments on seven benchmarks spanning semantic change detection, binary change localization, and building-damage assessment show that CogVis achieves state-of-the-art performance across all evaluated datasets. By sharing scene-level change perception, CogVis further avoids repeating category-agnostic temporal perception across queries and improves inference throughput by 28.50%.