Feature Recovery for Object Understanding After Irreversible Fire Damage

AI summary

Overview

Research area: Computer vision and multimodal representation learning, applied to post-disaster (post-fire) object understanding.

Technical level: Intermediate. Readers should be comfortable with object detection pipelines, vision-language models, and basic feature-space concepts, but the core ideas are explained in plain terms.

Scope: The paper introduces TRACE, a benchmark of 21.4K synthetic post-fire scenes plus 499 paired pristine-to-degraded object trajectories, and FRM, a lightweight plug-and-play module that corrects degraded visual features toward their pristine counterparts while keeping the host model frozen.

What This Paper Is About

When objects burn, they don't just get blurry or noisy — wood chars, plastic melts, metal oxidizes, and glass shatters, changing an object's shape, material, and appearance all at once. Existing detection benchmarks and robustness suites assume the underlying object is still structurally intact, so they don't test this setting. The paper builds a benchmark for it and proposes a method that helps frozen vision and vision-language models recover what an object was before it burned.

Key Contributions

1. TRACE benchmark. A transformation-aware benchmark with 21.4K real-image-grounded synthetic post-fire scenes (bounding boxes, material labels, severity captions) and 499 paired pristine-to-degraded object trajectories across 189 object types, each rendered at five severity levels (L0–L4) with part-level material annotation.

2. Five evaluation tasks. Degraded-object detection, pristine-state recovery and retrieval (in both feature space and gallery retrieval form), original material recovery, pristine description generation, and functional reasoning — covering localization through pre-degradation inference.

3. Empirical finding of systematic failure. State-of-the-art detectors, contrastive encoders, and five VLMs all degrade sharply as severity increases, indicating a representational limitation rather than a model-specific defect.

4. Feature Recovery Module (FRM). A lightweight residual Transformer with zero-gated initialization that maps degraded encoder features to pristine-aligned representations at a frozen host interface. It improves all five tasks across eight hosts (RF-DETR, CLIP, SigLIP2, and five VLMs) with roughly 9.0–30.8% relative gains. Dataset, pipelines, and training code are released.

Main Findings

• Baselines collapse under physical degradation. RF-DETR mAP drops 71% relatively from L0 to L4, and 92.9% of boxes localized at IoU ≥ 0.5 are misclassified. InternVL3.5 pristine-state retrieval R@1 falls from 93.85 to 28.11; Qwen2.5-VL falls from 76.92 to 32.31.

• Pixel-space restoration is the wrong tool. Restormer fine-tuned on TRACE pairs improves detection only modestly (0.338 → 0.375 mAP), and WildIR hurts it (0.266 mAP). Both also produce low pristine-state classification accuracy with frozen VLMs, suggesting that restoring local image quality does not restore pre-degradation identity cues.

• Domain adaptation fails. Marginal distribution-alignment methods (SFA, DATR) underperform even the frozen baseline, because degradation produces structured, per-instance drift that requires paired correction rather than aggregate alignment.

• FRM is additive to fine-tuning. Frozen RF-DETR improves from 0.338 to 0.442 mAP (+30.8% relative); fine-tuned RF-DETR goes from 0.572 to 0.647 (+13.1%), while pristine performance is preserved within 0.001 mAP. Fine-tuning broadens the encoder's input distribution; FRM corrects residual feature misalignment that persists afterward.

• Recovery generalizes across encoder families. FRM reduces MSE by 30.3% on CLIP ViT-L/14 and 43.4% on SigLIP2, with cosine similarity gains of +0.095 and +0.138. SigLIP2 degraded-to-pristine Top-1 improves by +14.8 points and Top-5 by +18.9 points, reaching 96.2% against a pristine-query ceiling of 100%.

• VLM gains scale with severity. Across five frozen VLMs, retrieval gains average only +0.73 points over L0–L1 but +14.66 points over L2–L4. Mean relative gains across hosts and severities are 12.5% (retrieval), 20.1% (material recovery), 13.2% (description generation), and 12.4% (functional reasoning).

• Weak baselines benefit most. InternVL3.5, which has the weakest degraded-input baseline, sees the largest absolute gains on material recovery (+9.81 F1) and description generation (+8.41 points). FRM's benefit tracks degradation sensitivity rather than model capacity.

• Recovered features support reasoning they were never trained for. FRM is trained only with paired feature supervision — no task labels — yet it improves functional reasoning on every host (mean +4.9 points), with +7.6 points at L3–L4. This suggests frozen language models already hold object-function knowledge and FRM supplies the visual evidence to invoke it.

• Ablations confirm the loss design. MSE alone restores retrieval but a normalized MSE (LN-MSE) drops retrieval below the no-FRM baseline, showing token magnitude matters. Combined MSE + 0.1·cosine loss gives the best tradeoff; R@1 is robust to the exact weight across λ ∈ {0.01, 0.1, 0.5} (1.16-point spread).

• Real-crop transfer is positive but smaller. On 500 real post-fire crops at L2–L4, Qwen3-VL gains +3.87 retrieval R@1, +4.67 material F1, +5.06 description, and +4.49 functional reasoning. Gains remain positive even at L4 (near-total destruction).

Methodology in Plain English

The researchers start from a simple observation: fire damage is not like camera noise. A charred object has lost parts and changed material, so you cannot invert the damage to recover the original pixels — the mapping is ambiguous or one-to-many. But a pretrained model already encodes rich semantics about objects, materials, and functions. That knowledge may still be reachable if the features can be nudged back toward where they would sit for an undamaged object.

To build data, they generate post-fire scenes and object degradation trajectories using Gemini 3 Pro Image, conditioned on two to four real post-fire reference images per scene (from roughly 700 images drawn mainly from Fire360). A Gemini 3 Flash vision-language judge scores candidates on realism, material fidelity, structural plausibility, and reference consistency, rejecting anything below 0.7 — yielding 21,400 accepted scenes from about 29,000 generations. Object trajectories use material-aware prompts that capture wood charring, plastic melting, metal oxidation, and glass fracture across five severity levels. Two authors independently labeled a stratified 300-object subset, reaching 93.4% agreement (Cohen's κ = 0.836). A separate set of 500 real post-fire crops is annotated the same way and held out for testing.

For the method, FRM is inserted at a chosen "tap point" inside a frozen model — after the encoder and before the projector in RF-DETR, or at the post-merger interface in Qwen3-VL. It is a small residual Transformer whose attention and MLP outputs are gated by learned channel-wise scales initialized to zero, so FRM starts as an exact identity function and training is stable. Given a degraded image, the frozen encoder produces features; FRM predicts a correction; the corrected features flow through the otherwise untouched downstream pipeline. The pristine image is run through the same frozen encoder without gradients to serve as the training target. The loss combines element-wise MSE (which constrains token magnitudes) with a token-level cosine term (which penalizes directional drift), weighted at λ = 0.1.

Because only FRM trains, encoder features can be precomputed once and reused. Training a head on ~2,000 paired examples takes about one hour on a single A100 with cached features, or three hours recomputing forward passes on the fly. Parameter overhead is configurable via a budget formula; the RF-DETR head uses 1.77M parameters (5.25% overhead) and the Qwen3-VL-4B head uses 314.6M (7.86% overhead).

Why This Matters

Impact on research. This is the first benchmark to study object-level physical degradation with paired pristine-to-degraded trajectories, enabling severity-stratified evaluation and feature-space recovery measurement. It reframes robustness away from image-quality corruptions toward a setting where the object itself has changed. The result that structured, per-instance feature drift defeats distribution-alignment methods is a useful negative finding for the adaptation literature, and the observation that a purely feature-space correction helps an untouched language model reason about function suggests where degradation-induced loss actually lives in a VLM pipeline.

Real-world applications:

• Emergency response. Firefighters locating hazardous items — aerosol cans, gas cylinders, chemical containers — inside collapsed or charred structures where visual cues have been destroyed.
• Insurance adjustment. Inventorying damaged property from partial remains, where adjusters currently work manually from incomplete evidence.
• Fire investigation. Reconstructing pre-incident scene contents to determine what was present before ignition.
• Debris triage and environmental safety. Recovering original material identity to determine whether an item is restorable or requires replacement, and to route debris for safe disposal.

Industry relevance. The plug-and-play property is the key commercial angle: FRM requires no retraining of the host model, works on a fixed ~5–8% parameter overhead budget, and trains in about an hour on cached features. That makes it deployable on top of existing frozen detection and VLM stacks rather than requiring a full fine-tuning cycle, and extensible to new transformation categories by encoding additional images and continuing training. The paper is also explicit that deployment risks are shared with general object recognition, and that pristine-state predictions are inferences from incomplete evidence that require uncertainty quantification and human oversight in legal or insurance contexts.

Future Directions

• Query-aware gating. FRM's recovery direction is task-dependent: it helps queries about original identity, materials, and function but erases the damage signal needed for queries like "describe the fire damage." A gating mechanism would let one host serve both pristine-state and damage-state queries.

• Cross-host transfer. FRM weights are tied to a specific (encoder, tap, feature space) configuration, so every new host needs its own trained module. Making the correction transferable across encoders would substantially reduce deployment cost.

• Reducing supervision requirements. Adding a new object identity currently requires a pristine visual anchor plus at least three degraded observations across severity levels. Relaxing this — through few-shot, weak, or unpaired supervision — is an open problem.

• Closing the synthetic-to-real gap and incorporating spatial context. Real-crop gains are positive but smaller than synthetic gains, reflecting generative-model limits, approximate real-image severity labels, cluttered scenes, and unmodeled failure modes. The authors also note that humans use room layout, neighboring objects, and surviving structural parts to identify remnants — cues FRM currently ignores. Extending the paired-supervision recipe to other irreversible transformations (flood damage, structural collapse, weathering) is the natural generalization, wherever pre-event visual references exist.

Target Audience

Researchers and practitioners in computer vision robustness, multimodal representation learning, and vision-language model adaptation will find the technical core most useful — particularly those working on domain shift, test-time adaptation, or image restoration who want to see where those paradigms break down under physical transformation. Robotics and disaster-response researchers evaluating perception systems for degraded environments are a second natural audience. Fire-safety engineers, forensic investigators, and insurance technologists will benefit from the framing of the problem and the task definitions, even if the modeling details are less central to their work. The paper is written for readers with intermediate background; the conceptual arguments are accessible, but the ablations and tap-point discussion assume familiarity with transformer-based vision pipelines.

Authors’ abstract

Objects in post-fire environments often undergo irreversible physical transformations that change their geometry, material state, and visual appearance. Detecting and identifying these remnants is critical for locating hazards, reconstructing pre-incident contents, and inventorying losses. Unlike standard image corruptions, these degradations affect the physical structure of the object itself. To study this setting, we introduce TRACE, a transformation-aware benchmark for post-fire object understanding. TRACE contains 21.4K real-image-grounded synthetic scenes and paired object-level pristine-to-degraded progressions spanning 499 object identities across 189 categories. We define five tasks targeting localization and pre-degradation understanding: degraded-object detection, pristine-state recovery and retrieval, original material recovery, pristine description generation, and functional reasoning. Existing models degrade sharply with severity. From the least to the most severe level, RF-DETR mAP decreases by 71% relative, while InternVL3.5 retrieval R@1 falls from 93.85 to 28.11. To address this, we propose the Feature Recovery Module (FRM), a plug-and-play module that maps degraded encoder features to pristine-aligned representations while keeping the host frozen. Trained only with paired feature supervision, FRM improves scene-level detection, CLIP/SigLIP2 feature recovery, and all four object-level VLM tasks, with larger gains under more severe degradation. Across VLM hosts and severity levels, relative gains average 12.5% for retrieval, 20.1% for material recovery, 13.2% for description generation, and 12.4% for functional reasoning.