Unified Video Dense Prediction from Disjoint Data

AI summary

Overview

Research area: Computer vision — unified multi-task dense prediction for images and video, built on a pretrained latent diffusion model.

Technical level: Advanced. The paper assumes familiarity with diffusion models, latent spaces, dense prediction tasks, and video temporal consistency evaluation.

Scope: The paper introduces UniD, a single video model that jointly predicts eight dense scene properties (depth, surface normals, semantic segmentation, boundaries, human parts, albedo, shading, and materials) learned from task-specific datasets that are never co-annotated.

What This Paper Is About

Dense scene understanding requires many different predictions at once — how far away things are, what they are made of, what their surfaces are like, and what category they belong to. In practice, each of these labels is collected in a completely different way and in a different kind of environment: geometric labels come from RGB-D or stereo captures in architecturally limited scenes, semantic labels come from manual annotation of diverse internet images, and intrinsic properties like albedo and shading are largely measurable only in synthetic domains. Because these datasets rarely share images, existing unified models either restrict training to fully co-annotated data or pay a large cost to pseudo-label everything. UniD attacks this directly: it learns a unified model from genuinely disjoint datasets by having per-task experts supervise a shared backbone through lightweight task projectors, with no annotation overlap and no pseudo-labels.

Key Contributions

1. A latent-space distillation framework for disjoint data. Rather than pseudo-labeling a shared dataset with specialists, UniD distills the specialists' latent embeddings on the fly. This avoids storing or computing K full-resolution label maps per frame, skips gradients through the (frozen) VAE decoder, and unifies every task's training objective as latent reconstruction.

2. UniD, a unified video model for eight dense tasks. Depth, surface normals, semantic segmentation, boundaries, human parts, albedo, shading, and materials are all predicted by one shared backbone with lightweight per-task latent projectors, producing all task outputs simultaneously in a single forward pass.

3. Temporal consistency without task-specific video annotations. The model uses Extended Self-Attention over a memory bank of past frames, plus a temporal latent stabilization loss adapted from temporal gradient matching. Tasks that have no video supervision inherit temporal coherence from tasks that do.

4. An extensible training recipe. Adding a new task requires only training a new specialist on its own labeled data and adding a new latent projector; no existing data needs re-labeling and the unified backbone architecture does not change.

Main Findings

• Competitive with task specialists on geometry and intrinsics. UniD (the unified backbone) achieves performance similar to the per-task specialists it distills from on depth, normals, albedo, shading, and boundaries. For surface normals it actually outperforms its own specialist on NYUv2, ScanNet, and Sintel, which the authors attribute to cross-task regularization from depth sharing the same geometric training domain.

• Strong out-of-distribution generalization versus a frozen DINOv3-H baseline. With identical training data, UniD beats DINOv3-H on all five depth benchmarks (e.g., AbsRel 0.059 vs. 0.092 on NYUv2, 0.063 vs. 0.089 on ScanNet, 0.301 vs. 0.314 on DIODE), and achieves state-of-the-art among unified methods on all but one OOD surface-normal benchmark metric.

• Clear OOD gains on intrinsics and boundaries. UniD improves albedo WHDR from 0.289 (DINOv3-H) to 0.207, shading P@0.7 from 87.8 to 94.6, and boundary odsF from 55.0 to 57.2, while matching DINOv3-H in-domain.

• Roughly 3x more robust to appearance shift. Under color jittering on Hypersim val, the UniD specialist's depth AbsRel deviation is 0.011 vs. 0.034 for DINOv3-H, and normal mean-error deviation is 2.2° vs. 5.8°.

• A systematic gap on classification tasks. On segmentation, UniD's plain latent reconstruction lags the specialists (Cityscapes mIoU 37.8 vs. 64.1; ADE20K 35.7 vs. 44.8). The gap persists even with K=1, indicating it is intrinsic to the L1 latent objective rather than to unification. The authors trace it to projector–backbone misalignment: cross-entropy-task specialist latents have much higher spatial Laplacian norms (2.52 vs. 1.40) than regression-task latents, and L1 distillation appears to suppress that high-frequency structure.

• Projector-only fine-tuning closes most of the gap. Freezing the backbone and fine-tuning only the task projectors raises Cityscapes mIoU from 37.8 to 58.4 and ADE20K mIoU from 35.7 to 44.8, which the authors read as evidence that the backbone does learn semantically rich representations under L1 distillation.

• Best temporal consistency among unified methods. UniD beats all unified baselines on every evaluated task in Table 6, and is competitive with video-specific specialists — though it marginally underperforms Video Depth Anything on depth (ScanNet AbsRel 0.103 vs. 0.063) while outperforming NormalCrafter on normals.

• Gains are not just the attention module. Adding the same Extended Self-Attention to DINOv3-H and DICEPTION yields only marginal and inconsistent improvements, and UniD still outperforms all such variants. Temporal stability comes from the combination of the module with unified training on video data.

• Best cross-task geometric consistency. On depth↔normal consistency on ScanNet, UniD achieves the best result among all methods (mean error 22.6, δ1 39.0), beating DICEPTION, DINOv3-H (36.0 / 14.1), and even the independent specialist ensemble (23.9 / 39.0).

• Human part ↔ semantic consistency is near-saturated. UniD reports 98.3% inclusion of predicted human part pixels within the human semantic category, marginally above DINOv3-H (98.1%) and the specialist ensemble (97.6%).

Methodology in Plain English

The approach runs in two stages.

Stage one — train per-task specialists. The authors start from Stable Diffusion v2 as a pretrained latent diffusion model with a frozen VAE encoder and decoder. For each of the eight tasks, they fine-tune a copy of the U-Net backbone plus a tiny pixel projector (a single 3x3 convolution) on that task's own dataset, using a single-step diffusion prediction instead of iterative denoising. The result is a task-specific "latent space" — an internal representation that encodes what that task cares about — plus a decoding path to full-resolution predictions. Crucially, the diffusion model's internet-scale pretraining gives these specialists visual priors that generalize beyond their narrow training domain.

Stage two — distill into one unified model. A single unified backbone, initialized from the same pretrained U-Net, processes the input and produces one shared representation. Small per-task latent projectors (DPT heads that fuse three intermediate backbone features at fusion dimension 256) map that shared representation into each specialist's latent space. The training target is the specialist's latent, computed on the fly — so no pseudo-labels ever need to be stored, and no dataset needs to be co-annotated. The loss is L1 latent reconstruction plus a temporal gradient matching term that enforces agreement between the prediction's frame-to-frame changes and the target's, masked in fast-changing regions. The unified model trains for 50k iterations with batch size 32 and learning rate 3e-5 over the union of all labeled data, sampling image and video data with equal weight.

Video handling. Every self-attention layer in the backbone is extended so that queries attend to keys and values from both the current frame and M historical frames stored in a memory bank. Setting M=0 reduces to single-frame prediction, which is what lets the model train on datasets that have no video annotations at all. The memory length and enqueue/dequeue frequency can be changed at inference without retraining.

Extension. To add a ninth task, you train one new specialist and one new latent projector. The unified backbone is untouched and nothing is re-labeled.

Why This Matters

Impact on research. Almost all prior unified dense prediction work assumes every training image is annotated for every task, which is unrealistic given how labels are actually collected. UniD shows that latent-space distillation over disjoint datasets is a viable alternative to the dominant pseudo-labeling paradigm, and that the generative priors of a web-scale diffusion model are what make cross-domain transfer work. It also provides evidence about when this recipe fails — classification tasks with cross-entropy objectives do not transfer cleanly through an L1 latent reconstruction target — which is a concrete, diagnosable finding for follow-up work.

Real-world applications.

• Embodied agents and robotics: a robot navigating or manipulating a scene needs depth, semantics, materials, and surface properties simultaneously and continuously, which is exactly the multi-task streaming setting UniD targets.
• Video editing and post-production: intrinsic decomposition (albedo and shading) plus semantic and boundary maps are the standard ingredients for relighting and selective manipulation of footage.
• Autonomous driving and scene reconstruction: depth, normals, boundaries, and semantics in a single fast pass over a driving video, with temporally stable outputs.
• Augmented reality and content creation: consistent geometry and material cues over time let virtual objects be inserted with plausible lighting and occlusion against real video.

Industry relevance. Adobe Research co-authored the work, and the paper's framing reflects production concerns: a shared backbone computes features once for all tasks, per-task overhead is minimal, and all predictions come out of a single forward pass rather than K separate passes as in DICEPTION. That efficiency profile, plus the ability to add tasks without re-labeling existing data, is directly relevant to any team that already owns a stack of task-specific models and wants to consolidate them.

Future Directions

1. Fixing the segmentation gap at its source. Projector-only fine-tuning closes most of the gap but requires a supervised fine-tuning step, and the authors show the problem persists even with a single segmentation task. A reconstruction objective that preserves the high-frequency structure cross-entropy specialists encode — motivated here by the 2.52 vs. 1.40 Laplacian-norm observation — would remove that step.

2. Scaling the number of tasks and measuring the cost. The paper claims extensibility in principle (add a specialist plus a projector), but does not report results for K>8 or quantify the degradation as K grows. Whether the shared backbone saturates is an open question.

3. Latency and memory characterization. The paper argues for efficiency and notes that the memory queue length is adjustable at inference, but the content provided does not report wall-clock latency, throughput, or memory figures. A practical deployment study, especially on long videos, is a natural next step.

4. Beyond the eight dense tasks. The framework is agnostic to task type in principle, but the reported evaluation is limited to dense prediction. Whether the same latent-distillation recipe transfers to tasks with fundamentally different output structures — detection, tracking, or 3D reconstruction — is untested.

Target Audience

Researchers and engineers working on multi-task learning, dense prediction, or diffusion-based perception — particularly those who already have per-task models trained on separate datasets and want to unify them without re-annotating anything. It is also relevant to practitioners in video understanding, AR/VR, computational photography, and robotics who need temporally stable multi-property scene predictions in a single pass. Readers should be comfortable with diffusion model terminology (VAE latents, U-Net backbones, single-step vs. iterative denoising) and with standard dense prediction metrics (AbsRel, δ1, mean angular error, mIoU, odsF). Beginners will find the motivation section accessible but will need background for the method.

Authors’ abstract

Scene understanding requires simultaneous prediction about geometry, appearance, and semantics. However, existing task-specific annotations are fragmented across incompatible, domain-specific datasets. Current unified systems circumvent this by restricting training to fully co-annotated data, or by incurring the large computational cost of pseudo-labeling. To mitigate this, we introduce UniD, a unified video model that jointly predicts eight dense scene properties-depth, surface normals, semantic segmentation, boundaries, human parts, albedo, shading, and materials-all learned from disjoint, domain-specific datasets. We propose a simple yet effective distillation step in which per-task experts supervise a unified backbone through lightweight task projectors, eliminating the need for annotation overlap or pseudo-labeling. Our key insight is that the strong visual priors of a pretrained diffusion model are sufficient to bridge the domain gaps introduced by disjoint training sources, enabling robust generalization to scene-task combinations never seen during training. UniD achieves competitive performance against per-task specialists and multi-task baselines, with strong generalization to out-of-distribution scenarios and enhanced temporal and cross-task consistency. Code and video results are available at https://unid-video.github.io/.