SocialFusion: Addressing Social Degradation in Pre-trained Vision-Language Models

AI summary

Overview

• Research area: Computer vision and multimodal machine learning — specifically visual social interaction understanding with pre-trained vision-language models (VLMs).
• Technical level: Advanced. The paper combines controlled encoder ablations, linear representation probing, gradient conflict analysis, and VLM architecture design.
• Scope: The paper diagnoses why pre-trained VLMs fail to jointly learn five social perception tasks, introduces the term "social degradation" for the cause, and proposes SocialFusion, a minimal fusion model that achieves positive transfer across all five tasks.

What This Paper Is About

Powerful pre-trained VLMs (Qwen2-VL 2B, Sail-VL 1.5 2B, and MolmoE at 1B active/7B total parameters) are surprisingly bad at learning multiple social perception tasks at once, and often get worse on a task when other tasks are trained alongside it. The authors trace this failure to what they call "social degradation": the visual-linguistic pre-training process itself degrades the visual encoder's ability to represent nuanced social information. Their goal is to avoid that degradation entirely by pairing a frozen, un-degraded visual encoder with a language model through a deliberately minimal connection.

Key Contributions

1. Identification of "social degradation." Through controlled experiments comparing visual encoders before and after VLM pre-training — both inside the Gaze-LLE architecture and inside their own SocialFusion framework — the authors show that VLM pre-training makes visual encoders worse at social interaction understanding tasks.
2. Mechanistic analysis under two lenses. They investigate the cause through decodability (linear representation probing on raw visual features) and compatibility (gradient conflict analysis), finding that reduced decodability is the primary factor, with pre-training only slightly reducing inter-task gradient alignment.
3. The SocialFusion model. A unified generalist framework that learns a minimal connection between a frozen CLIP visual encoder and a frozen Llama 3.2 1B language model, with native support for bounding box inputs and 2D gaze heatmap outputs.
4. Strong empirical results. SocialFusion is the only tested model to achieve positive transfer on all ten tested metrics when trained jointly, and it sets new state-of-the-art results on the HaGRIDv2 and PISC benchmarks.

Main Findings

• Every baseline VLM suffers negative transfer. When trained jointly on all five tasks, Sail-VL 1.5 and MolmoE each achieved positive transfer on only two of ten metrics (one of five tasks). Qwen2-VL achieved positive transfer on seven of ten metrics but performed worse than the other baselines overall on most tasks, and three of ten metrics were still harmed.
• SocialFusion is the only model with all-positive transfer. It improved on all ten tested metrics when moving from single-task to joint training. For example, HaGRIDv2 mAP went from 99.7 to 99.9, PISC Domain from 91.7 to 94.4, PISC Relation from 88.0 to 90.1, LAM mAP from 85.0 to 86.1, GazeFollow minimum L2 from 0.073 to 0.065, and AffectNet mAP from 67.1 to 68.0.
• VLM pre-training degrades vision encoders on gaze estimation. Tested inside the Gaze-LLE architecture, Qwen2-VL's encoder got worse after VLM pre-training (minimum L2 0.076 to 0.114, average L2 0.140 to 0.184, AUC 93.8 to 90.9). MolmoE degraded similarly (0.049 to 0.072, 0.107 to 0.138, 95.3 to 94.1), and Sail-VL 1.5 degraded slightly (0.049 to 0.051, 0.108 to 0.110, 95.3 to 95.2).
• Encoders degrade broadly across social tasks. Tested inside SocialFusion, pre-trained encoders performed worse on most tasks. MolmoE showed a clear and pronounced degradation on all tested social tasks; Qwen2-VL's results were mixed only on social relationship recognition; Sail-VL's original encoder beat its pre-trained version on many tasks and matched it on others.
• Decodability, not compatibility, is the main culprit. Linear probes on raw encoder features performed better before VLM pre-training on nearly every task–encoder pair: Qwen2-VL on every task except HaGRIDv2, Sail-VL 1.5 on every task except HaGRIDv2 and partially AffectNet, and MolmoE on every single task.
• Gradient conflict is small and does not reach true conflict. Gradient Conflict Degree (GCD) was slightly higher for pre-trained encoders (Qwen2-VL 0.961 vs. 0.941, MolmoE 0.950 vs. 0.947, Sail-VL 1.5 0.934 vs. 0.931), but all values remained below 1, the threshold at which gradients are considered conflicting.
• Partial pairwise synergies exist, but they do not explain everything. Among the ten possible task pairs, LAM plus GazeFollow notably improved GazeFollow, and HaGRIDv2 plus PISC improved PISC. Some tasks, such as AffectNet and HaGRIDv2, improved with no single partner, suggesting that full joint training produces a broader "social competency" effect that only appears as the number of tasks grows.
• New state of the art on two benchmarks, competitive on two, behind on one. SocialFusion reached 99.8 mAP on HaGRIDv2 (SOTA 89.4) and 94.2 / 88.5 mAP on PISC Domain / Relation (SOTA 87.0 / 79.5), setting new state of the art on those tasks. It was behind SOTA on LAM (76.0 mAP, 92.0 accuracy vs. SOTA 81.0 / 93.0), GazeFollow (0.065 / 0.128 / 94.0 vs. SOTA 0.041 / 0.099 / 95.8), and AffectNet (52.8 accuracy vs. SOTA 66.32).

Methodology in Plain English

The authors start by taking three widely used open-source VLMs and fine-tuning each with a LoRA adapter, both on each social task separately and on all tasks together. They shuffle datasets and randomly undersample them to the minimum size each epoch to avoid imbalanced training, and intersperse gaze heatmap batches with text batches. Comparing the separate and joint settings reveals negative transfer.

To test whether the visual encoder is the problem, they swap each VLM's encoder into a controlled framework and compare it against the same encoder before it went through VLM pre-training — for example, DFN ViT for Qwen2-VL, SailViT for Sail-VL, and CLIP for MolmoE. To find the mechanism, they run two diagnostic analyses: linear probes, which train a simple linear classifier on flattened raw encoder features to measure how easily each task can be read out of the representation; and gradient conflict analysis, which computes gradients for each task at initialization and measures the angle between them via the Gradient Conflict Degree.

For their own model, SocialFusion, they freeze CLIP (336×336 input, 14×14 patches, producing a 24×24 feature map) and freeze Llama 3.2 1B, connecting them with a three-layer MLP connector (hidden dimension 4096) and adding a LoRA adapter of rank 32 to the LLM's linear layers. Bounding boxes are turned into a binary mask and multiplied by one shared learned embedding that is added to the intersecting patches. Text-output tasks are posed as next-token prediction over a task-description prompt listing class options; gaze is handled by linearly projecting the final image features to a 24×24 map and upscaling to a heatmap, supervised against a 2D Gaussian (σ = 3) with pixel-wise binary cross-entropy. The total loss combines the LLM cross-entropy and the heatmap loss with a scaling hyperparameter λ. Training used AdamW with an initial learning rate of 2e-4, 500 warm-up steps, cosine decay, and batch size 32.

Why This Matters

Impact on research. The paper reframes a common assumption: that pre-training a VLM necessarily produces a stronger visual encoder. It shows that for social perception, standard VLM pre-training can actively harm the encoder's visual representations, and that the main loss is in how easily social information can be read out of those features rather than in task interference. This suggests that current VLM pre-training strategies may be detrimental to general social competence and motivates more socially-aware training paradigms.

Real-world applications:

• Augmented reality systems that need to interpret the social dynamics of a scene.
• Human–robot interaction, where robots must read gestures, gaze, and expressions to behave appropriately.
• Assistive technologies, such as tools to help people with autism interpret social cues.
• Any application requiring one model to handle gesture, gaze, facial expression, conversation dynamics, and social situation analysis at once rather than a separate specialist model per task.

Industry relevance. Specialist models per social task are accurate but impractical to deploy and maintain. SocialFusion shows that a frozen visual backbone plus a small connector and a small language model can be competitive while unifying tasks through plain-text prompts — an attractive cost profile, since only the connector, bounding box embedding, LoRA adapter, and gaze projection are trained. The finding that freezing the encoder helps is directly relevant to teams deciding what to fine-tune in production VLMs.

Future Directions

• Designing non-degrading pre-training. The paper's central implication is that new, socially-aware VLM pre-training paradigms are needed — ones that preserve fine-grained social decodability in the visual encoder rather than eroding it.
• Scaling to more multimodal inputs and more tasks. The authors explicitly note that SocialFusion's design should promote future work unifying additional multimodal inputs, and that extending to tasks with more than two bounding boxes may require a distinct embedding vector per box rather than the single shared embedding used here.
• Closing the AffectNet gap. SocialFusion remains behind state of the art on AffectNet, and the paper's comparison against related AffectNet work is truncated in the provided content, leaving the path to closing that gap unaddressed.
• Explaining the "social competency" effect. The paper observes that some tasks improve only under full joint training with no single helpful partner task, and concludes that a general social training effect emerges as task count increases. The mechanism behind this effect is not resolved and is left as an open question.

Target Audience

Researchers and engineers working on vision-language models, multimodal representation learning, and social signal processing will get the most from this paper, particularly those interested in transfer learning, multi-task learning, and parameter-efficient fine-tuning. It is also relevant to practitioners building human-centered AI systems for augmented reality, robotics, and assistive technology, who need unified social perception rather than one model per task. Readers should be comfortable with VLM architectures, linear probing, and gradient-based optimization analysis.

Authors’ abstract

Understanding social interactions from visual cues is a fundamental challenge for a socially competent AI. While powerful pre-trained vision-language models (VLMs) have shown remarkable general capabilities, they surprisingly struggle to unify and learn multiple social perception tasks simultaneously, often exhibiting negative transfer. We identify that this negative transfer stems from a critical issue we term "social degradation," whereby the general visual-linguistic pre-training process of VLMs impairs the visual encoder's ability to represent nuanced social information. We investigate this behavior further under two lenses: decodability through linear representation probing and compatibility through gradient conflict analysis, revealing that both play a role in the degradation, especially the former, which is significantly compromised in the VLM pre-training process. To address these issues, we propose SocialFusion, a unified framework that learns a minimal connection between a frozen visual encoder and a language model. Compared with existing VLMs, it exhibits positive transfer across all five social tasks, leveraging synergies between them to enhance overall performance and achieves comparable performance to task-specific state-of-the-art models on various benchmarks. Our findings suggest that current VLM pre-training strategies may be detrimental to acquiring general social competence and highlight the need for more socially-aware training paradigms.