Knowledge-Guided Multimodal Reasoning over Interacting Streams for Video-Level Ambivalence and Hesitancy Recognition

AI summary

Overview

Research area: Computer vision and multimodal affective computing, applied to health behaviour change — specifically the automatic recognition of ambivalence and hesitancy from video.

Technical level: Advanced. The paper combines frozen multimodal encoders, a streaming sequence model with prototype discovery and next-window prediction, dense window-level supervision, threshold calibration, and a knowledge-guided large language model with late fusion.

Scope: The paper proposes PRISM-AH, a framework that treats ambivalence and hesitancy as a multimodal conflict unfolding over time, and evaluates it on a public test partition of 525 labelled videos.

What This Paper Is About

Ambivalence and hesitancy (A/H) are conflicting emotional states that tend to precede someone delaying or abandoning a health behaviour change, such as quitting smoking or taking up exercise. Detecting them automatically is hard because the signal is not located in any single channel: it emerges from disagreement between and within facial expressions, voice, language, and body movement, and it looks different from person to person. The goal of this work is to build a video-level recogniser that captures that cross-modal disagreement over time rather than treating each modality independently.

Key Contributions

1. A framework that reframes A/H as multimodal conflict over time. PRISM-AH (Predictive Reasoning over Interacting Streams for Multimodal Ambivalence/Hesitancy Recognition) explicitly scores dissonance across modalities within short, aligned time windows rather than pooling modality features into a single static representation.

2. A lightweight streaming model built on frozen encoders. Vision, audio, and text encoders are frozen and aligned into short windows; a small streaming model then scores cross-modal dissonance, predicts each next window to expose a "hesitation surprise" signal, discovers behaviour prototypes, and is conditioned on participant metadata.

3. Dense window-level annotation used as an auxiliary objective, with threshold calibration. Window-level labels supervise the model in addition to the video-level task, and the decision threshold is tuned for macro F1 rather than raw accuracy.

4. Knowledge-guided LLM reasoning with guarded late fusion. A large language model reasons over structured evidence using the dataset's expert cue taxonomy, and its verdict is fused into the pipeline late — only when validation performance improves, rather than unconditionally.

Main Findings

• Reported A/H recognition performance: On the labelled public test partition of 525 videos, PRISM-AH reaches a macro F1 of 0.6133, against a reported zero-shot baseline of 0.2827.

• The gain is attributed to the reasoning component: The abstract states that the reasoning gain was validated to transfer from the validation split to the larger test partition, which is the paper's evidence that the LLM reasoning stage is not merely overfitting the validation set.

• Macro F1 was the optimisation target: Threshold calibration is described as being aimed at macro F1, implying the authors treated balanced per-class performance as the metric of interest rather than overall accuracy.

• The pipeline is deliberately staged and conditional: Frozen encoders, a lightweight streaming model, and an LLM reasoner are combined, but the LLM's contribution is admitted only when validation performance improves — the abstract presents this as a design safeguard, not an ablation result.

• Details not available in the abstract: The abstract does not report the identity or size of the encoders, the LLM used, the size of the training or validation partitions, per-modality ablations, or any comparison against non-zero-shot baselines. Those specifics are not in the abstract and are not reproduced here.

Methodology in Plain English

The authors start from pre-trained models for vision, audio, and text, and freeze them — meaning those models are not retrained. Their outputs are aligned into short time windows so that, within each window, the system has a view of what the face, voice, and words are doing at the same moment.

A small streaming model then processes those windows. Instead of only asking "is this person ambivalent?", it does several things at once: it measures how much the different modalities disagree with each other in a window, it tries to predict what the next window will look like (so that when reality diverges from the prediction, there is a measurable "surprise" signal that may correspond to a moment of hesitation), and it identifies recurring patterns of behaviour, called prototypes. Information about the participant is fed in as conditioning, so the model can account for the fact that people express hesitation differently.

The model is trained not just on a single label per video but also on dense labels attached to individual windows, which gives it a much finer-grained teaching signal. Because the classes are imbalanced, the final decision cutoff is tuned to maximise macro F1.

Finally, a large language model is given structured evidence derived from these streams and asked to reason about it using a taxonomy of expert cues provided by the dataset. Its judgement is combined with the streaming model's output only at the end, and only if doing so improves results on the validation data.

Why This Matters

Impact on research. Most affective computing pipelines treat each modality separately and then concatenate features. This work argues that the disagreement between modalities is itself the signal for ambivalence and hesitancy — a reframing that could generalise to other states defined by internal conflict, such as uncertainty, reluctance, or deception. It also demonstrates a pattern for combining a learned streaming model with an LLM reasoner under a validation-gated fusion rule, which is a reusable design for keeping an LLM's contribution honest.

Real-world applications.

• Digital health coaching and behaviour-change apps that could detect when a user is wavering and intervene at that moment rather than after they have dropped out.
• Clinical and counselling settings, where a therapist or triage system could be alerted to patient hesitancy about a treatment plan.
• Telehealth intake and remote consultation, where video is already being captured and hesitation about following advice is clinically relevant.
• Conversational agents and motivational-interviewing tools that adapt their responses when a user shows conflicting signals about a goal.

Industry relevance. The framework is built on frozen encoders plus a lightweight streaming model, which is an attractive cost profile for deployment compared with fine-tuning large multimodal models. Any product that already captures video and cares about adherence — wellness platforms, insurance-linked health programmes, remote patient monitoring, and educational or coaching platforms — has a plausible use for this kind of signal.

Future Directions

• Identifying which modalities carry the signal. The abstract frames A/H as arising from disagreement across facial, vocal, linguistic, and bodily channels, but reports no per-modality breakdown. Determining which streams and which cross-modal conflicts matter most is an open question.

• Testing generalisation beyond one dataset and one cue taxonomy. The LLM reasoning stage depends on the dataset's expert cue taxonomy. Whether that taxonomy transfers to other populations, languages, or health behaviours is not addressed.

• Understanding what the LLM adds. Because fusion is gated on validation performance, the size and consistency of the reasoning contribution across splits is the natural follow-up question, along with whether a smaller or domain-specific reasoner would do as well.

• Moving from recognition to intervention. The stated motivation is that A/H precedes delay or abandonment of behaviour change. Whether detecting it in time actually enables an effective intervention is a separate, unaddressed question.

Target Audience

Researchers and graduate students in multimodal affective computing, computer vision, and health informatics; practitioners building video-based behaviour or adherence monitoring systems; and applied machine learning engineers interested in architectures that combine frozen encoders, streaming temporal models, and validation-gated LLM reasoning. Readers looking for detailed architectural specifications, ablation studies, or comparisons against non-zero-shot baselines will not find them in the abstract alone.

Authors’ abstract

Ambivalence and hesitancy (A/H) are conflicting affective states that precede the delay or abandonment of health behaviour change. Recognition of A/H at the video level is difficult, since the signal arises from disagreement across and within facial, vocal, linguistic, and bodily modalities, and manifests differently across individuals. The proposed PRISM-AH (Predictive Reasoning over Interacting Streams for Multimodal Ambivalence/Hesitancy Recognition), is a framework that treats A/H as a multimodal conflict that unfolds over time. Frozen vision, audio, and text encoders are aligned into short time windows and passed to a lightweight streaming model that scores cross-modal dissonance, predicts each next window to expose a hesitation surprise signal, discovers behaviour prototypes, and is conditioned on participant metadata. Dense window-level annotations supervise the model as an auxiliary objective, and the decision threshold is calibrated for macro F1. A knowledge-guided large language model then reasons over structured evidence using the expert cue taxonomy of the dataset, and its verdict is fused late only when validation performance improves. On the labelled public test partition of 525 videos, PRISM-AH attains a macro F1 of 0.6133, compared to the reported zero-shot baseline of 0.2827. The reasoning gain is validated to transfer from validation to the larger test partition.