Vision Language Models for Dynamic Human Activity Recognition in Healthcare Settings

AI summary

Overview

Research area: Human activity recognition (HAR) for remote health monitoring, using Vision Language Models (VLMs) as an alternative to conventional deep learning classifiers.

Technical level: Intermediate. The paper assumes familiarity with activity recognition pipelines and multimodal models, but its core ideas—caption-based ground truth and four evaluation metrics—are explained without deep mathematical machinery.

Scope: The paper builds a descriptive-caption dataset over the Toyota Smarthome video dataset and uses four evaluation methods to compare three open-source VLMs plus GPT-4o against seven traditional vision-based HAR models on the dataset's standard cross-subject and cross-view protocols.

What This Paper Is About

Conventional HAR models must be trained on large labeled datasets and can only output one of a fixed set of predefined activity classes, which makes them awkward to bolt onto generative-AI healthcare systems that are already answering clinicians' natural-language questions. VLMs can instead describe what a patient is doing in free text, but there has been no fair, standard way to score these free-form outputs—strict keyword checks can underrate them and loose checks can inflate them. The paper's goal is to supply both a descriptive-caption ground truth and a set of evaluation methods, then use them to test whether VLMs can match or beat traditional deep learning models at recognizing activities in an elderly-monitoring setting.

Key Contributions

1. A descriptive caption dataset for HAR. Ground-truth labels for the Toyota Smarthome Dataset are expanded into full descriptive captions using an iterative GPT-4o-based framework, producing text suitable for evaluating generative model outputs rather than single class names.

2. Four evaluation methods for judging VLM activity recognition. Keyword Matching, VLM-as-Judge, BERTScore, and Cosine Similarity are proposed and empirically screened against each other to identify which are reliable.

3. A benchmark comparison of VLMs against deep learning models. Three open-source VLMs (Llama3.2-Vision-11B, DeepSeek-VL2-Small, InternVL2.5-8B) and GPT-4o are evaluated on the Toyota Smarthome cross-subject (CS) and cross-view (CV1, CV2) protocols against seven vision-based deep learning models.

4. A released artifact. Code and dataset are made available at https://github.com/gouga10/VLMs-HAR-RHMS.git.

Main Findings

• BERTScore is unusable for this task. It rated every model's outputs as correct (100.0 for GPT-4o, InternVL-2.5, DeepSeek-VL2, and Llama3.2-Vision), because it weights all tokens rather than the activity being described. Example: for video Usetelephone_p02_r00_v15_c06.mp4, InternVL-2.5's caption "The person is looking at a cup on the counter in the kitchen." scored a precision of 0.9215 against the ground truth "The person is using the phone."

• VLM-as-Judge also underperformed. Even with GPT-4o as evaluator, it judged InternVL-2.5's "The person is washing a cup in the kitchen." as incorrect against the ground truth "The person is washing something in the kitchen sink." The authors read this as evidence that their caption dataset is not biased toward GPT-4o's own phrasing.

• Keyword Matching and Cosine Similarity were selected as the reliable metrics and carried forward into the second experimental phase.

• VLMs are competitive on cross-subject evaluation. Under keyword matching, Llama3.2-Vision reached 67.4% MCA, above AssembleNet++ (63.6), LTN (65.9), VPN++ (69.0 is higher, but the paper lists it among models Llama3.2-Vision exceeds on the stated comparisons), and MotionFormer (65.8). Under cosine similarity, the reported numbers rise further: InternVL-2.5 reached 83.8% and DeepSeek-VL2 78.6%, both above the best deep learning CS result of 72.9% (π-ViT).

• VLMs lead on CV1 but are overtaken on CV2. Under keyword matching, Llama3.2-Vision's 56.2% is the best CV1 result, ahead of π-ViT's 55.2%. In CV2, where deep learning models train on more cameras, π-ViT reaches 64.8% and surpasses the VLMs, though the VLMs remain above several other deep learning models.

• Output verbosity explains Llama3.2-Vision's collapse under similarity scoring. It scores 54.0 on Phase 1 similarity versus 67.4 on keywords; longer captions raise the chance of containing a curated keyword but dilute semantic similarity to the ground-truth caption. DeepSeek-VL2 and InternVL2.5 produce more concise outputs and align better with ground truth.

• Not reported: GPT-4o appears in the Phase 1 table but has no Phase 2 MCA values; the paper also does not report Phase 2 results for BERTScore or VLM-as-Judge, having dropped them after Phase 1. Table 2 lists identical CV1 and CV2 values for each VLM (for example, 41.6/41.6 for DeepSeek-VL2 under keywords and 52.2/52.2 under similarity).

Methodology in Plain English

The researchers started from the Toyota Smarthome Dataset, a collection of 16,115 videos covering 31 daily activities performed by 18 participants aged 60 to 80, recorded from seven camera viewpoints—all properties that resemble a remote monitoring scenario for elderly patients. Only RGB frames were used, although depth and skeleton data are also available in the dataset.

Because those videos come with plain class labels (like "Cook_Cleanup") and VLMs need richer text to be judged fairly, the authors built a caption-generation pipeline. Keyframes were extracted with the Katna library, which picks frames by analyzing frame differences, brightness, and entropy, and by clustering image histograms with K-Means. GPT-4o was then asked to produce a curated keyword list for each ground-truth class label, and a VLM generated a caption from the visual input, the label, and those keywords. A check confirmed whether the caption contained at least one keyword; if not, the prompt was refined and GPT-4o was re-invoked. The loop repeated until a relevant keyword appeared, and a final automated check confirmed that 100% of captions contained one or more of the corresponding keywords.

For evaluation, four scoring strategies were defined: Keyword Matching (does the output contain curated words), VLM-as-Judge (a second VLM outputs only True or False), BERTScore (precision only, with a 0.9 threshold for a correct recognition), and Cosine Similarity using the all-MiniLM-L6-v2 embedding model, with a 0.5 threshold determined empirically from a manual review of 20 examples.

Experiments ran in two phases. Phase 1 tested all models on a 10-samples-per-class subset purely to screen which evaluation methods were trustworthy. Phase 2 used the dataset's official protocol: cross-subject (training on subjects 3, 4, 6, 7, 9, 12, 13, 15, 17, 19, and 25; testing on the remaining 7) and cross-view (testing on Camera 2, with CV1 training on Camera 1 only and CV2 training on Cameras 1, 3, 4, 6, and 7; only the 19 activities captured by both cameras are scored). The metric is Mean Class Accuracy, which averages per-class accuracy. VLMs were given two input frames each, except Llama3.2-Vision-11B, which by architectural constraint accepts only a single frame. Open-source models were prioritized for healthcare privacy and security reasons, with GPT-4o included as a benchmark.

Why This Matters

Impact on research. The paper addresses a methodological gap rather than just reporting accuracy: it shows that two commonly used automatic metrics (BERTScore and VLM-as-Judge) are unreliable for generative activity recognition, and it provides a caption-level ground truth that other groups can reuse for fine-tuning and evaluation. It also establishes a direct, protocol-matched comparison between VLMs and seven established vision models, which prior work on VLMs for HAR had not done.

Real-world applications:

• Remote health monitoring systems for elderly patients living independently, where activity is inferred without exposing raw video to clinicians.
• Clinical query interfaces, where a single VLM both recognizes what a patient is doing and answers questions like "What is the patient doing?".
• Assisted-living and smart-home deployments that need privacy-preserving monitoring without a fixed taxonomy of activities.
• Consolidation of an existing RAG-based monitoring engine (the authors cite REMONI and Ho et al.'s system) into fewer models, lowering compute overhead.

Industry relevance. The finding that open-source VLMs are competitive with purpose-built deep learning classifiers, without training on the target dataset, matters for vendors building monitoring products who want fewer specialized models in their stack. The privacy argument for open-source models, and the practical warning that verbose model outputs score well under keyword checks but poorly under semantic checks, are directly actionable for anyone shipping these systems.

Future Directions

• Fine-tuning VLMs on the descriptive caption dataset. The authors explicitly suggest their dataset can serve as a resource for further fine-tuning, which has not yet been tested.
• Improving Llama3.2-Vision's output control. Its verbosity drives the gap between keyword and similarity scores; the paper identifies the problem but does not solve it beyond prompting.
• Closing the CV2 gap. Deep learning models overtake VLMs when trained on more camera views, so the question of how VLMs can absorb additional viewpoints—and why CV1 and CV2 scores are reported as identical for every VLM—remains open.
• Replacing or repairing the discarded metrics. BERTScore and VLM-as-Judge failed here; the paper does not report whether a better version of either, or a learned evaluator, could work for free-form HAR output.

Target Audience

Researchers and graduate students working on human activity recognition, video understanding, or multimodal evaluation; healthcare AI engineers building remote patient monitoring or assisted-living systems; and practitioners who need to decide whether to use a VLM or a purpose-trained classifier for activity detection and how to score either one fairly.

Authors’ abstract

As generative AI continues to evolve, Vision Language Models (VLMs) have emerged as promising tools in various healthcare applications. One area that remains relatively underexplored is their use in human activity recognition (HAR) for remote health monitoring. VLMs offer notable strengths, including greater flexibility and the ability to overcome some of the constraints of traditional deep learning models. However, a key challenge in applying VLMs to HAR lies in the difficulty of evaluating their dynamic and often non-deterministic outputs. To address this gap, we introduce a descriptive caption data set and propose comprehensive evaluation methods to evaluate VLMs in HAR. Through comparative experiments with state-of-the-art deep learning models, our findings demonstrate that VLMs achieve comparable performance and, in some cases, even surpass conventional approaches in terms of accuracy. This work contributes a strong benchmark and opens new possibilities for the integration of VLMs into intelligent healthcare systems.