Multimodal Data Fusion to Capture Dynamic Interactions between Built Environment and Vulnerable Older Adults

AI summary

Overview

Research area: Human-Computer Interaction / urban analytics — specifically the fusion of wearable sensing, computer vision, and spatial mapping to study how micro-scale built environment (BE) features affect vulnerable older pedestrians.

Technical level: Intermediate (the paper is a framework and feasibility demonstration; it combines familiar sensing concepts with named vision and SLAM models, but readers do not need deep expertise in any single one).

Scope (one sentence): The paper presents a synchronized multimodal data-collection and fusion pipeline — eye tracking, kinematic sensors, physiological monitors, GPS, first-person and third-person video — and demonstrates on naturalistic walking sessions in Singapore that it can reconstruct continuous walking trajectories and annotate them with walkway material and width, as a basis for inclusive, age-friendly urban design.

What This Paper Is About

Urban planners need to know how fine-grained built environment details — pavement surfaces, crossings, lighting, path geometry — affect older adults who have mobility difficulties, but surveys, interviews and GIS-based audits only give aggregated and self-reported views that miss moment-to-moment physical and perceptual responses. The authors build and test a multimodal sensing framework that records gaze, gait, physiology, video and location simultaneously while vulnerable older adults walk their own routes, then fuses these streams into a single spatiotemporal representation of the walk. The goal is to quantify which specific urban segments are behaviorally and perceptually significant, so that design decisions can be grounded in measured evidence rather than recall.

Key Contributions

1. A synchronized multimodal data-fusion framework that integrates wearable sensing (gaze, gait, physiological data) with environmental perception (video and GPS/SLAM mapping) into time-based or distance-based feature segments describing real walking sessions.
2. A trajectory reconstruction method that survives GPS loss: VINS-Fusion SLAM is run on chest-mounted video, reliable GPS segments are used as spatial anchors, and each SLAM-derived local trajectory is aligned to global coordinates using the Umeyama similarity transformation.
3. Automated micro-scale built environment annotation: walkway surface material is classified across 14 urban categories using Grounding DINO for region proposals, Segment Anything Model (SAM) for mask extraction, and a linear probe; walkway width is estimated from OpenPose skeletal keypoints, converting the pixel distance between walkway edges at foot level into metres using body-height references.
4. A prototype interactive dashboard that visualizes synchronized third-person video, first-person video with fixation overlays, and physiological signals, navigable by time or walking distance, with the stated potential to feed into urban digital twins for hotspot analysis.

Main Findings

• Trajectories stay continuous where GPS fails: In Singapore settings including covered walkways, underground linkways and semi-enclosed public spaces, SLAM provided smooth, locally consistent motion tracking while GPS anchored the trajectory globally; the fused trajectories captured direction changes and path curvature at sub-metre resolution across indoor–outdoor transitions.
• Walkway attributes can be recovered automatically: Grounding DINO plus SAM identified the regions participants actually walked on, and the example trajectory in the paper is colour-coded by estimated walkway width (narrow-to-wide transitions) and labelled with recognized surface materials including brushed concrete, asphalt, granite and exposed aggregate concrete.
• Perceptual and physiological signals can be geo-referenced: The framework projects gaze fixations onto scene frames and intersects them with Mask2Former-segmented objects (using the Mapillary Vistas dataset), and maps EDA and HRV indicators onto the reconstructed trajectory to flag stress-responsive micro-environments such as narrow corridors or uneven crossings.
• Gait can be monitored as a response to the environment: IMUs on feet, wrists, trunk and head, plus an Axivity A6 on the lower back, support temporal stride segmentation and metrics such as step count, mean stride time and stride-time variability (STV), which the authors describe as potentially indicating adaptive responses to obstacles or uncomfortable surfaces in participants with knee OA.
• The paper reports no quantitative performance results: No participant count, no classification accuracy for materials or widths, and no statistical correlations between BE features and behaviour are reported; the results are described as preliminary and illustrative.

Methodology in Plain English

The team recruited community-dwelling older adults aged 45 to 90 in the Yio Chu Kang district of Singapore, through healthcare institutions and community networks. To be eligible, a person had to have a clinical diagnosis of knee osteoarthritis or at least one fall in the past two years — conditions associated with reduced balance confidence and altered gait. Each participant walked a self-selected outdoor route, observed over sessions typically lasting 60 to 90 minutes, through covered walkways, pedestrian crossings, green corridors and commercial access paths.

While walking, participants wore eye-tracking glasses (Pupil Labs Neon) for gaze and first-person video, an Empatica EmbracePlus wristband for electrodermal activity, skin temperature and heart rate, ZurichMOVE inertial measurement units on feet, wrists, trunk and head, and an Axivity A6 on the lower back for high-frequency accelerometer and gyroscope data. A researcher followed at a distance with a GoPro for third-person context.

The analysis then runs in layers. Location is reconstructed by running VINS-Fusion SLAM on chest-mounted video and stitching the result to GPS anchors via the Umeyama transformation. Gaze fixations and saccades are detected with the I-DT algorithm using adaptive thresholds and optical-flow correction to compensate for head movement, then projected onto segmented scene objects. Physiology is decomposed into phasic and tonic EDA components with detected peaks, alongside HRV measures such as RMSSD and pNN10. Gait is segmented into strides to compute step count, mean stride time and stride-time variability. Finally, walkway material and width are extracted from video using Grounding DINO, SAM and OpenPose. Everything is aligned into fused, time- or distance-based segments, and shown in an interactive dashboard.

Why This Matters

Impact on research: The work positions individual-centred wearable sensing as a complement to, rather than a replacement for, surveys and GIS audits. Its distinctive move is linking first-person and third-person perceptual, behavioural and physiological data to specific micro-scale environmental features — something the authors argue existing aggregated geospatial approaches cannot do — and it frames this as a scalable path toward large-population studies.

Real-world applications:

• Auditing specific walkway segments — width, surface material, geometry — to identify where older pedestrians show elevated arousal or unstable gait.
• Redesigning walkway geometry or surface materials where EDA peaks and wide fixation dispersion suggest increased cognitive load or perceived risk.
• Hotspot analysis across many participants' trajectories to find locations or design features that consistently elicit stress, attention shifts or gait adaptation, informing evidence-based urban design improvements.
• Feeding fused representations into urban digital twins to support holistic, aggregate evaluation of accessibility and comfort.

Industry relevance: The pipeline is built entirely from commercially available wearables (Pupil Labs, Empatica, Axivity, ZurichMOVE, GoPro) plus open or published vision and SLAM components, which makes it plausible for urban planning consultancies, accessibility auditors, mobility-device makers, and smart-city digital-twin platforms to adopt. The dashboard concept also speaks to the needs of planners who must communicate findings to non-technical stakeholders.

Future Directions

• Expand the participant pool to investigate statistical association and enable more comprehensive correlation analyses across modalities, particularly between BE features and gait patterns in vulnerable populations.
• Integrate the multimodal outputs into city-scale digital twins, which the authors say will deepen understanding of how the BE influences mobility among vulnerable groups and enable hotspot visualization and policy making.
• Move from feasibility to validation: because no accuracy figures, participant counts, or BE–behaviour correlations are reported, the open question is whether the automated material and width estimates and the physiological indicators hold up quantitatively at scale.
• Turn indicators into design guidance: the paper suggests thresholds such as EDA peaks or fixation dispersion signalling risk, but how such signals should translate into concrete design standards is not yet established.

Target Audience

This paper suits urban planners and public-health researchers interested in age-friendly and inclusive design; human-computer interaction and ubiquitous-computing researchers working on multimodal sensing and wearable data fusion; computer-vision practitioners applying segmentation, pose estimation and SLAM to real-world urban data; and accessibility or smart-city technology teams exploring digital twins and evidence-based auditing. Readers looking for quantitative benchmark results will find feasibility and method descriptions instead, and will need to wait for the authors' stated follow-up work.

Authors’ abstract

Ensuring safe and inclusive mobility for vulnerable older adults is an emerging priority in urban planning. However, existing data sources such as surveys or GIS-based audits provide limited insight into how micro-scale built environment (BE) features influence real-world behavior and perception. This study presents a novel multimodal data-fusion approach that integrates wearable and environmental sensing to dynamically represent human-environment interactions and quantify the BE impacts on mobility among vulnerable older adults, specifically those with knee osteoarthritis or a history of falls. Data collected during naturalistic walking sessions in Singapore, are used to demonstrate this framework of synchronized streams from eye tracking, kinematic sensors, physiological monitors, GPS, and video recordings. Preliminary results show how AI-driven data fusion can uncover behaviorally and perceptually significant urban segments, providing a basis for actionable insights in inclusive design. This human-centered analytical approach advances the representation of urban environments from the perspective of vulnerable pedestrians, establishing a foundation for evidence-based, age-friendly city planning.