WPI - Computer Science Department , PhD Dissertation Defense Wen Ge " Deep Multi-Modal Context-Aware Human Activity Recognition"

Wen Ge 

PhD candidate

WPI – Computer Science Department 

Friday, December 13, 2024

Time: 10:30 a.m. – 12 :30 p.m.

Zoom: https://wpi.zoom.us/j/5062082299?pwd=L0tXekJRbWFwNEcrNjNkeGNCTUNkQT09

Committee members : 

 Advisor : Prof. Emmanuel Agu, Computer Science - Data Science

 Prof. Elke Rundensteiner, Computer Science - Data Science

 Prof. Yanhua Li, Computer Science - Data Science

Prof. Songbai Ji, Biomedical Engineering

Abstract :

Context-Aware Human Activity Recognition (CA-HAR) from mobile sensor data faces significant challenges, including analyzing complex spatial-temporal patterns to learn intricate relationships between activities and contexts, and ensuring adaptability to noisy and diverse real-world environments. This dissertation tackles these challenges and proposes six innovative approaches that leverage smartphone sensor signals for robust CA-HAR architectures. 

The first two CA-HAR approaches, CRUFT and QCRUFT directly predict activity and context from complex spatial-temporal patterns in smartphone sensor signals. The next three CA-HAR approaches HHGNN, DHC-HGL, and CLAUDIA, proposed graphical neural networks to address the challenge of modeling co-occurring activities and contexts in label space, reformulating the multi-label classification problem into a node representation task. Finally, our last proposed CA-HAR approach, SEAL, addressed multi-label classification challenges through multi-modality alignment. SEAL introduced semantically enriched label representations by leveraging language models to encode the meaning of activity and context labels. SEAL tackles the challenge of capturing nuanced relationships between labels that binary encoding schemes often fail to capture. By embedding activity and context labels in a shared semantic space and aligning these with sensor data representations, the SEAL model exhibits an enhanced ability to generalize across various datasets, ensuring strong performance even in noisy conditions. The substantial improvements over state-of-the-art baselines demonstrate that integrating spatial-temporal analysis, graph-based learning, and semantic label encoding are crucial contributions to developing adaptive and robust CA-HAR systems for real-world applications.