Vision-language models for real-time personalized hip exoskeleton control

This thesis investigates a multimodal human–machine interface in which verbal feedback and egocentric visual observations, acquired through head-mounted smart glasses, are translated into updates to a hip exoskeleton assist controller. The objective is to support explicit, user-guided modulation of assistance in the loop of ambulation, rather than reliance on offline parameter sweeps or optimization objectives that omit linguistic intent and environmental context.

The technical approach couples automatic speech recognition and vision-language reasoning with structured generation of controller-consistent commands, subject to real-time execution constraints on wearable and edge computing hardware. Session-level learning methods are additionally considered when single-utterance supervision is insufficient to disambiguate assistive regimes.

This project addresses estimation of biological joint moments during gait in post-stroke populations using temporal convolutional networks applied to musculoskeletal simulation–aligned inertial measurement data. Heterogeneous sensor configurations are harmonized via OpenSim-based workflows and scripted preprocessing; large-scale able-bodied corpora provide pretraining, with transfer and light subject-specific adaptation under stringent data budgets.

The formulation emphasizes implicit personalization at the user level: predictive accuracy is improved with minimal per-participant supervision relative to the pretraining regime, as reported in the American Society of Biomechanics abstract and poster materials. ASB 2025 paper (PDF) · Poster

The following studies evaluate whether commodity pose-estimation software can support quantitative kinematic analysis in stroke at cohort scales that are impractical for laboratory motion capture alone. Data volumes span large numbers of reaching actions and video frames, with statistical linkage to clinically relevant compensation patterns.

Reaching kinematics with MediaPipe

A gamified upper-limb reaching protocol was instrumented with OpenCV and MediaPipe; subsequent analysis related estimated kinematics to maladaptive compensation strategies. Results are reported in the Journal of Neuroengineering and Rehabilitation (2025): journal article (Springer).

Fine two-dimensional trajectories versus motion capture

Low-cost hand tracking was compared to a laboratory gold standard on a large set of brief, dexterous tracing movements; agreement metrics support scalable monitoring workflows. Results appear in JMIR Formative Research (2024): journal article (JMIR).