Multimodal AI Researcher & Engineer

MR.HuBo explores scalable supervised human-to-robot motion retargeting by reversing the usual data-pairing direction. Instead of requiring large collections of matched human and robot demonstrations, robot poses are sampled first, mapped to corresponding human-body configurations, and filtered using a learned human body prior to remove implausible or extreme poses. The resulting paired data supports a two-stage supervised retargeting pipeline using continuous 6D rotation representations.

The system was demonstrated in real time from RGB-based human pose input to physical humanoid motion. This work became my first-author paper at IEEE/RSJ IROS 2024.

My Master's thesis extended motion retargeting beyond kinematic mapping by asking a different question: when a motion is mechanically feasible, does it still communicate the intended action to a human observer? I investigated semantic post-refinement through Reinforcement Learning from Human Feedback (RLHF), using pairwise human preferences to learn a reward model and PPO to refine retargeted robot motion toward human-perceived intent.

The refined and non-refined motions were evaluated in a randomized, blind human study with 20 participants using 1–7 Likert ratings for similarity and suitability relative to the intended action description. Statistical analysis used ANOVA and Tukey post-hoc testing with p < 0.05.

menu_bookThesis

As a later exploratory project, I investigated whether language could provide a more scalable semantic signal for human motion. I implemented Transformer-based human-motion and text representation components and explored cosine similarity as a measure of semantic correspondence between natural-language descriptions and human motion.

This was a short exploratory prototype. The robot-side bridge, conversion of semantic similarity into a functioning robot-domain reward, robot policy adaptation, and a complete feedback-learning loop were not implemented and remained conceptual extensions.

Developed and iterated an industrial computer-vision pipeline for semiconductor wafer defect detection across 11 defect classes. The work covered dataset engineering, object-detection training, model comparison, threshold/error analysis, and refinement using YOLO and RT-DETR.

The documented project performance improved from 66% to 98.9% over the course of model and data iteration.