Emotion Analysis and Transfer for Facial Expressions and Body Movements
Introduction
This research project focuses on the intersection of human-computer interaction (HCI) and generative animation, specifically exploring how to synthesize and control emotional expression across multimodal data [Stef et al. SKIMA2018]. By leveraging advanced machine learning architectures like StarGAN, the work [Chan et al. CGVC2020, Chan and Ho Computers2021] evolves from early data-driven techniques for relative emotion strength in 3D character motion [Ho et al. D2AT2017, Chan et al. CAVW2019, Irimia et al. MIG2019] to sophisticated frameworks for full-body and hand-specific animation. The research addresses a critical gap in digital communication: the ability to automatically translate emotional nuances - such as varying intensities and cross-domain styles - into realistic 3D movements and facial expressions.
Building on this foundational work in synthesis, the project also examines the interpersonal and social dimensions of emotional data. With the introduction of EmotiV [Zhang et al. CHI2026], the research transitions from generating isolated animations to facilitating automatic emotion sharing in collaborative environments, such as online co-watching. By integrating facial expression recognition with real-time feedback loops, the project aims to create more empathetic and immersive digital experiences. Ultimately, this body of work seeks to harmonize technical precision in multimodal synthesis with the practical human need for authentic, shared emotional connection in virtual spaces.
Building on this foundational work in synthesis, the project also examines the interpersonal and social dimensions of emotional data. With the introduction of EmotiV [Zhang et al. CHI2026], the research transitions from generating isolated animations to facilitating automatic emotion sharing in collaborative environments, such as online co-watching. By integrating facial expression recognition with real-time feedback loops, the project aims to create more empathetic and immersive digital experiences. Ultimately, this body of work seeks to harmonize technical precision in multimodal synthesis with the practical human need for authentic, shared emotional connection in virtual spaces.
Publications
The Team