Modelling Two-person and Human-object Close Interactions
Introduction
This project focuses on developing computational methods to represent, analyze, and synthesize contact-rich interactions in a physically and semantically meaningful way. At its core, the work models interactions through spatial relationships between body parts and between humans and objects, preserving contact constraints and relative geometry during motion adaptation. Early contributions established multi-resolution and spatial relationship-preserving motion adaptation frameworks [Ho et al. SIGGRAPH2010, Ho et al. VRST2014], enabling realistic modification of close character interactions while maintaining structural consistency. These ideas were extended to interactive partner control [Ho et al. ACM TOMM2013] and constrained motion adaptation [Ho and Shum ICRA2013], forming a principled foundation for handling interdependent motion in both virtual characters and humanoid robots.
Building on this geometric foundation, the project integrates learning-based approaches for recognition and synthesis. Skeleton-aware graph convolutional networks and two-stream recurrent architectures model the structural dependencies and temporal dynamics inherent in human–human [Men et al. ICPR2020] and human–object [Zhu et al. IEEE TIM2024, Zhu et al. IEEE SMC2022] interactions. Geometric feature enhancement further improves interaction detection by explicitly encoding relational cues between entities. On the generative side, conditional and hierarchical GAN frameworks - with class-aware discriminators [Men et al. C&G2022] and multi-hot embeddings - enable controllable synthesis, augmentation, and "mix-and-match" [Goel et al. CGF2022] recombination of close interactions. A more recent method [Li et al. CVPRW2024] incorporates skeleton priors to guide data augmentation and improve realism and generalization. Together, these methodological advances unify geometry-aware modeling, relational learning, and controllable generative frameworks to systematically address recognition, comparison [Shen et al. IEEE TVCG2020, Ho et al. VRST2015], adaptation, and synthesis [Yin et al. IEEE TVCG2019] of close interactions.
Readers are also referred to a closely related project on topology-based motion analysis and synthesis.
Building on this geometric foundation, the project integrates learning-based approaches for recognition and synthesis. Skeleton-aware graph convolutional networks and two-stream recurrent architectures model the structural dependencies and temporal dynamics inherent in human–human [Men et al. ICPR2020] and human–object [Zhu et al. IEEE TIM2024, Zhu et al. IEEE SMC2022] interactions. Geometric feature enhancement further improves interaction detection by explicitly encoding relational cues between entities. On the generative side, conditional and hierarchical GAN frameworks - with class-aware discriminators [Men et al. C&G2022] and multi-hot embeddings - enable controllable synthesis, augmentation, and "mix-and-match" [Goel et al. CGF2022] recombination of close interactions. A more recent method [Li et al. CVPRW2024] incorporates skeleton priors to guide data augmentation and improve realism and generalization. Together, these methodological advances unify geometry-aware modeling, relational learning, and controllable generative frameworks to systematically address recognition, comparison [Shen et al. IEEE TVCG2020, Ho et al. VRST2015], adaptation, and synthesis [Yin et al. IEEE TVCG2019] of close interactions.
Readers are also referred to a closely related project on topology-based motion analysis and synthesis.
Publications
The Team