SMARTBabies - Sensing Movement using Action Recognition Technology in Babies


Introduction

The pursuit of early diagnosis of cerebral palsy has been an active research area with some very promising results using tools such as the General Movements Assessment (GMA). In this project, we explore the feasibility of extracting pose-based features from video sequences to automatically classify infant body movement into two categories, normal and abnormal. The classification was based upon the GMA, which was carried out on the video data by an independent expert reviewer. We proposed two pose-based feature sets, Histograms of Joint Orientation 2D (HOJO2D) and Histograms of Joint Displacement 2D (HOJD2D) [McCay et al. EMBC2019], for use in our new deep learning architectures [McCay et al. IEEE Access 2020] for classifying infant movements into normal or abnormal categories.

While machine learning-based frameworks have obtained excellent performance in a wide range of visual understanding tasks, most of the existing frameworks can be considered black-box approaches since most of the classification frameworks only output the predicted label without specifying exactly what influences the classification decision. Whilst this is acceptable in typical computer vision tasks, it is less preferable in healthcare applications, since it is essential for the clinicians to verify the prediction as well. In order to make our proposed framework more interpretable, we include a visualization module [McCay et al. BHI2021] as well as making deep learning models more interpretable [Zhu et al. BHI2021] by highlighting the body-parts that are contributing to the classification decision.

Readers are also referred to a closely related project on human motion analysis for healthcare applications.

Publications

  1. Kevin D. McCay, Edmond S. L. Ho, Dimitrios Sakkos, Wai Lok Woo, Claire Marcroft, Patricia Dulson and Nicholas D. Embleton, "Towards Explainable Abnormal Infant Movements Identification: A Body-part Based Prediction and Visualisation Framework"conference , Proceedings of the 2021 IEEE EMBS International Conference on Biomedical & Health Informatics (BHI), accepted, 2021. Preprint video
  2. Manli Zhu, Qianhui Men, Edmond S. L. Ho, Howard Leung and Hubert P. H. Shum, "Interpreting Deep Learning based Cerebral Palsy Prediction with Channel Attention"conference , Proceedings of the 2021 IEEE EMBS International Conference on Biomedical & Health Informatics (BHI), accepted, 2021. Preprint
  3. Kevin McCay, Edmond S. L. Ho, Hubert P. H. Shum, Gerhard Fehringer, Claire Marcroft and Nicholas D. Embleton, "Abnormal Infant Movements Classification with Deep Learning on Pose-based Features" journal , IEEE Access, vol. 8, pp. 51582-51592, 2020. PDF code bibtex
  4. Kevin D. McCay, Edmond S. L. Ho, Claire Marcroft and Nicholas D. Embleton, "Establishing Pose Based Features Using Histograms for the Detection of Abnormal Infant Movements" conference, 41th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pp. 5469-5472, July 2019. PDF code bibtex

The Team

Kevin D. McCay

PhD Student, Northumbria University
kevin.d.mccay@northumbria.ac.uk

Dr. Edmond S. L. Ho

Senior Lecturer, Northumbria University
e.ho@northumbria.ac.uk

Claire Marcroft

Neonatal Physiotherapist, Newcastle upon Tyne Hospitals, NHS Foundation Trust/Newcastle University
Claire.Marcroft@newcastle.ac.uk

Patricia Dulson

Clinical Specialist Physiotherapist in Neonates, Newcastle upon Tyne Hospitals, NHS Foundation Trust
p.dulson@nhs.net

Prof. Nicholas D. Embleton

Consultant Neonatal Paediatrician and Professor of Neonatal Medicine, Newcastle upon Tyne Hospitals, NHS Foundation Trust/Newcastle University
nicholas.embleton@newcastle.ac.uk

Dr. Hubert P. H. Shum

Associate Professor, Durham University
hubert.shum@durham.ac.uk


Ethical Approval

Pilot study into Human Activity Recognition and Classification Techniques for the Early Detection of Movement Difficulties in Infants (IRAS project ID: 252317, REC reference: 19/LO/0606)