Illumination-aware Foreground Segmentation


This project tackles the problem of background subtraction in videos, even under extremely challenging conditions. We design architectures of convolutional neural networks specifically targeted to counter the aforementioned challenges. We first propose a 3D CNN that models the spatial and temporal information of the scene simultaneously. The network [Sakkos et al. IEEE Access2019] is deep enough to successfully cover more than 50 different scenes of various conditions with no need for any fine-tuning. These conditions include illumination (day or night), weather (sunny, rainy or snowing), background movements (trees moving from the wind, fountains etc) and others. Next, we propose a data augmentation method [Sakkos et al. JEIM2021, Sakkos et al. IEEE SKIMA2019Best Paper Award] specifically targeted to illumination changes. We show that artificially augmenting the data set with this method significantly improves the segmentation results, even when tested under sudden illumination changes. We also present a post-processing method that exploits the temporal information of the input video. Finally, we propose a complex deep learning model which learns the illumination of the scene and performs background subtraction simultaneously. Based on adversarial machine learning, the model comprises of six sub-networks, three generators and three discriminators. We show that jointly training the model to perform illumination change from bright to dark and vice versa, and background subtraction, yields substantial performance improvements over the state-of-the-art.


  1. Dimitrios Sakkos, Edmond S. L. Ho, Hubert P. H. Shum and Garry Elvin, "Image Editing based Data Augmentation for Illumination-insensitive Background Subtraction"journal , Journal of Enterprise Information Management, accepted, 2020. PDF Preprint code bibtex
  2. Dimitrios Sakkos, Edmond S. L. Ho and Hubert P. H. Shum, "Illumination-aware Multi-task GANs for Foreground Segmentation"journal , IEEE Access, vol 7, pp. 10976-10986, Jan 2019. PDF bibtex
  3. Dimitrios Sakkos, Edmond S. L. Ho and Hubert P. H. Shum, "Illumination-Based Data Augmentation for Robust Background Subtraction Best Paper Award"conference , The International Conference on Software, Knowledge, Information Management and Applications (SKIMA 2019), accepted, Aug 2019. PDF Preprint code bibtex

The Team

Dr. Dimitrios Sakkos

PhD Alumnus, Northumbria University

Dr. Edmond S. L. Ho

Senior Lecturer, Northumbria University

Dr. Hubert P. H. Shum

Associate Professor, Durham University