Illumination-aware Foreground Segmentation


Introduction

This project tackles the problem of background subtraction in videos, even under extremely challenging conditions. We design architectures of convolutional neural networks specifically targeted to counter the aforementioned challenges. We first propose a 3D CNN that models the spatial and temporal information of the scene simultaneously. The network [Sakkos et al. IEEE Access2019] is deep enough to successfully cover more than 50 different scenes of various conditions with no need for any fine-tuning. These conditions include illumination (day or night), weather (sunny, rainy or snowing), background movements (trees moving from the wind, fountains etc) and others. Next, we propose a data augmentation method [Sakkos et al. JEIM2021, Sakkos et al. IEEE SKIMA2019Best Paper Award] specifically targeted to illumination changes. We show that artificially augmenting the data set with this method significantly improves the segmentation results, even when tested under sudden illumination changes. We also present a post-processing method that exploits the temporal information of the input video. Finally, we propose a complex deep learning model which learns the illumination of the scene and performs background subtraction simultaneously. Based on adversarial machine learning, the model comprises of six sub-networks, three generators and three discriminators. We show that jointly training the model to perform illumination change from bright to dark and vice versa, and background subtraction, yields substantial performance improvements over the state-of-the-art.

Publications


Dr. Dimitrios Sakkos

PhD Alumnus, Northumbria University
dksakkos@gmail.com

Dr. Edmond S. L. Ho

Senior Lecturer, University of Glasgow
Shu-Lim.Ho@glasgow.ac.uk

Dr. Hubert P. H. Shum

Associate Professor, Durham University
hubert.shum@durham.ac.uk