Dropout as an Implicit Gating Mechanism For Continual Learning

Abstract

In recent years, neural networks have demonstrated an outstanding ability to achieve complex learning tasks across various domains. However, they suffer from the ‘‘catastrophic forgetting’’ problem when they face a sequence of learning tasks, where they forget the old ones as they learn new tasks. This problem is also highly related to the ‘‘stability-plasticity dilemma’’. The more plastic the network, the easier it can learn new tasks, but the faster it also forgets previous ones. Conversely, a stable network cannot learn new tasks as fast as a very plastic network. However, it is more reliable to preserve the knowledge it has learned from the previous tasks. Several solutions have been proposed to overcome the forgetting problem by making the neural network parameters more stable, and some of them have mentioned the significance of dropout in continual learning. However, their relationship has not been sufficiently studied yet. In this paper, we investigate this relationship and show that a stable network with dropout learns a gating mechanism such that for different tasks, different paths of the network are active. Our experiments show that the stability achieved by this implicit gating plays a very critical role in leading to performance comparable to or better than other involved continual learning algorithms to overcome catastrophic forgetting.

Publication
CVPR 2020 Workshop on Continual Learning in Computer Vision (CLVISION), June 14, 2020, Seattle, WA, USA
Iman Mirzadeh
Iman Mirzadeh
Research Assistant

I am a PhD student and Graduate Research Assistant at the Washington State University Embedded and Pervasive Systems Laboratory (EPSL) under supervision of Dr. Hassan Ghasemzadeh. I am interested in the real-world challenges of working with machine learning models such as energy constraints and human-in-the-loop interactions with these models. Specifically, I am focusing on Model Optimization (such as model compression), where my goal is to build more efficient models or use the existing models more efficiently. Before joining EPSL, I was an ML Engineer at Sokhan AI, where we provided accurate and scalable Natural Language Processing (NLP) and Computer Vision (CV) services to businesses.

Hassan Ghasemzadeh
Hassan Ghasemzadeh
Director

Hassan Ghasemzadeh (Zadeh) is an Associate Professor of Computer Science in the School of Electrical Engineering and Computer Science at Washington State University (WSU). Prior to joining WSU in 2014, he was a Research Manager at the UCLA Wireless Health Institute and an Adjunct Professor of Biomedical Informatics at San Diego State University. He received his Ph.D. in Computer Engineering from the University of Texas at Dallas in 2010, and spent the academic year 2010-2011 as a Postdoctoral Fellow at the West Health Institute. He was Founding Chair of Computer Science and Engineering Department at Azad University, Damavand, 2003-2006. He received his M.S. degree in Computer Engineering from University of Tehran, Tehran, Iran, in 2001 and his B.S. degree in Computer Engineering from Sharif University of Technology, Tehran, Iran in 1998. He received the 2019 WSU GPSA Academic Advisor Excellence Award, 2018 NSF CAREER Award, 2018 WSU EECS Early Career Award, 2018 WSU VCEA Outstanding Communication, Connection, and Engagement Award, 2016 NSF CRII Award, and 2011 IEEE RTAS Best Paper Award.