Improved Knowledge Distillation via Teacher Assistant

Abstract

Despite the fact that deep neural networks are powerful models and achieve appealing results on many tasks, they are too gigantic to be deployed on edge devices like smart-phones or embedded sensor nodes. There has been efforts to compress these networks, and a popular method is knowledge distillation, where a large (teacher) pre-trained network is used to train a smaller ( student) network. However, in this paper, we show that the student network performance degrades when the gap between student and teacher is large. Given a fixed student network, one cannot employ an arbitrarily large teacher, or in other words, a teacher can effectively transfer its knowledge to students up to a certain size, not smaller. To alleviate this shortcoming, we introduce multi-step knowledge distillation which employs an intermediate-sized network (teacher assistant) to bridge the gap between the student and the teacher. Moreover, we study the effect of teacher assistant size and extend the framework to multi-step distillation. Theoretical analysis and extensive experiments on CIFAR-10, CIFAR-100 and ImageNet datasets and on CNN and ResNet architectures substantiate the effectiveness of our proposed approach.

Publication
AAAI Conference on Artificial Intelligence (AAAI) 2020
Iman Mirzadeh
Iman Mirzadeh
Research Assistant

I am a PhD student and Graduate Research Assistant at the Washington State University Embedded and Pervasive Systems Laboratory (EPSL) under supervision of Dr. Hassan Ghasemzadeh. I am interested in the real-world challenges of working with machine learning models such as energy constraints and human-in-the-loop interactions with these models. Specifically, I am focusing on Model Optimization (such as model compression), where my goal is to build more efficient models or use the existing models more efficiently. Before joining EPSL, I was an ML Engineer at Sokhan AI, where we provided accurate and scalable Natural Language Processing (NLP) and Computer Vision (CV) services to businesses.

Hassan Ghasemzadeh
Hassan Ghasemzadeh
Director

Hassan Ghasemzadeh (Zadeh) is an Associate Professor of Computer Science in the School of Electrical Engineering and Computer Science at Washington State University (WSU). Prior to joining WSU in 2014, he was a Research Manager at the UCLA Wireless Health Institute and an Adjunct Professor of Biomedical Informatics at San Diego State University. He received his Ph.D. in Computer Engineering from the University of Texas at Dallas in 2010, and spent the academic year 2010-2011 as a Postdoctoral Fellow at the West Health Institute. He was Founding Chair of Computer Science and Engineering Department at Azad University, Damavand, 2003-2006. He received his M.S. degree in Computer Engineering from University of Tehran, Tehran, Iran, in 2001 and his B.S. degree in Computer Engineering from Sharif University of Technology, Tehran, Iran in 1998. He received the 2019 WSU GPSA Academic Advisor Excellence Award, 2018 NSF CAREER Award, 2018 WSU EECS Early Career Award, 2018 WSU VCEA Outstanding Communication, Connection, and Engagement Award, 2016 NSF CRII Award, and 2011 IEEE RTAS Best Paper Award.