2021 IEEE International Conference on Acoustics, Speech and Signal Processing
6-11 June 2021 • Toronto, Ontario, Canada
Extracting Knowledge from Information
| Paper ID | MLSP-21.5 | ||
| Paper Title | ENHANCING DATA-FREE ADVERSARIAL DISTILLATION WITH ACTIVATION REGULARIZATION AND VIRTUAL INTERPOLATION | ||
| Authors | Xiaoyang Qu, Jianzong Wang, Jing Xiao, Ping An Technology (Shenzhen) Co., Ltd., China | ||
| Session | MLSP-21: Generative Neural Networks | ||
| Location | Gather.Town | ||
| Session Time: | Wednesday, 09 June, 15:30 - 16:15 | ||
| Presentation Time: | Wednesday, 09 June, 15:30 - 16:15 | ||
| Presentation | Poster | ||
| Topic | Machine Learning for Signal Processing: [MLR-TRL] Transfer learning | ||
| IEEE Xplore Open Preview | Click here to view in IEEE Xplore | ||
| Abstract | Knowledge distillation refers to a technique of transferring the knowledge from a large learned model or an ensemble of learned models to a small model. This method relies on access to the original training set, which might not always be available. A possible solution is a data-free adversarial distillation framework, which deploys a generative network to transfer the teacher model's knowledge to the student model. However, the data generation efficiency is low in the data-free adversarial distillation. We add an activation regularizer and a virtual interpolation method to improve the data generation efficiency. The activation regularizer enables the students to match the teacher's predictions close to activation boundaries and decision boundaries. The virtual interpolation method can generate virtual samples and labels in-between decision boundaries. Our experiments show that our approach surpasses state-of-the-art data-free distillation methods. The student model can achieve 95.42% accuracy on CIFAR-10 and 77.05% accuracy on CIFAR-100 without any original training data. Our model's accuracy is 13.8% higher than the state-of-the-art data-free method on CIFAR-100. | ||