2021 IEEE International Conference on Acoustics, Speech and Signal Processing
6-11 June 2021 • Toronto, Ontario, Canada
Extracting Knowledge from Information
| Paper ID | MLSP-14.6 | ||
| Paper Title | LEARNING A TREE OF NEURAL NETS | ||
| Authors | Arman Zharmagambetov, Miguel Á. Carreira-Perpiñán, University of California, Merced, United States | ||
| Session | MLSP-14: Learning Algorithms 1 | ||
| Location | Gather.Town | ||
| Session Time: | Wednesday, 09 June, 13:00 - 13:45 | ||
| Presentation Time: | Wednesday, 09 June, 13:00 - 13:45 | ||
| Presentation | Poster | ||
| Topic | Machine Learning for Signal Processing: [MLR-LEAR] Learning theory and algorithms | ||
| IEEE Xplore Open Preview | Click here to view in IEEE Xplore | ||
| Abstract | Much of the success of deep learning is due to choosing good neural net architectures and being able to train them effectively. A type of architecture that has been long sought is one that combines decision trees and neural nets. This is straightforward if the tree makes soft decisions (i.e., an input instance follows all paths in the tree with different probabilities), because the model is differentiable. However, the optimization is much harder if the tree makes hard decisions, but this produces an architecture that is much faster at inference, since an instance follows a single path in the tree. We show that it is possible to train such architectures, with guaranteed monotonic decrease of the loss, and demonstrate it by learning trees with linear decision nodes and deep nets at the leaves. The resulting architecture improves state-of-the-art deep nets, by achieving comparable or lower classification error but with fewer parameters and faster inference time. In particular, we show that, rather than improving a ResNet by making it deeper, it is better to construct a tree of small ResNets. The resulting tree-net hybrid is also more interpretable. | ||