Informatics and Applications
2020, Volume 14, Issue 4, pp 55-62
DEEP LEARNING NEURAL NETWORK STRUCTURE OPTIMIZATION
- M. S. Potanin
- K. O. Vayser
- V. A. Zholobov
- V. V. Strijov
Abstract
The paper investigates the optimal model structure selection problem. The model is a superposition of
generalized linear models. Its elements are linear regression, logistic regression, principal components analysis,
autoencoder and neural network. The model structure refers to values of structural parameters that determine the
form of final superposition. This paper analyzes the model structure selection method and investigates dependence
of accuracy, complexity and stability of the model on it. The paper proposes an algorithm for selection of the neural
network optimal structure. The proposed method was tested on real and synthetic data. The experiment resulted in
significant structural complexity reduction of the model while maintaining accuracy of approximation.
[+] References (15)
- Cybenko, G. 1989. Approximation by superpositions of
a sigmoidal function. Math. Control Signal. 2(4):303-314.
- Bakhteev, O. Yu, and V. V. Strijov. 2018. Deep learning
model selection of suboptimal complexity. Automat. Rem.
Contr. 79(8):1474-1488.
- Potanin, M. S., K. O. Vajser, V. A. Zholobov, and
V. V. Strijov. 2019. Prilozhenie k stat'e: vychislitel'nyy eksperiment po vyboru universal'noy modeli i bazovye teoremy [Appendix to the paper: Computational expirement and basic theorems]. Available at:
https: //github.com/MarkPotanin/GeneticOpt (accessed
November 5, 2020).
- Hinton, G. E., and R. R. Salakhutdinov. 2006. Reducing
the dimensionality of data with neural networks. Science
313(5786):504-507.
- LeCun, Y., J. Denker, and S. Solla. 1989. Optimal brain
damage. Adv. Neur In. 2:598-605.
- Hassibi, B., and D. G. Stork. 1992. Second order derivatives for network pruning: Optimal brain surgeon. Adv.
Neur. In. 5:164-171.
- Dong, X., S. Chen, and S. Pan. 2017. Learning to prune
deep neural networks via layer-wise optimal brain surgeon.
Adv. Neur. In. 30:4857-4867.
- Molchanov, P., S. Tyree, T. Karras, T. Aila, and J. Kautz.
2016. Pruning convolutional neural networks for resource
Received December 2, 2019
efficient transfer learning. arXiv:1611.06440 [cs.LG]. Available at: https://arxiv.org/abs/1611.06440 (accessed November 5, 2020).
- Chaber, P., and M. Lawrynczuk. 2018. Pruning of recurrent neural models: an optimal brain damage approach. Nonlinear Dynam. 92(2):763-780.
- Elsken, T, J. H. Metzen, and F. Hutter. 2018. Neural architecture search: A survey. arXiv:1808.05377 [stat.ML]. Available at: https://arxiv.org/abs/1808.05377 (accessed November 5, 2020).
- Hutter, F, L. Kotthoff, and J. Vanschoren. 2019. Automated machine learning-methods, systems, challenges. Springer. 223 p.
- Rao, C. R. 1973. Linear statistical inference and its applications. Vol. 2. New York, NY: Wiley. 548 p.
- UCI Machine Learning Repository. Available at: http:// archive.ics.uci.edu/ml (accessed November 5, 2020).
- Katrutsa, A. M., and V. V. Strijov. 2015. Stress test procedure for feature selection algorithms. Chemometr Intell. Lab. 142:172-183.
- Glorot, X., and Yo. Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. 13th Conference (Internatioanl) on Artificial Intelligence and Statistics Proceedings. 249-256.
[+] About this article
Title
DEEP LEARNING NEURAL NETWORK STRUCTURE OPTIMIZATION
Journal
Informatics and Applications
2020, Volume 14, Issue 4, pp 55-62
Cover Date
2020-12-30
DOI
10.14357/19922264200408
Print ISSN
1992-2264
Publisher
Institute of Informatics Problems, Russian Academy of Sciences
Additional Links
Key words
model selection; linear models; autoencoders; neural networks; structure; genetic alghorithm
Authors
M. S. Potanin  , K. O. Vayser , V.A. Zholobov , and V. V. Strijov , 
Author Affiliations
Moscow Institute of Physics and Technology, 9 Institutskiy Per., Dolgoprudny, Moscow Region 141700, Russian
Federation
A. A. Dorodnicyn Computing Center, Federal Research Center "Computer Science and Control" of the Russian
Academy of Sciences, 40 Vavilov Str., Moscow 119333, Russian Federation
|