Structure Learning for Deep Neural Networks with Competitive Synaptic Pruning

Ahmadi, A.; Mahboobi Esfanjani, R.

doi:10.22061/jecei.2024.11017.758

Document Type : Original Research Paper

Authors

Department of Electrical Engineering, Sahand University of Technology, Tabriz, Iran.

https://doi.org/10.22061/jecei.2024.11017.758

Abstract

Background and Objectives: A predefined structure is usually employed for deep neural networks, which results in over- or underfitting, heavy processing load, and storage overhead. Training along with pruning can decrease redundancy in deep neural networks; however, it may lead to a decrease in accuracy.
Methods: In this note, we provide a novel approach for structure optimization of deep neural networks based on competition of connections merged with brain-inspired synaptic pruning. The efficiency of each network connection is continuously assessed in the proposed scheme based on the global gradient magnitude criterion, which also considers positive scores for strong and more effective connections and negative scores for weak connections. But a connection with a weak score is not removed quickly; instead, it is eliminated when its net score reaches a predetermined threshold. Moreover, the pruning rate is obtained distinctly for each layer of the network.
Results: Applying the suggested algorithm to a neural network model of a distillation column in a noisy environment demonstrates its effectiveness and applicability.
Conclusion: The proposed method, which is inspired by connection competition and synaptic pruning in the human brain, enhances learning speed, preserves accuracy, and reduces costs due to its smaller network size. It also handles noisy data more efficiently by continuously assessing network connections

Keywords

Main Subjects

Machine Learning

Open Access

This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit: http://creativecommons.org/licenses/by/4.0/

Publisher’s Note

JECEI Publisher remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Publisher

Shahid Rajaee Teacher Training University

References

[1] Z. Allen-Zhu, Y. Li, Y. Liang, "Learning and overparameterized neural networks, going beyond two layers," Adv. Neural Inf. Process. Syst., 32, 2019.

[2] C. Cortes, X. Gonzalvo, V. Kuznetsov, M. Mohri, S. Yang, "AdaNet: Adaptive structural learning of artificial neural networks," in Proc. 34th International Conference on Machine Learning, 70: 874-883, 2017.

[3] O. A. Montesinos L., A. Montesinos L., J. C. Montesinos, Multivariate Statistical Machine Learning Methods for Genomic Prediction, Springer, 2022.

[4] S. Li, T. Hoefler, "Chimera: efficiently training large-scale neural networks with bidirectional pipelines," in Proc. International Conference for High Performance Computing, Networking, Storage and Analysis, 2021.

[5] V. Sze, Y. H. Chen, T. J. Yang, J. Emer, "Efficient processing of deep neural networks: A tutorial and survey," arXiv preprint arXiv:1703.09039, 2017.

[6] S. A. Janowsky, "Pruning versus clipping in neural networks," Phys. Rev. A, 39(12): 6600-6603, 1989.

[7] G. Chechik, I. Meilijson, E. Ruppin, "Neuronal regulation: a biologically plausible mechanism for efficient synaptic pruning in development," Neurocomputing, 26-27: 633-639, 1999.

[8] M. V. Johnston et al., "Plasticity and injury in the developing brain," Dev. Neurosci., 31: 1-10, 2009.

[9] G. Hinton, O. Vinyals, J. Dean, "Distilling knowledge in a neural network," arXiv preprint arXiv:1503.02531, 2015.

[10] S. Han, J. Pool, J. Tran, W. J. Dally, "Learning both weights and connections for efficient neural networks," Adv. Neural Inf. Process. Syst., 2015: 1135–1143, 2015.

[11] P. Molchanov et al., "Importance estimation for neural network pruning," arXiv preprint arXiv:1906.10771, 2019.

[12] P. Molchanov et al., "Pruning convolutional neural networks for resource-efficient inference," in Proc. ICLR 2017, 2017.

[13] Y. LeCun et al., "Optimal brain damage," Adv. Neural Inf. Process. Syst., 2, 1990.

[14] B. Hassibi, D. G. Stork, "Second-order derivatives for network pruning: Optimal Brain Surgery," Adv. Neural Inf. Process. Syst., 5: 164-171, 1993.

[15] Y. He et al., "Filter pruning via geometric median for deep convolutional neural network acceleration," in Proc. CVPR 2019: 4340-4349, 2019.

[16] R. Yu et al., "NISP: pruning networks using neuron importance score propagation," in Proc. IEEE CVPR 2018: 9194-9203, 2018.

[17] H. Li et al., "Pruning filters for efficient convolutional neural networks," arXiv preprint arXiv:1608.08710, 2016.

[18] Y. He et al., "Soft filter pruning for accelerating deep convolutional neural networks," arXiv preprint arXiv:1808.06866, 2018.

[19] N. Srivastava et al., "Dropout: A simple way to prevent neural networks from overfitting," J. Mach. Learn. Res., 15: 1929-1958, 2014.

[20] Z. Liu et al., "Dropout reduces underfitting," in Proc. 40th International Conference on Machine Learning, 202, 2023.

[21] X. Sun et al., "MeProp: Sparsified back propagation for accelerated deep learning with reduced overfitting," arXiv preprint arXiv:1706.06197, 2017.

[22] S. A. Mirjalili, Evolutionary Algorithms and Neural Networks, Springer, 2019.

[23] M. Kotyrba et al., "The influence of genetic algorithms on learning possibilities of artificial neural networks," Computers, 2022.

[24] F. Zhao et al., "Towards a brain-inspired developmental neural network by adaptive synaptic pruning," in Proc. ICONIP 2017, 2017.

[25] A. Ahmadi, B. Mashoufi, "A new optimized approach for artificial neural network training using genetic algorithms and parallel processing," Int. Rev. Comput. Softw., 7(5): 1828-6003, 2012.

[26] M. Morini et al., "Strategies and tools for studying microglial-mediated synapse elimination and refinement," Front. Immunol., 2021.

[27] R. P. Guidorzi et al., "The range error test in the structural identification of linear multivariable systems," IEEE Trans. Autom. Control, AC-27: 1044-1054, 1982.

[28] A. Saha and S. Patra, "Modeling and control of distillation column using ANFIS," J. Autom.Control, 51: 51-59, 2021.

[29] H. Zhao, J. Zhang, M. Wang, "Modeling and optimization of distillation column processes using neural networks and genetic algorithms," IEEE Access, 7: 76345-76354, 2019.

[30] A. Singh, P. Gupta, R. Verma, “Hybrid ANFIS and neural network-based approach for fault detection in industrial distillation processes,” Appl. Soft Comput., 96, 106703.

LETTERS TO EDITOR

Journal of Electrical and Computer Engineering Innovations (JECEI) welcomes letters to the editor for the post-publication discussions and corrections which allows debate post publication on its site, through the Letters to Editor. Letters pertaining to manuscript published in JECEI should be sent to the editorial office of JECEI within three months of either online publication or before printed publication, except for critiques of original research. Following points are to be considering before sending the letters (comments) to the editor.

[1] Letters that include statements of statistics, facts, research, or theories should include appropriate references, although more than three are discouraged.

[2] Letters that are personal attacks on an author rather than thoughtful criticism of the author’s ideas will not be considered for publication.

[3] Letters can be no more than 300 words in length.

[4] Letter writers should include a statement at the beginning of the letter stating that it is being submitted either for publication or not.

[5] Anonymous letters will not be considered.

[6] Letter writers must include their city and state of residence or work.

[7] Letters will be edited for clarity and length.

Name *

Email Address *

Affiliation *

Comments *

Security Code *

Journal of Electrical and Computer Engineering Innovations (JECEI)

Structure Learning for Deep Neural Networks with Competitive Synaptic Pruning

References

References

Send comment about this article

Volume 13, Issue 1
January 2025
Pages 189-196

Structure Learning for Deep Neural Networks with Competitive Synaptic Pruning

References

References

Send comment about this article

Volume 13, Issue 1January 2025Pages 189-196

Volume 13, Issue 1
January 2025
Pages 189-196