Volume 19, Issue 3 (JIAEEE Vol.19 No.3 2022)
Journal of Iranian Association of Electrical and Electronics Engineers 2022, 19(3): 145-152 |
Back to browse issues page
Download citation:
BibTeX | RIS | EndNote | Medlars | ProCite | Reference Manager | RefWorks
Send citation to:
BibTeX | RIS | EndNote | Medlars | ProCite | Reference Manager | RefWorks
Send citation to:
Ghiassirad H, Aliyari Shoorehdeli M, Farivar F. To Analysis and Compare 21 Weight Constraints in Stochastic Gradient Descent Algorithm Using Kernel Method. Journal of Iranian Association of Electrical and Electronics Engineers 2022; 19 (3) :145-152
URL: http://jiaeee.com/article-1-1276-en.html
URL: http://jiaeee.com/article-1-1276-en.html
Faculty of Electrical Engineering, Department of Mechatronics Engineering, Industrial Control Center of Excellence, APAC Research Group, K. N. Toosi University of Technology
Abstract: (878 Views)
Imposing a constraint on the gradient descend algorithm for the purpose of neural network training with limited weights has several applications such as network transparency, reducing network volume in terms of storage, increasing the level of generalizability. In addition, it speeds up convergence and finds a more accurate answer. In this paper, using the kernel trick as a method for imposing various constraints on the training algorithm, 21 different constraints have been compared those 16 constraints of which are novel and inspired by the existing uncertainty in biological neural networks. There have been no data augmentation or regularization techniques are used to clearly show the effect of each constraint functions. To solve the classification problem of MNIST, CIFAR-10 and CIFAR-100 datasets for each constraint function, 63 experiments have been done. The results show constraint functions have different impact on solving each dataset and specially our biologically inspired constraints may lead to train a more accurate neural network than the other constraints.
Keywords: Deep Neural Network, Biological Neural Network, Constrained Gradient Descent, Regularization, Kernel Method, and Constraint.
Type of Article: Research |
Subject:
Control
Received: 2021/02/25 | Accepted: 2022/02/27 | Published: 2022/09/2
Received: 2021/02/25 | Accepted: 2022/02/27 | Published: 2022/09/2
Send email to the article author
| Rights and permissions | |
 |
This Journal is an open access Journal Licensed under the Creative Commons Attribution-NonCommercial 4.0 International License. (CC BY NC 4.0) |
