Volume 22, Issue 2 (JIAEEE Vol.22 No.2 2025) Journal of Iranian Association of Electrical and Electronics Engineers 2025, 22(2): 111-126 | Back to browse issues page

Mendeley

Zotero

RefWorks

Ghavami Z, Mollaie Emamzadeh M, Maghfoori Farsangi M. An intelligent Gradient Method using Particle Swarm Optimization. Journal of Iranian Association of Electrical and Electronics Engineers 2025; 22 (2) :111-126
URL: http://jiaeee.com/article-1-1617-en.html

An intelligent Gradient Method using Particle Swarm Optimization

Zahra Ghavami

, Mohammad Mollaie Emamzadeh ^*

, Maliheh Maghfoori Farsangi

Shahid Bahonar University of Kerman

Abstract: (803 Views)

The Gradient algorithm is the simplest and most widely used method in optimization problems and machine learning. The convergence rate of this method strongly depends on choosing the suitable value for the step size. Choosing a very small value can cause a low convergence rate, on the other hand, choosing a very large value may also cause divergence and oscillation around the optimal point. Usually, the step size is chosen larger in the initial steps of optimization and as the execution steps progress and the optimal solution is approached, its value decreases. The optimal setting of this hyper-parameter is experimentally and by trial and error and has to be done separately for every problem, so it takes a lot of time. On the other hand, in swarm intelligence optimization methods, including the particle swarm optimization (PSO) algorithm, the step size (the length of movement) is automatically adjusted during the execution of the optimization method. Also, in these methods, few parameters need to be tuned and a predetermined range is available for this purpose. In this paper, by combining the PSO and Gradient method, an intelligent optimization method based on the gradient is presented, which does not need to adjust the step size. The performance of the proposed gradient algorithm is evaluated on ten benchmark functions and it is observed that the proposed algorithm has a better convergence rate than the Classic Gradient algorithm in reaching the optimal solution.

Keywords: An intelligent Gradient Method using Particle Swarm Optimization

Full-Text [PDF 1948 kb] (109 Downloads)

Type of Article: Research | Subject: Control
Received: 2023/07/16 | Accepted: 2024/08/11 | Published: 2025/08/15

References

1. [1] O. A. Al-Shahri et al., "Solar photovoltaic energy optimization methods, challenges and issues: A comprehensive review", Journal of Cleaner Production, vol. 284, p. 125465, 2021. [DOI:10.1016/j.jclepro.2020.125465]

2. [2] Nezamabadi-pour, Hossein. Genetic Algorithm: Basic and Advanced Concepts. Kerman: Shahid Bahonar University of Kerman, 2010.

3. [3] R. Bansal, "Optimization methods for electric power systems: An overview", International Journal of Emerging Electric Power Systems, vol. 2, no. 1, 2005. [DOI:10.2202/1553-779X.1021]

4. [4] A. Mustapha, L. Mohamed, and K. Ali, "An overview of gradient descent algorithm optimization in machine learning: Application in the ophthalmology field", in Smart Applications and Data Analysis: Third International Conference, Morocco, 2020, pp. 349-359. [DOI:10.1007/978-3-030-45183-7_27]

5. [5] T.T., Trung, H. T., Nguyen. "Backtracking gradient descent method and some applications in large scale optimisation. Part 2: Algorithms and experiments", Applied Mathematics & Optimization, Vol. 84, no. 3, pp. 2557-2586, 2021. [DOI:10.1007/s00245-020-09718-8]

6. [6] Y., Lei, K., Tang. "Learning rates for stochastic gradient descent with nonconvex objectives", IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 43, no. 12, pp. 4505-4511, 2021. [DOI:10.1109/TPAMI.2021.3068154]

7. [7] Ghiassirad H, Aliyari Shoorehdeli M, Farivar F. "To Analysis and Compare 21 Weight Constraints in Stochastic Gradient Descent Algorithm Using Kernel Method", Journal of Iranian Association of Electrical and Electronics Engineers 2022; 19 (3) :145-152. URL: http://jiaeee.com/article-1-1276-fa.html [DOI:10.52547/jiaeee.19.3.145]

8. [8] S. Ruder, "An overview of gradient descent optimization algorithms", arXiv preprint arXiv:1609.04747, 2016.

9. [9] M. D. Zeiler, "Adadelta: an adaptive learning rate method", arXiv preprint arXiv:1212.5701, 2012.

10. [10] Z. Fu, S.-C. Chu, J. Watada, C.-C. Hu, and J.-S. Pan, "Software and hardware co-design and implementation of intelligent optimization algorithms", Applied Soft Computing, vol. 129, p. 109639, 2022. [DOI:10.1016/j.asoc.2022.109639]

11. [11] C. Blum, "Ant colony optimization: Introduction and recent trends", Physics of Life reviews, vol. 2, no. 4, pp. 353-373, 2005. [DOI:10.1016/j.plrev.2005.10.001]

12. [12] E. Kaya, B. Gorkemli, B. Akay, and D. Karaboga, "A review on the studies employing artificial bee colony algorithm to solve combinatorial optimization problems", Engineering Applications of Artificial Intelligence, vol. 115, p. 105311, 2022. [DOI:10.1016/j.engappai.2022.105311]

13. [13] M. G. Sahab, V. V. Toropov, and A. H. Gandomi, "A review on traditional and modern structural optimization: problems and techniques", Metaheuristic applications in structures and infrastructures, pp. 25-47, 2013. [DOI:10.1016/B978-0-12-398364-0.00002-4]

14. [14] A. G. Hussien et al., "Crow search algorithm: theory, recent advances, and applications", IEEE Access, vol. 8, pp. 173548-173565, 2020. [DOI:10.1109/ACCESS.2020.3024108]

15. [15] S. Mirjalili, S. M. Mirjalili, and A. Lewis, "Grey wolf optimizer", Advances in engineering software, vol. 69, pp. 46-61, 2014. [DOI:10.1016/j.advengsoft.2013.12.007]

16. [16] E. Rashedi, H. Nezamabadi-Pour, and S. Saryazdi, "GSA: a gravitational search algorithm", Information sciences, vol. 179, no. 13, pp. 2232-2248, 2009. [DOI:10.1016/j.ins.2009.03.004]

17. [17] H. Zhang, J. Li, M. Hong, and Y. Man, "Artificial intelligence algorithm-based multi-objective optimization model of flexible flow shop smart scheduling", Applications of Artificial Intelligence in Process Systems Engineering, pp. 447-472, 2021. [DOI:10.1016/B978-0-12-821092-5.00008-5]

18. [18] B. A. S. Emambocus, M. B. Jasser, and A. Amphawan, "A survey on the optimization of artificial neural networks using swarm intelligence algorithms", IEEE Access, vol. 11, pp. 1280-1294, 2023. [DOI:10.1109/ACCESS.2022.3233596]

19. [19] M. Jain, V. Saihjpal, N. Singh, and S. B. Singh, "An Overview of Variants and Advancements of PSO Algorithm", Applied Sciences, vol. 12, no. 17, p. 8392, 2022. [DOI:10.3390/app12178392]

20. [20] J. Kennedy and R. Eberhart, "Particle swarm optimization", in Proceedings of ICNN'95-international conference on neural networks, 1995, vol. 4: IEEE, pp. 1942-1948. [DOI:10.1109/ICNN.1995.488968]

21. [21] L. Zhen, Y. Liu, W. Dongsheng, and Z. Wei, "Parameter estimation of software reliability model and prediction based on hybrid wolf pack algorithm and particle swarm optimization", IEEE Access, vol. 8, pp. 29354-29369, 2020. [DOI:10.1109/ACCESS.2020.2972826]

22. [22] R. Padmanabhan and P. Seiler, "Analysis of Gradient Descent with Varying Step Sizes using Integral Quadratic Constraints", arXiv preprint arXiv:2210.00644, 2022.

23. [23] M. Bazaraa, H. Sherali, and C. Shetty, Nonlinear theory and algorithm, 3rd ed. USA: Wiley-Interscience, 2006.

24. [24] S. Biswas, S. Nath, S. Dey, and U. Majumdar, "Tangent-cut optimizer on gradient descent: an approach towards Hybrid Heuristics", Artificial Intelligence Review, pp. 1-27, 2022.

25. [25] E. Taş and M. MEMMEDLİ, "Near optimal step size and momentum in gradient descent for quadratic functions", Turkish Journal of Mathematics, vol. 41, no. 1, pp. 110-121, 2017. [DOI:10.3906/mat-1411-51]

26. [26] S. Guan and B. Biswal, "Spline adaptive filtering algorithm based on different iterative gradients: Performance analysis and comparison", Journal of Automation and Intelligence, vol. 2, no. 1, pp. 1-13, 2023. [DOI:10.1016/j.jai.2022.100008]

27. [27] Z. Gu, K. Jin, Y. Meng, L. Xue, and L.-H. Zhang, "On Kahan's automatic step-size control and an anadromic gradient descent iteration", Technical report, 2022, URL https://www. researchgate. net/publication/362126629_ On_Kahan's_automatic_ step-size_control_and_an_ anadromic_gradient_descent_iteration, 2022.

28. [28] A. Soodabeh and V. Manfred, "A learning rate method for full-batch gradient descent", Műszaki Tudományos Közlemények, vol. 13, no. 1, pp. 174-177, 2020. [DOI:10.33894/mtk-2020.13.33]

29. [29] Y. Qiao, B. van Lew, B. P. Lelieveldt, and M. Staring, "Fast automatic step size estimation for gradient descent optimization of image registration", IEEE transactions on medical imaging, vol. 35, no. 2, pp. 391-403, 2015. [DOI:10.1109/TMI.2015.2476354]

30. [30] Y. Xue, Y. Wang, and J. Liang, "A self-adaptive gradient descent search algorithm for fully-connected neural networks", Neurocomputing, vol. 478, pp. 70-80, 2022. [DOI:10.1016/j.neucom.2022.01.001]

31. [31] R. Padmanabhan and P. Seiler, "Analysis of Gradient Descent with Varying Step Sizes using Integral Quadratic Constraints", arXiv preprint arXiv : 2210.00644, 2022.

32. [32] M. Ravaut and S. Gorti, "Gradient descent revisited via an adaptive online learning rate", arXiv preprint arXiv:1801.09136, 2018.

33. [33] K. Zeng, J. Liu, Z. Jiang, and D. Xu, "A decreasing scaling transition scheme from Adam to SGD", Advanced Theory and Simulations, vol. 5, no. 7, p. 2100599, 2022. [DOI:10.1002/adts.202100599]

34. [34] X. Wei and H. Huang, "A survey on several new popular swarm intelligence optimization algorithms", Research Square, 2023, doi: 10.21203/rs.3.rs-2450545/v1. [DOI:10.21203/rs.3.rs-2450545/v1]

35. [35] F. Parandeh Motlagh, V. Khatibi Bardsiri, and A. Khatibi Bardsiri, "Human-Whale cooperation optimization (HWO) algorithm: A metaheuristic algorithm for solve optimization problems", International Journal of Nonlinear Analysis and Applications, vol. 14, no. 1, pp. 2279-2300, 2023.

36. [36] P. K. Gupta, B. Yadav, A. Kumar, and S. K. Himanshu, "Machine learning and artificial intelligence application in constructed wetlands for industrial effluent treatment: advances and challenges in assessment and bioremediation modeling", Bioremediation for Environmental Sustainability, pp. 403-414, 2021. [DOI:10.1016/B978-0-12-820524-2.00016-X]

37. [37] N. H. Phong, A. Santos, and B. Ribeiro, "PSO-convolutional neural networks with heterogeneous learning rate", IEEE Access, vol. 10, pp. 89970-89988, 2022. [DOI:10.1109/ACCESS.2022.3201142]

38. [38] D. A. Ejigu and X. Liu, "Gradient descent-particle swarm optimization based deep neural network predictive control of pressurized water reactor power", Progress in Nuclear Energy, vol. 145, p. 104108, 2022. [DOI:10.1016/j.pnucene.2021.104108]

39. [39] S. Liu and X. Huang, "Multi‐Objective Optimization for the Cross Brace of a Computer Numerical Control Gantry Machine Tool Based on Intelligent Algorithms", Advanced Intelligent Systems, vol. 5, no. 3, p. 2200368, 2023. [DOI:10.1002/aisy.202200368]

40. [40] A. Eleyan, M. S. Salman, and B. Al-Sheikh, "Application of optimization algorithms for classification problem", International Journal of Electrical and Computer Engineering, vol. 12, no. 4, p. 4373, 2022. [DOI:10.11591/ijece.v12i4.pp4373-4379]

41. [41] A. Mehboodi, H. Nezamabadi-pour, M. Soleimanpour-moghadam, "Multi-robot Path Planning in a 3D Environment by Modified Particle Swarm Optimization Algorithm", Journal of Iranian Association of Electrical and Electronics Engineers 2022; 19 (3) :163-174. URL: http://jiaeee.com/article-1-1073-en.html [DOI:10.52547/jiaeee.19.3.163]

42. [42] M. Vakilifard, A. Sahafi, A. M. Rahmani, P. Sheikholharam Mashhadi, "FRA-PSO: A two-stage Resource Allocation Algorithm in Cloud Computing", Journal of Iranian Association of Electrical and Electronics Engineers 2023; 20 (1) :43-49. URL: http://jiaeee.com/article-1-1237-fa.html [DOI:10.52547/jiaeee.20.1.43]

43. [43] M. P. Deisenroth, A. A. Faisal, and C. S. Ong, Mathematics for machine learning. Cambridge University Press, 2020. [DOI:10.1017/9781108679930]

44. [44] F. A. Hashim, E. H. Houssein, K. Hussain, M. S. Mabrouk, and W. Al-Atabany, "Honey Badger Algorithm: New metaheuristic algorithm for solving optimization problems", Mathematics and Computers in Simulation, vol. 192, pp. 84-110, 2022. [DOI:10.1016/j.matcom.2021.08.013]

45. [45] M. J. Goldanloo and F. S. Gharehchopogh, "A hybrid OBL-based firefly algorithm with symbiotic organisms search algorithm for solving continuous optimization problems", The Journal of Supercomputing, vol. 78, no. 3, pp. 3998-4031, 2022. [DOI:10.1007/s11227-021-04015-9]

46. [46] M. Dehghani, Š. Hubálovský, and P. Trojovský, "Tasmanian devil optimization: a new bio-inspired optimization algorithm for solving optimization algorithm", IEEE Access, vol. 10, pp. 19599-19620, 2022. [DOI:10.1109/ACCESS.2022.3151641]

Send email to the article author

Rights and permissions
	This Journal is an open access Journal Licensed under the Creative Commons Attribution-NonCommercial 4.0 International License. (CC BY NC 4.0)