Volume 22, Issue 2 (JIAEEE Vol.22 No.2 2025) Journal of Iranian Association of Electrical and Electronics Engineers 2025, 22(2): 3-16 | Back to browse issues page

Mendeley

Zotero

RefWorks

Roshanravan S, Shamaghdari S. Integrated Fault-Tolerant Attitude Control of Quadrotor Using Reinforcement Learning. Journal of Iranian Association of Electrical and Electronics Engineers 2025; 22 (2) :3-16
URL: http://jiaeee.com/article-1-1608-en.html

Integrated Fault-Tolerant Attitude Control of Quadrotor Using Reinforcement Learning

Sajad Roshanravan

, Saeed Shamaghdari ^*

Iran University of Science and Technology (IUST)

Abstract: (899 Views)

This paper deals with the optimal attitude control problem for quadrotor unmannd air vehicles with unknown nonlinear dynamics subject to component and actuator faults. The proposed integrated optimal fault tolerant control (FTC) scheme is based on reinforcement learning (RL) algorithm, without requiring prior knowledge of the system dynamics. To solve the Hamilton-Jacobi-Bellman (HJB) equation, an identifier-critic-based online RL strategy is employed with a dual neural network (NN) approximation structure. The forgetting factor in the proposed identifier update law is variable and dependent on the state estimation errors and measurement noise estimation. Choosing this variable forgetting factor increases the convergence speed and decreases the estimation error of identifier NN weights compared to the constant one while maintaining its robustness. When a fault occurs, the system continues to operate under the former control policy until the fault is detected. On the other hand, the optimal control design in the RL framework requires the initial stabilizing control condition. In order to make it possible to initiate the control learning process from the former applied FTC, this condition is relaxed by leveraging a stabilizing term in the critic update law. The Uniformly Ultimately Boundedness (UUB) of identifier and critic NN weight errors and, as a result, the convergence of the control input to the neighborhood of the optimal solution are all proved by Lyapunov theory. In the proposed method, changes in the values of faults are detected by comparing the HJB error to a predefined threshold. Finally, the simulation results are given to validate the effectiveness of the developed method.

Keywords: Quadrature Attitude Control, Component and Actuator Faults, Fault-Tolerant Optimal Control, Fault Detection, Reinforcement Learning

Full-Text [PDF 1993 kb] (137 Downloads)

Type of Article: Research | Subject: Control
Received: 2023/06/19 | Accepted: 2024/01/9 | Published: 2025/08/15

References

1. [1] D. K. Villa, A. S. Brandao, M. Sarcinelli-Filho, "A survey on load transportation using multirotor UAVs", Journal of Intelligent & Robotic Systems, vol. 98, no. 2, pp. 267-296, 2020. [DOI:10.1007/s10846-019-01088-w]

2. [2] Yousefi S A, Kazemi M H. Control of Quadcopter with Suspended Load by Internal Model Control and Input Shaping for Reducing the Load Oscillation. Journal of Iranian Association of Electrical and Electronics Engineers 2022; 19 (3) :215-226. [DOI:10.52547/jiaeee.19.3.215]

3. [3] Barshooi A H, Amirkhani A. Cooperative Control of Mobile Robots in Creating a Runway Platform for Quadrotor Landing. Journal of Iranian Association of Electrical and Electronics Engineers 2023; 20 (1) :153-162. [DOI:10.52547/jiaeee.20.1.153]

4. [4] W. Zhao, H. Liu, F. L. Lewis, "Data-driven fault-tolerant control for attitude synchronization of nonlinear quadrotors", IEEE Transactions on Automatic Control, vol. 66, no. 11, pp. 5584-5591, 2021. [DOI:10.1109/TAC.2021.3053194]

5. [5] A. A. Amin, K. M. Hasan, "A review of fault tolerant control systems: advancements and applications", Measurement, vol. 143, pp. 58-68, 2019. [DOI:10.1016/j.measurement.2019.04.083]

6. [6] C. Liu, B. Jiang, R. J. Patton, K. Zhang, "Decentralized output sliding-mode fault-tolerant control for heterogeneous multiagent systems", IEEE transactions on cybernetics, vol. 50, no. 12, pp. 4934-4945, 2019. [DOI:10.1109/TCYB.2019.2912636]

7. [7] S. Roshanravan, Sobhani B. Gendeshmin, S. Shamaghdari, "Design of an actuator fault-tolerant controller for an air vehicle with nonlinear dynamics", Proceedings of the Institution of Mechanical Engineers, Part G: Journal of Aerospace Engineering, vol. 233, no. 10, pp. 3534-3546, 2019. [DOI:10.1177/0954410018801254]

8. [8] S. Roshanravan, S. Shamaghdari, "Simultaneous fault detection and isolation and fault-tolerant control using supervisory control technique: asynchronous switching approach", Proceedings of the Institution of Mechanical Engineers, Part I: Journal of Systems and Control Engineering, vol. 234, no. 8, pp. 900-911, 2020. [DOI:10.1177/0959651819893891]

9. [9] S. Roshanravan, S. Shamaghdari, "Adaptive Fault-Tolerant Tracking Control for Affine Nonlinear Systems With Unknown Dynamics via Reinforcement Learning", IEEE Transactions on Automation Science and Engineering, vol. 21, no. 1, pp. 569-580, 2024. [DOI:10.1109/TASE.2022.3223702]

10. [10] J.-X. Zhang, G.-H. Yang, "Prescribed performance fault-tolerant control of uncertain nonlinear systems with unknown control directions", IEEE Transactions on Automatic Control, vol. 62, no. 12, pp. 6529-6535, 2017. [DOI:10.1109/TAC.2017.2705033]

11. [11] G. Liu, N. Sun, T. Yang, Y. Fang, "Reinforcement Learning-Based Prescribed Performance Motion Control of Pneumatic Muscle Actuated Robotic Arms With Measurement Noises", IEEE Transactions on Systems, Man, and Cybernetics: Systems, 2022. [DOI:10.1109/TSMC.2022.3207575]

12. [12] J.-X. Zhang, G.-H. Yang, "Fault-tolerant output-constrained control of unknown Euler-Lagrange systems with prescribed tracking accuracy", Automatica, vol. 111, pp. 108606, 2020. [DOI:10.1016/j.automatica.2019.108606]

13. [13] H. Zhang, L. Cui, Y. Luo, "Near-optimal control for nonzero-sum differential games of continuous-time nonlinear systems using single-network ADP", IEEE transactions on cybernetics, vol. 43, no. 1, pp. 206-216, 2012. [DOI:10.1109/TSMCB.2012.2203336]

14. [14] Z. Wang, L. Liu, H. Zhang, G. Xiao, "Fault-tolerant controller design for a class of nonlinear MIMO discrete-time systems via online reinforcement learning algorithm", IEEE Transactions on Systems, Man, and Cybernetics: Systems, vol. 46, no. 5, pp. 611-622, 2015. [DOI:10.1109/TSMC.2015.2478885]

15. [15] L. Liu, Z. Wang, H. Zhang, "Adaptive fault-tolerant tracking control for MIMO discrete-time systems via reinforcement learning algorithm with less learning parameters", IEEE Transactions on Automation Science and Engineering, vol. 14, no. 1, pp. 299-313, 2016. [DOI:10.1109/TASE.2016.2517155]

16. [16] Z. Wang, L. Liu, Y. Wu, H. Zhang, "Optimal fault-tolerant control for discrete-time nonlinear strict-feedback systems based on adaptive critic design", IEEE Transactions on Neural Networks and Learning Systems, vol. 29, no. 6, pp. 2179-2191, 2018. [DOI:10.1109/TNNLS.2018.2810138]

17. [17] Q.-Y. Fan, G.-H. Yang, "Active complementary control for affine nonlinear control systems with actuator faults", IEEE Transactions on Cybernetics, vol. 47, no. 11, pp. 3542-3553, 2016. [DOI:10.1109/TCYB.2016.2569406]

18. [18] Q. Y. Fan, G. H. Yang, "Adaptive fault‐tolerant control for affine non‐linear systems based on approximate dynamic programming", IET Control Theory & Applications, vol. 10, no. 6, pp. 655-663, 2016. [DOI:10.1049/iet-cta.2015.1081]

19. [19] H. Jiang, H. Zhang, Y. Liu, J. Han, "Neural-network-based control scheme for a class of nonlinear systems with actuator faults via data-driven reinforcement learning method", Neurocomputing, vol. 239, pp. 1-8, 2017. [DOI:10.1016/j.neucom.2017.01.047]

20. [20] M. E. Dehshalie, M. B. Menhaj, M. Karrari, "Fault tolerant cooperative control for affine multi-agent systems: An optimal control approach", Journal of the Franklin Institute, vol. 356, no. 3, pp. 1360-1378, 2019. [DOI:10.1016/j.jfranklin.2018.09.038]

21. [21] S. Zhang, C. Huang, K. Ji, H. Zhang, "Prescribed performance incremental adaptive optimal fault-tolerant control for nonlinear systems with actuator faults", ISA transactions, vol. 120, pp. 99-109, 2022. [DOI:10.1016/j.isatra.2021.03.011]

22. [22] P. Deptula, Z. I. Bell, E. A. Doucette, J. W. Curtis, W. E. Dixon, "Data-based reinforcement learning approximate optimal control for an uncertain nonlinear system with control effectiveness faults", Automatica, vol. 116, pp. 108922, 2020. [DOI:10.1016/j.automatica.2020.108922]

23. [23] H. Lin, B. Zhao, D. Liu, C. Alippi, "Data-based fault tolerant control for affine nonlinear systems through particle swarm optimized neural networks", IEEE/CAA Journal of Automatica Sinica, vol. 7, no. 4, pp. 954-964, 2020. [DOI:10.1109/JAS.2020.1003225]

24. [24] K. Li, Y. Li, "Adaptive nn optimal consensus fault-tolerant control for stochastic nonlinear multiagent systems", IEEE Transactions on Neural Networks and Learning Systems, 2021. [DOI:10.1109/TFUZZ.2021.3094716]

25. [25] H.-J. Ma, L.-X. Xu, G.-H. Yang, "Multiple environment integral reinforcement learning-based fault-tolerant control for affine nonlinear systems", IEEE transactions on cybernetics, vol. 51, no. 4, pp. 1913-1928, 2019. [DOI:10.1109/TCYB.2018.2889679]

26. [26] K. Zhang, H. Zhang, Z. Gao, H. Su, "Online adaptive policy iteration based fault-tolerant control algorithm for continuous-time nonlinear tracking systems with actuator failures", Journal of the Franklin Institute, vol. 355, no. 15, pp. 6947-6968, 2018. [DOI:10.1016/j.jfranklin.2018.07.009]

27. [27] J. Lan, R. J. Patton, "A new strategy for integration of fault estimation within fault-tolerant control", Automatica, vol. 69, pp. 48-59, 2016. [DOI:10.1016/j.automatica.2016.02.014]

28. [28] F. Sabbghian-Bidgoli, M. Farrokhi, "Polynomial fuzzy observer-based integrated fault estimation and fault-tolerant control with uncertainty and disturbance", IEEE Transactions on Fuzzy Systems, vol. 30, no. 3, pp. 741-754, 2020. [DOI:10.1109/TFUZZ.2020.3048505]

29. [29] A. Mishra, S. Ghosh, "Simultaneous identification and optimal tracking control of unknown continuous-time systems with actuator constraints", International Journal of Control, vol. 95, no. 8, pp. 2005-2023, 2022. [DOI:10.1080/00207179.2021.1890824]

30. [30] S. Pakkhesal, S. Shamaghdari, "Sum‐of‐squares‐based policy iteration for suboptimal control of polynomial time‐varying systems", Asian Journal of Control, vol. 24, no. 6, pp. 3022-3031, 2022. [DOI:10.1002/asjc.2689]

31. [31] T. Dierks, S. Jagannathan, "Optimal control of affine nonlinear continuous-time systems", In Proceedings of the 2010 American control conference, pp. 1568-1573, IEEE, 2010. [DOI:10.1109/ACC.2010.5531586]

32. [32] X. Yang, D. Liu, Q. Wei, "Robust tracking control of uncertain nonlinear systems using adaptive dynamic programming", In International conference on neural information processing, pp. 9-16. Springer, Cham, 2015. [DOI:10.1007/978-3-319-26555-1_2]

33. [33] Y.-C. Choi, H.-S. Ahn, "Nonlinear control of quadrotor for point tracking: Actual implementation and experimental tests", IEEE/ASME transactions on mechatronics, vol. 20, no. 3, pp. 1179-1192, 2014. [DOI:10.1109/TMECH.2014.2329945]

34. [34] C. Edwards, T. Lombaerts, and H. Smaili, "Fault tolerant flight control", Lecture notes in control and information sciences, vol. 399, pp. 1-560, 2010. [DOI:10.1007/978-3-642-11690-2]

35. [35] J. Na, M. N. Mahyuddin, G. Herrmann, X. Ren, P. Barber, "Robust adaptive finite time parameter estimation and control for robotic systems", International Journal of Robust and Nonlinear Control, vol. 25, no. 16, pp. 3045-3071, 2015. [DOI:10.1002/rnc.3247]

36. [36] H. Modares, F. L. Lewis, Z.-P. Jiang, "${H} _ {{infty}} $ Tracking control of completely unknown continuous-time systems via off-policy reinforcement learning", IEEE transactions on neural networks and learning systems, vol. 26, no. 10, pp. 2550-2562, 2015. [DOI:10.1109/TNNLS.2015.2441749]

37. [37] H. Modares, F. L. Lewis, "Optimal tracking control of nonlinear partially-unknown constrained-input systems using integral reinforcement learning", Automatica, vol. 50, no. 7, pp. 1780-1792, 2014. [DOI:10.1016/j.automatica.2014.05.011]

38. [38] X. Wang, Q. Wang, C. Sun, "Prescribed performance fault-tolerant control for uncertain nonlinear MIMO system using actor-critic learning structure", IEEE Transactions on Neural Networks and Learning Systems, 2021. [DOI:10.1109/TNNLS.2021.3057482]

Send email to the article author

Rights and permissions
	This Journal is an open access Journal Licensed under the Creative Commons Attribution-NonCommercial 4.0 International License. (CC BY NC 4.0)