Hostname: page-component-669899f699-ggqkh Total loading time: 0 Render date: 2025-04-30T03:33:15.876Z Has data issue: false hasContentIssue false
  • English
  • Français

Curriculum reinforcement learning-based drifting along a general path for autonomous vehicles

Published online by Cambridge University Press:  02 December 2024

Kai Yu Open the ORCID record for Kai Yu [Opens in a new window] ,
Mengyin Fu ,
Xiaohui Tian Open the ORCID record for Xiaohui Tian [Opens in a new window] ,
Shuaicong Yang  and
Yi Yang
Kai Yu
Affiliation:
School of Automation, Beijing Institute of Technology, Beijing, China
Mengyin Fu
Affiliation:
School of Automation, Beijing Institute of Technology, Beijing, China School of Automation, Nanjing University of Science and Technology, Nanjing, China
Xiaohui Tian
Affiliation:
School of Automation, Beijing Institute of Technology, Beijing, China
Shuaicong Yang
Affiliation:
School of Automation, Beijing Institute of Technology, Beijing, China
Yi Yang*
Affiliation:
School of Automation, Beijing Institute of Technology, Beijing, China
*
Corresponding author: Yi Yang; Email: [email protected]
Get access Rights & Permissions [Opens in a new window]

Abstract

Expert drivers possess the ability to execute high sideslip angle maneuvers, commonly known as drifting, during racing to navigate sharp corners and execute rapid turns. However, existing model-based controllers encounter challenges in handling the highly nonlinear dynamics associated with drifting along general paths. While reinforcement learning-based methods alleviate the reliance on explicit vehicle models, training a policy directly for autonomous drifting remains difficult due to multiple objectives. In this paper, we propose a control framework for autonomous drifting in the general case, based on curriculum reinforcement learning. The framework empowers the vehicle to follow paths with varying curvature at high speeds, while executing drifting maneuvers during sharp corners. Specifically, we consider the vehicle’s dynamics to decompose the overall task and employ curriculum learning to break down the training process into three stages of increasing complexity. Additionally, to enhance the generalization ability of the learned policies, we introduce randomization into sensor observation noise, actuator action noise, and physical parameters. The proposed framework is validated using the CARLA simulator, encompassing various vehicle types and parameters. Experimental results demonstrate the effectiveness and efficiency of our framework in achieving autonomous drifting along general paths. The code is available at https://github.com/BIT-KaiYu/drifting.

Keywords

autonomous vehiclesdrifting controlreinforcement learningcurriculum learningrandomization
Type
Research Article
Information
Robotica , Volume 42 , Issue 10 , October 2024 , pp. 3263 - 3280
Copyright
© The Author(s), 2024. Published by Cambridge University Press

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Article purchase

Temporarily unavailable

References

Tian, X., Yang, S., Yang, Y., Song, W. and Fu, M., “A multi-layer drifting controller for all-wheel drive vehicles beyond driving limits,” IEEE/ASME Trans. Mechatron. 29(2), 1–11 (2023). doi: 10.1109/TMECH.2023.3298660 Google Scholar
Goh, J. Y., Thompson, M., Dallas, J. and Balachandran, A., “Beyond the stable handling limits: Nonlinear model predictive control for highly transient autonomous drifting,” Veh. Syst. Dyn. 62(10), 2590–2613 (2024).CrossRefGoogle Scholar
Dong, H., Yu, H. and Xi, J., “Phase portrait analysis and drifting control of unmanned tracked vehicles,” IEEE Trans. Intell. Veh. 10(1), 1–16 (2024). doi: 10.1109/TIV.2024.3356608 Google Scholar
Inagaki, S., Kushiro, I. and Yamamoto, M., “Analysis on vehicle stability in critical cornering using phase-plane method,” JSAE Rev. 2(16), 216–216 (1995).Google Scholar
Ono, E., Hosoe, S., Tuan, H. D. and Doi, S., “Bifurcation in vehicle dynamics and robust front wheel steering control,” IEEE Trans. Control Syst. Technol. 6(3), 412–420 (1998).CrossRefGoogle Scholar
Voser, C., Hindiyeh, R. Y. and Gerdes, J. C., “Analysis and control of high sideslip manoeuvres,” Veh. Syst. Dyn. 48(S1), 317–336 (2010).CrossRefGoogle Scholar
Velenis, E., Katzourakis, D., Frazzoli, E., Tsiotras, P. and Happee, R., “Steady-state drifting stabilization of rwd vehicles,” Control. Eng. Pract. 19(11), 1363–1376 (2011).CrossRefGoogle Scholar
Goh, J. Y., Goel, T. and Gerdes, J. C., “Toward automated vehicle control beyond the stability limits: drifting along a general path,” J. Dyn. Sys., Meas., Control. 142(2), 1–13 (2020).CrossRefGoogle Scholar
Zhang, F., Gonzales, J., Li, S. E., Borrelli, F. and Li, K., “Drift control for cornering maneuver of autonomous vehicles,” Mechatronics. 54(1), 167–174 (2018)CrossRefGoogle Scholar
Chen, G., Zhao, X., Gao, Z. and Hua, M., “Dynamic drifting control for general path tracking of autonomous vehicles,” IEEE Trans. Intell. Veh. 8(3), 2527–2537 (2023). doi: 10.1109/TIV.2023.3235007 CrossRefGoogle Scholar
Sutton, R. S. and Barto, A. G.. Reinforcement Learning: An Introduction (MIT Press, USA, 2018).Google Scholar
Rose, L., Bazzocchi, M. C. and Nejat, G., “A model-free deep reinforcement learning approach for control of exoskeleton gait patterns,” Robotica. 40(7), 2189–2214 (2022).CrossRefGoogle Scholar
Zhang, D., Ju, R. and Cao, Z., “Reinforcement learning-based motion control for snake robots in complex environments,” Robotica. 42(4), 947–961 (2024).CrossRefGoogle Scholar
Kaufmann, E., Bauersfeld, L., Loquercio, A., Müller, M., Koltun, V. and Scaramuzza, D., “Champion-level drone racing using deep reinforcement learning,” Nature. 620(7976), 982–987 (2023).CrossRefGoogle ScholarPubMed
Domberg, F., Wembers, C. C., Patel, H. and Schildbach, G., “Deep drifting: Autonomous drifting of arbitrary trajectories using deep reinforcement learning,” 2022 IEEE International Conference on Robotics and Automation (ICRA), Philadelphia, USA (IEEE, 2022) pp. 7753–7759 CrossRefGoogle Scholar
Cai, P., Mei, X., Tai, L., Sun, Y. and Liu, M., “High-speed autonomous drifting with deep reinforcement learning,” IEEE Robot. Autom. Lett. 5(2), 1247–1254 (2020).CrossRefGoogle Scholar
Narvekar, S., Peng, B., Leonetti, M., Sinapov, J., Taylor, M. E. and Stone, P., “Curriculum learning for reinforcement learning domains: a framework and survey,” J. Mach. Learn. Res. 21(1), 7382–7431 (2020).Google Scholar
Edelmann, J. and Plöchl, M., “Handling characteristics and stability of the steady-state powerslide motion of an automobile,” Regul. Chaotic Dyn. 14(6), 682–692 (2009).CrossRefGoogle Scholar
Hindiyeh, R. Y. and Gerdes, J. C., “Equilibrium analysis of drifting vehicles for control design,” Dynamic Systems and Control Conference, Philadelphia, USA (2009) pp. 181–188.Google Scholar
Baars, M., Hellendoorn, H. and Alirezaei, M., “Control of a scaled vehicle in and beyond stable limit handling,” IEEE Trans. Veh. Technol. 70(7), 6427–6437 (2021).CrossRefGoogle Scholar
Khan, M., Youn, E., Youn, I. and Wu, L., “Steady state drifting controller for vehicles travelling in reverse direction,” 2018 15th International Bhurban Conference on Applied Sciences and Technology (IBCAST), Islamabad, Pakistan (IEEE, 2018) pp. 263–268.CrossRefGoogle Scholar
Peterson, M. T., Goel, T. and Gerdes, J. C., “Exploiting linear structure for precision control of highly nonlinear vehicle dynamics,” IEEE Trans. Intell. Veh. 8(2), 1852–1862 (2022).CrossRefGoogle Scholar
Bellegarda, G. and Nguyen, Q., “Dynamic vehicle drifting with nonlinear mpc and a fused kinematic-dynamic bicycle model,” IEEE Contr. Syst. Lett. 6(1), 1958–1963 (2021)CrossRefGoogle Scholar
Hu, C., Zhou, X., Duo, R., Xiong, H., Qi, Y., Zhang, Z. and Xie, L., “Combined fast control of drifting state and trajectory tracking for autonomous vehicles based on mpc controller,” 2022 IEEE International Conference on Robotics and Automation (ICRA), Philadelphia, USA (IEEE, 2022) pp. 1373–1379.CrossRefGoogle Scholar
Weber, T. P. and Gerdes, J. C., “Modeling and control for dynamic drifting trajectories,” IEEE Trans. Intell Veh. 9(2), 1–11 (2023). doi: 10.1109/TIV.2023.3340918 Google Scholar
Acosta, M. and Kanarachos, S., “Teaching a vehicle to autonomously drift: a data-based approach using neural networks,” Knowl. Based. Syst. 153(1), 12–28 (2018)CrossRefGoogle Scholar
Zhou, X., Hu, C., Duo, R., Xiong, H., Qi, Y., Zhang, Z., Su, H. and Xie, L., “Learning-based mpc controller for drift control of autonomous vehicles,” 2022 IEEE 25th International Conference on Intelligent Transportation Systems (ITSC), Macau, China(IEEE, 2022) pp. 322–328.CrossRefGoogle Scholar
Cutler, M. and How, J. P., “Autonomous drifting using simulation-aided reinforcement learning,” 2016 IEEE International Conference on Robotics and Automation (ICRA), Stockholm, Sweden (IEEE, 2016) pp. 5442–5448.CrossRefGoogle Scholar
Hindiyeh, R. Y.. Dynamics and Control of Drifting in Automobiles (Stanford University, USA, 2013)Google Scholar
Betz, J., Zheng, H., Liniger, A., Rosolia, U., Karle, P., Behl, M., Krovi, V. and Mangharam, R., “Autonomous vehicles on the edge: A survey on autonomous vehicle racing,” IEEE O J Intell. Transp. Syst. 3(1), 458–488 (2022)CrossRefGoogle Scholar
Schulman, J., Wolski, F., Dhariwal, P., Radford, A. and Klimov, O., “Proximal policy optimization algorithms,” arXiv: 1707.06347, 1-12 (2017)Google Scholar
Schulman, J., Moritz, P., Levine, S., Jordan, M. and Abbeel, P., “High dimensional continuous control using generalized advantage estimation,” arXiv: 1506.02438, 1-14 (2015)Google Scholar
Dosovitskiy, A., Ros, G., Codevilla, F., Lopez, A. and Koltun, V., “Carla: An open urban driving simulator,” Conference on Robot Learning, California, USA (PMLR, 2017) pp. 1–16.Google Scholar
Chen, L., He, Y., Wang, Q., Pan, W. and Ming, Z., “Joint optimization of sensing, decision-making and motion-controlling for autonomous vehicles: A deep reinforcement learning approach,” IEEE Trans. Veh. Technol. 71(5), 4642–4654 (2022).CrossRefGoogle Scholar
Sun, W., Wang, X. and Zhang, C., “A model-free control strategy for vehicle lateral stability with adaptive dynamic programming,” IEEE Trans. Ind. Electron. 67(12), 10693–10701 (2019).CrossRefGoogle Scholar
Bae, I., Kim, J. H. and Kim, S., “Steering rate controller based on curvature of trajectory for autonomous driving vehicles,” 2013 IEEE Intelligent Vehicles Symposium (IV), Gold Coast City, Australia (IEEE, 2013) 1381–1386.CrossRefGoogle Scholar
Bengio, Y., Louradour, J., Collobert, R. and Weston, J., “Curriculum learning,” Proceedings of the 26th Annual International Conference on Machine Learning, Montreal, Canada (2009) pp. 41–48.Google Scholar
Rudin, N., Hoeller, D., Reist, P. and Hutter, M., “Learning to walk in minutes using massively parallel deep reinforcement learning,” Conference on Robot Learning, Auckland, New Zealand (PMLR, 2022). 91–100.Google Scholar
Wang, H.-C., Huang, S.-C., Huang, P.-J., Wang, K.-L., Teng, Y.-C., Ko, Y.-T., Jeon, D. and Wu, I.-C., “Curriculum reinforcement learning from avoiding collisions to navigating among movable obstacles in diverse environments,” IEEE Robot. Autom. Lett. 8(5), 2740–2747 (2023).CrossRefGoogle Scholar
Hoeller, D., Rudin, N., Sako, D. and Hutter, M., “Anymal parkour: Learning agile navigation for quadrupedal robots,” Sci. Robot. 9(88), eadi7566 (2024).CrossRefGoogle ScholarPubMed
Wang, X., Chen, Y. and Zhu, W., “A survey on curriculum learning,” IEEE Trans. Pattern Anal. Mach. Intell. 44(9), 4555–4576 (2022).Google ScholarPubMed
Tobin, J., Fong, R., Ray, A., Schneider, J., Zaremba, W. and Abbeel, P., “Domain randomization for transferring deep neural networks from simulation to the real world,” 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Vancouver, Canada (IEEE, 2017) pp. 23–30.CrossRefGoogle Scholar
Peng, X. B., Andrychowicz, M., Zaremba, W. and Abbeel, P., “Sim-to-real transfer of robotic control with dynamics randomization,” 2018 IEEE International Conference on Robotics and Automation (ICRA), Brisbane, Australia (IEEE, 2018) pp. 3803–3810.CrossRefGoogle Scholar
Chen, Y., Zeng, C., Wang, Z., Lu, P. and Yang, C., “Zero-shot sim-to-real transfer of reinforcement learning framework for robotics manipulation with demonstration and force feedback,” Robotica. 41(3), 1015–1024 (2023).CrossRefGoogle Scholar
Xiao, R., Yang, C., Jiang, Y. and Zhang, H., “One-shot sim-to-real transfer policy for robotic assembly via reinforcement learning with visual demonstration,” Robotica. 42(4), 1074–1093 (2024).CrossRefGoogle Scholar
Hindiyeh, R. Y. and Gerdes, J. C., “A controller framework for autonomous drifting: Design, stability, and experimental validation,” J. Dyn. Sys., Meas., Control. 136(5), 051015 (2014).CrossRefGoogle Scholar