ARPO: Optimal Robust Policy Optimization Models from Intrinsic State-Adversarial Markov Decision Process

doi:10.1007/s40305-026-00682-8

Journal of the Operations Research Society of China ›› 2026, Vol. 14 ›› Issue (3): 833-859.doi: 10.1007/s40305-026-00682-8

Previous Articles Next Articles

ARPO: Optimal Robust Policy Optimization Models from Intrinsic State-Adversarial Markov Decision Process

Jia-Yu Lv, Hao-Ran Li, Cong-Ying Han, Tian-De Guo

School of Mathematical Sciences, University of Chinese Academy of Sciences, Beijing 100049, China

Received:2025-04-25 Revised:2026-03-01 Accepted:2026-03-05 Online:2026-09-30 Published:2026-10-15
Supported by:
This work is supported by the National Key R&D Program of China (NO. 2021YFA1000403), and the National Natural Science Foundation of China (Nos.12431012 and U23B2012).

Abstract

Abstract: Reinforcement learning (RL) faces significant challenges when deployed in real-world scenarios with potential adversarial perturbations. The Intrinsic State-Adversarial Markov Decision Process (ISA-MDP) framework considers state perturbation to enhance policy robustness while preserving its natural performance. Building upon ISA-MDP, this paper proposes the first policy optimization framework and develops two distinct algorithms for different scenarios: Deterministic Adversarially Robust Policy Gradient (DARPG) and stochastic adversarially robust policy optimization, which effectively solve our formulated optimization problem. Furthermore,we provide a convergence analysis for both algorithms under specified theoretical assumptions.

Key words: Reinforcement learning, Intrinsic State-Adversarial Markov decision process, Adversarial robustness, Policy optimization framework, Robust policy optimization

CLC Number:

90-11·68T20·68T07

Jia-Yu Lv, Hao-Ran Li, Cong-Ying Han, Tian-De Guo. ARPO: Optimal Robust Policy Optimization Models from Intrinsic State-Adversarial Markov Decision Process[J]. Journal of the Operations Research Society of China, 2026, 14(3): 833-859.

TrendMD

Add to citation manager EndNote|Reference Manager|ProCite|BibTeX|RefWorks

URL: https://www.jorsc.shu.edu.cn/EN/10.1007/s40305-026-00682-8

https://www.jorsc.shu.edu.cn/EN/Y2026/V14/I3/833

[1] Bellman, R.: A Markovian decision process. J. Math. Mech. 6, 679-684(1957)
[2] Dulac-Arnold, G., Mankowitz, D., Hester, T.: Challenges of real-world reinforcement learning (2019). arXiv:1904.12901
[3] Fawzi, A., Balog, M., Huang, A., Hubert, T., Romera-Paredes, B., Barekatain, M., Novikov, A., Ruiz, F.J.R., Schrittwieser, J., Swirszcz, G., et al.: Discovering faster matrix multiplication algorithms with reinforcement learning. Nature. 610(7930), 47-53(2022)
[4] Haarnoja, T., Zhou, A., Abbeel, P., et al.: Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In: International Conference on Machine Learning, pp. 1861-1870(2018)
[5] Huang, S. H., Papernot, N., Goodfellow, I. J., Duan, Y., Abbeel, P.: Adversarial attacks on neural network policies. In: 5th International Conference on Learning Representations Workshop Proceedings, pp. 24-26(2017)
[6] Jumper, J., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., Tunyasuvunakool, K., Bates,. R., Žídek, A., Potapenko, A., et al.: Highly accurate protein structure prediction with AlphaFold. Nature. 596(7873), 583-589(2021)
[7] Kakade, S.M.: A natural policy gradient. Adv. Neural Inf. Proc. Syst. 14(2001)
[8] Kakade, S., Langford, J.: Approximately optimal approximate reinforcement learning. In: Proceedings of the Nineteenth International Conference on Machine Learning, pp. 267-274(2002)
[9] Khodadadian, S., Jhunjhunwala, P. R., Varma, S. M., Maguluri, S.T.: On the linear convergence of natural policy gradient algorithm. In: IEEE Conference on Decision and Control, pp. 3794-3799(2021)
[10] Li, H., Zhang, Z., Luo, W., Han, C., Hu, Y., Guo, T., Liao, S.: Towards optimal adversarial robust Qlearning with Bellman infinity-error. In: Proceedings of the 41st International Conference on Machine Learning, Article No. 1180, pp. 1-49, Vienna, Austria (2024)
[11] Li, H., Zhang, Z., Luo, W., Han, C., Lv, J., Guo, T., Hu, Y.: Towards optimal adversarial robust reinforcement learning with infinity measurement error (2025). arXiv:2502.16734
[12] Lillicrap, T. P., Hunt, J. J., Pritzel, A., et al.: Continuous control with deep reinforcement learning (2015). arXiv:1509.02971
[13] Liu, J., Li, W., Wei, K.: Projected policy gradient converges in a finite number of iterations (2023). arXiv:2311.01104
[14] Liu, J., Li, W., Wei, K.: Elementary analysis of policy gradient methods (2024). arXiv:2404.03372
[15] Lowe, R., Wu, Y. I., Tamar, A., et al.: Multi-agent actor-critic for mixed cooperative-competitive environments. Adv. Neural Inf. Proc. Syst 30(2017)
[16] Miki, T., Lee, J., Hwangbo, J., Wellhausen, L., Koltun, V., Hutter, M.: Learning robust perceptive locomotion for quadrupedal robots in the wild. Sci. Robot. 7(62), eabk2822(2022)
[17] Moos, J., Hansel, K., Abdulsamad, H., Stark, S., Clever, D., Peters, J.: Robust reinforcement learning: A review of foundations and recent advances. Mach. Learn. Knowl. Extr. 4(1), 276-315(2022)
[18] Pinto, L., Davidson, J., Sukthankar, R., et al.: Robust adversarial reinforcement learning. In: International Conference on Machine Learning, pp. 2817-2826(2017)
[19] Schulman, J., Levine, S., Abbeel, P., et al.: Trust region policy optimization. In: International Conference on Machine Learning, pp. 1889-1897(2015)
[20] Schulman, J., Wolski, F., Dhariwal, P., et al.: Proximal policy optimization algorithms (2017). arXiv:1707.06347
[21] Silver, D., Huang, A., Maddison, C.J., Guez, A., Sifre, L., Van Den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al.: Mastering the game of Go with deep neural networks and tree search. Nature. 529(7587), 484-489(2016)
[22] Silver, D., Hubert, T., Schrittwieser, J., et al.: Mastering chess and shogi by self-play with a general reinforcement learning algorithm (2017). arXiv:1712.01815
[23] Sutton, R. S., McAllester, D., Singh, S., et al.: Policy gradient methods for reinforcement learning with function approximation. Adv. Neural Inf. Proc. Syst. 12(1999)
[24] Wiesemann, W., Kuhn, D., Rustem, B.: Robust Markov decision processes. Math. Oper. Res. 38(1), 153-183(2013)
[25] Williams, R.J.: Simple statistical gradient-following algorithms for connectionist reinforcement learning. Mach. Learn. 8, 229-256(1992)
[26] Zhang, H., Chen, H., Xiao, C., Li, B., Liu, M., Boning, D., Hsieh, C.-J.: Robust deep reinforcement learning against adversarial perturbations on state observations. Adv. Neural. Inf. Process. Syst. 33, 21024-21037(2020)
[27] Zhang, H., Chen, H., Boning, D., Hsieh, C.-J.: Robust reinforcement learning on state observations with learned optimal adversary. In: The 9th International Conference on Learning Representations (2021)

ARPO: Optimal Robust Policy Optimization Models from Intrinsic State-Adversarial Markov Decision Process

PDF

Knowledge

Abstract

Cite this article

share this article

Related Articles 1

Recommended Articles

Metrics

Comments