Liu, Z., Bai, Q., Blanchet, J., Dong, P., Xu, W., Zhou, Z. & Zhou, Z.. (2022). Distributionally Robust $Q$-Learning. Proceedings of the 39th International Conference on Machine Learning, in Proceedings of Machine Learning Research 162:13623-13643 Available from https://proceedings.mlr.press/v162/liu22a.html.
Abstract
Reinforcement learning (RL) has demonstrated remarkable achievements in simulated environments. However, carrying this success to real environments requires the important attribute of robustness, which the existing RL algorithms often lack as they assume that the future deployment environment is the same as the training environment (ie simulator) in which the policy is learned. This assumption often does not hold due to the discrepancy between the simulator and the real environment and, as a result, and hence renders the learned policy fragile when deployed. In this paper, we propose a novel distributionally robust -learning algorithm that learns the best policy in the worst distributional perturbation of the environment. Our algorithm first transforms the infinite-dimensional learning problem (since the environment MDP perturbation lies in an infinite-dimensional space) into a finite-dimensional dual problem and subsequently uses a multi-level Monte-Carlo scheme to approximate the dual value using samples from the simulator. Despite the complexity, we show that the resulting distributionally robust -learning algorithm asymptotically converges to optimal worst-case policy, thus making it robust to future environment changes. Simulation results further demonstrate its strong empirical robustness.
Authors
Zijian Liu, Qinxun Bai, Jose Blanchet, Perry Dong, Wei Xu, Zhengqing Zhou, Zhengyuan Zhou
Publication date
2022/6/28
Conference
International Conference on Machine Learning
Pages
13623-13643
Publisher
PMLR
