Report title: Deep Reinforcement Learning via Noncrossing Quantile Regression

Reporter: Professor Feng Xingdong Shanghai University of Finance and Economics

Reporting time: 10:10-11:10, November 19, 2020

Report location: Tencent Conference ID: 241 771 974 Conference password: 123456

School contact: Zhu Fukang fzhu@jlu.edu.cn

Report summary:

     Distributional reinforcement learning (DRL) estimates the distribution over future returns instead of the mean to more efficiently capture the intrinsic uncertainty of MDPs. However, batch-based DRL algorithms cannot guarantee the non-decreasing property of learned quantile curves especially at the early training stage , leading to abnormal distribution estimates and reduced model interpretability. To address these issues, we introduce a general DRL framework by using non-crossing quantile regression to ensure the monotonicity constraint within each sampled batch, which can be incorporated with some well-known DRL algorithm . We demonstrate the validity of our method from both the theory and model implementation perspectives. Experiments on Atari 2600 Games show that some state-of-art DRL algorithms with the non-crossing modification can significantly outperform their baselines in terms of faster convergence speeds and better testing performance. In particular, our method can effectively recover the distribution information and thus dramatically increase the exploration efficiency when the reward space is extremely sparse.

Brief introduction of the speaker:

      Feng Xingdong, Dean of the School of Statistics and Management, Shanghai University of Finance and Economics, professor of statistics, and doctoral supervisor. His research fields include data dimensionality reduction, robust methods, quantile regression and its application in economic problems, big data statistical calculations, etc. He has published many papers in the top international statistical journals JASA, AoS, JRSSB, and Biometrika. Selected as an Elected member of the International Statistical Association in 2018, Vice President of the National Association of Young Statisticians in 2019, Professional Member of the 7th Committee of the National Statistical Textbook Editing Committee in 2019 (Data Science and Big Data Technology Application Group), 2020 Served as a member of the State Council Disciplinary Review Group (Statistics).