Risk-Sensitive Markov Decision Processes and Reinforcement Learning

Reinforcement learning (RL) has been receiving intensive research attention since the significant success of AlphaGo. Markov decision processes (MDPs) are used as the mathematical model of RL. However, most of the current RL and MDP studies focus on the optimization objective of cumulative discounted rewards. Risk-related objectives are also important for many practical systems, such as the risk management in finance. In this talk, we will introduce some theoretical results on the long-run variance optimization problems in the framework of MDPs, where variance is a widely used metric for measuring risk. Because of the quadratic form of variance function, the long-run variance cost function depends on the whole policy and is not Markovian. The long-run variance optimization problem is not a standard MDP model and the classical Bellman optimality equation does not hold. We study this problem from a new perspective called the sensitivity-based optimization, which help us derive some new advances: by defining a pseudo-variance quantity, we derive a variance difference formula which has an elegant form to quantify the difference of long-run variances under any two policies. Based on the variance difference formula, we obtain a necessary and sufficient condition for the local optimum in the mixed policy space, which is only a necessary condition for the global optimum. We further derive the so-called Bellman local optimality equation and the optimality of deterministic policies is also proved. We further develop a policy iteration type algorithm to minimize the long-run variance and its local convergence is also proved. Finally, we extend the theoretical results to RL algorithmic studies and apply them to several practical problems, including the power fluctuation reduction of wind and battery storage system and the portfolio management problem in financial engineering. The latest research discovers that our work is able to extend to MDPs with CVaR (Conditional Value-at-Risk) objectives.