Skip to:
  1. Main navigation
  2. Main content
  3. Footer
Working Paper

Applications of Markov Chain Approximation Methods to Optimal Control Problems in Economics

In this paper we explore some of the benefits of using the finite-state Markov chain approximation (MCA) method of Kushner and Dupuis (2001) to solve continuous-time optimal control problems. We first show that the implicit finite-difference scheme of Achdou et al. (2017) amounts to a limiting form of the MCA method for a certain choice of approximating chains and policy function iteration for the resulting system of equations. We then illustrate the benefits of departing from policy function iteration by showing that using variations of modified policy function iteration to solve income fluctuation problems in two and three dimensions can lead to an increase in the speed of convergence of more than an order of magnitude. We then show that the MCA method is also well-suited to solving portfolio problems with highly correlated state variables, a setting that commonly occurs within general equilibrium models with financial frictions and for which it is difficult to construct monotone (and hence convergent) finite-difference schemes.

Working Papers of the Federal Reserve Bank of Cleveland are preliminary materials circulated to stimulate discussion and critical comment on research in progress. They may not have been subject to the formal editorial review accorded official Federal Reserve Bank of Cleveland publications. The views expressed in this paper are those of the authors and do not represent the views of the Federal Reserve Bank of Cleveland or the Federal Reserve System.

Suggested Citation

Phelan, Tom, and Keyvan Eslami. 2021. “Applications of Markov Chain Approximation Methods to Optimal Control Problems in Economics.” Federal Reserve Bank of Cleveland, Working Paper No. 21-04.