Skip to main content

Applications of Markov Chain Approximation Methods to Optimal Control Problems in Economics


In this paper we explore some of the benefits of using the finite-state Markov chain approximation (MCA) method of Kushner and Dupuis (2001) to solve continuous-time optimal control problems. We first show that the implicit finite-difference scheme of Achdou et al. (2017) amounts to a limiting form of the MCA method for a certain choice of approximating chains and policy function iteration for the resulting system of equations. We then illustrate the benefits of departing from policy function iteration by showing that using variations of modified policy function iteration to solve income fluctuation problems in two and three dimensions can lead to an increase in the speed of convergence of more than an order of magnitude. We then show that the MCA method is also well-suited to solving portfolio problems with highly correlated state variables, a setting that commonly occurs within general equilibrium models with financial frictions and for which it is difficult to construct monotone (and hence convergent) finite-difference schemes.

JEL Codes: C63, E00, G11.
Keywords: Dynamic programming, financial frictions.


Suggested citation: Phelan, Thomas, and Keyvan Eslami. 2021. “Applications of Markov Chain Approximation Methods to Optimal Control Problems in Economics.” Federal Reserve Bank of Cleveland, Working Paper No. 21-04. https://doi.org/10.26509/frbc-wp-202104.

Upcoming EventsSEE ALL

  • 09.15.22

    FedUniversity: Internships at the Federal Reserve Bank of Cleveland

    Join us on September 15, noon to 1:00pm, to learn about the Cleveland Fed’s annual internship program.

  • 09.29.22

    Inflation: Drivers and Dynamics 2022

    Inflation: Drivers and Dynamics 2022, co-sponsored by the Center for Inflation Research and the European Central Bank will take place on September 29-30, 2022 in Cleveland, OH.

  • 10.06.22

    Conference on Real-Time Data Analysis, Methods, and Applications

    The Federal Reserve Bank of Cleveland is sponsoring the Annual Conference on Real-Time Data Analysis, Methods, and Applications in Macroeconomics and Finance to be held in Cleveland, Ohio, at the Cleveland Reserve Bank on Thursday and Friday, October 6–7, 2022.

  • 11.07.22

    Women in Economics Symposium

    Join the Federal Reserve Bank of Cleveland at the Women in Economics Symposium, which aims to encourage women to pursue a career in economics by discussing the successful career paths of diverse economics professionals and raising awareness of diversity and inclusion in the profession.

  • 11.17.22

    2022 Financial Stability Conference: Frontier Risks, a New Normal, and Policy Challenges

    The Federal Reserve Bank of Cleveland and the Office of Financial Research will host their 10th annual financial stability conference on November 17–18, 2022. This year’s conference is titled Financial Stability: Frontier Risks, a New Normal, and Policy Challenges.