<p>For continuous-time stochastic dynamic programming, the small, nontechnical <a href="http://rads.stackoverflow.com/amzn/click/0415269377"><em>Art of Smooth Pasting</em></a> by Dixit is a wonderful option. It does a very effective job of conveying the basic intuition.</p>

<p>Stokey's more recent <a href="http://rads.stackoverflow.com/amzn/click/0691135053"><em>The Economics of Inaction</em></a> is also decent, but for a practical-minded person it probably underperforms Dixit - its much greater length and somewhat heavier notation do not yield commensurate rewards.</p>

<hr>

<p>If the underlying stochastic processes are not Itō diffusions, then I'm not sure what the best reference is. The most common such case I've seen (and that I use myself) is the case of discretely many exogenous states, where if we are currently in state $s$ there is some constant hazard rate $\lambda_{s,s'}$ of a switch to state $s'$. Fortunately, this is a pretty simple case in practice: one can just modify the HJB equation to account for the flow probability of switching from $V(\cdot,s)$ to $V(\cdot,s')$. (You can see this, for instance, in equations (1)-(5) in this <a href="http://economics.mit.edu/files/7720">Acemoglu and Akcigit paper</a>. Conceptually it's no different from setting up the HJB equation when we have an Itō diffusion as the driving process, except that it's simpler because we just get a system of linear equations and we don't need to think about Itō's lemma, etc.)</p>

<p>Of course, maybe there is a good textbook reference for this too - but unlike the potentially much more complicated cases involving stochastic calculus, this is straightforward enough that a text has never seemed necessary to me.</p>

CC BY-SA 3.0
2015-01-07T05:27:06.700Z
*The Economics of Inaction* looks great. I looked a little at it. Given your description of *The Art of Smooth Pasting*, I'd really like to check it out, but it seems a little hard to get a hold of. Thanks for the recommendations!
jmbejara
CC BY-SA 3.0
2015-01-06T00:57:42.390Z
2015-01-06T01:51:21.773Z
2015-01-06T01:51:21.773Z
nominally rigid
<p>Dynamic Programming &amp; Optimal Control by Bertsekas</p>

<p>Introduction to Modern Economic Growth by Acemoglu</p>

<p>The Acemoglu book, even though it specializes in growth theory, does a very good job presenting continuous time dynamic programming.</p>

CC BY-SA 3.0
2015-01-06T00:22:41.073Z
2015-01-06T00:22:41.073Z
Bryce
<p><a href="http://rads.stackoverflow.com/amzn/click/0387260455" rel="nofollow">Controlled Markov Processes and Viscosity Solutions</a> by Fleming and Soner includes a number of applications to Finance and Differential Games.</p>

CC BY-SA 3.0
2015-02-21T00:30:10.067Z
2015-02-21T00:30:10.067Z
Jefferson Huang
<p>I think Kamien and Schwartz's <a href="http://rads.stackoverflow.com/amzn/click/048648856X" rel="noreferrer"><em>Dynamic Optimization: The Calculus of Variations and Optimal Control in Economics and Management</em></a> is pretty well known.</p>

CC BY-SA 3.0
2015-09-24T09:09:18.570Z
It is a very good but note that the book is not really an introduction book for dynamic optimization.
optimal control
CC BY-SA 3.0
2015-01-06T01:37:33.610Z
2015-01-06T01:37:33.610Z
jmbejara
<p>A really nice methodology for approximating the HJB is the upwind scheme, which I learnt quite quickly using <a href="http://www.princeton.edu/%7Emoll/HACTproject.htm" rel="nofollow">Ben Moll et al's notes and codes</a></p>

<p>The examples are continuous time versions of familiar heterogenous agents economies models such as Hugget and Aiyagari.</p>

CC BY-SA 3.0
2015-02-21T11:59:09.217Z
2015-02-21T11:59:09.217Z
FooBar
<p><a href="http://www.waelde.com/pdf/AIO.pdf" rel="nofollow"> Applied Intertemporal Optimization</a> by Klaus Wälde is a very very nice book, even for those who are not really familiar with mathematics.</p>

<p>The book treats deterministic and stochastic models, both in discrete and continuous time.</p>

<p>I would really say for this book "Dynamic Optimization for dummies". I was not familiar at all with dynamic optimization but this book let me to get through.</p>

CC BY-SA 4.0
2019-03-29T16:57:46.770Z
This book looks fantastic. I love the organization. The tables at the beginning really illustrate how thorough it is. The book covers most cases of interest and provides many examples. (Apologies for the 4 year delay in responding... :) )
jmbejara
CC BY-SA 4.0
2019-03-29T17:16:59.323Z
Happy that it helps! I learned so much from this book :)
optimal control
CC BY-SA 3.0
2015-09-24T09:13:00.340Z
2015-09-24T09:13:00.340Z
optimal control
<p>Does anyone know of good references to learn continuous-time dynamic programming? The references don't have to be books. They could be links to online resources as well. Links to clear, concise discussions of even just the basics would be helpful.</p>

CC BY-SA 3.0
2015-01-05T23:08:35.793Z
deterministic, stochastic, or both? depends quite a bit on which.
nominally rigid
CC BY-SA 3.0
2015-01-05T23:09:46.180Z
Sorry. Both would be helpful, but I actually do have stochastic in mind. Thanks!
jmbejara
CC BY-SA 3.0
2015-01-09T18:41:36.783Z
Just thought I'd mention that if we are going to do these types of questions, we should keep with the [decided convention](http://meta.economics.stackexchange.com/questions/169/consensus-on-list-questions/171#171) of providing 1 recommendation per answer.
cc7768
<optimization><dynamic-programming><continuous-time>
References to learn continuous-time dynamic programming
jmbejara
# References to learn continuous-time dynamic programming

(Asked by: username_0 on 2015-01-05)

Does anyone know of good references to learn continuous-time dynamic programming? The references don't have to be books. They could be links to online resources as well. Links to clear, concise discussions of even just the basics would be helpful.

**Tags:** optimization, dynamic-programming, continuous-time
> **Comments:**
> **username_1**: deterministic, stochastic, or both? depends quite a bit on which.  
> **username_0**: Sorry. Both would be helpful, but I actually do have stochastic in mind. Thanks!  
> **username_2**: Just thought I'd mention that if we are going to do these types of questions, we should keep with the [decided convention](http://meta.economics.stackexchange.com/questions/169/consensus-on-list-questions/171#171) of providing 1 recommendation per answer.  

---

## 6 answers

### Answer by username_1 on 2015-01-06. Score: 9

For continuous-time stochastic dynamic programming, the small, nontechnical [*Art of Smooth Pasting*](http://rads.stackoverflow.com/amzn/click/0415269377) by Dixit is a wonderful option. It does a very effective job of conveying the basic intuition.

Stokey's more recent [*The Economics of Inaction*](http://rads.stackoverflow.com/amzn/click/0691135053) is also decent, but for a practical-minded person it probably underperforms Dixit - its much greater length and somewhat heavier notation do not yield commensurate rewards.

---

If the underlying stochastic processes are not Itō diffusions, then I'm not sure what the best reference is. The most common such case I've seen (and that I use myself) is the case of discretely many exogenous states, where if we are currently in state $s$ there is some constant hazard rate $\lambda\_{s,s'}$ of a switch to state $s'$. Fortunately, this is a pretty simple case in practice: one can just modify the HJB equation to account for the flow probability of switching from $V(\cdot,s)$ to $V(\cdot,s')$. (You can see this, for instance, in equations (1)-(5) in this [Acemoglu and Akcigit paper](http://economics.mit.edu/files/7720). Conceptually it's no different from setting up the HJB equation when we have an Itō diffusion as the driving process, except that it's simpler because we just get a system of linear equations and we don't need to think about Itō's lemma, etc.)

Of course, maybe there is a good textbook reference for this too - but unlike the potentially much more complicated cases involving stochastic calculus, this is straightforward enough that a text has never seemed necessary to me.

> **Comments:**
> **username_0**: *The Economics of Inaction* looks great. I looked a little at it. Given your description of *The Art of Smooth Pasting*, I'd really like to check it out, but it seems a little hard to get a hold of. Thanks for the recommendations!  

---

### Answer by username_3 on 2015-01-06. Score: 6

Dynamic Programming & Optimal Control by Bertsekas

Introduction to Modern Economic Growth by Acemoglu

The Acemoglu book, even though it specializes in growth theory, does a very good job presenting continuous time dynamic programming.


---

### Answer by username_0 on 2015-01-06. Score: 5

I think Kamien and Schwartz's [*Dynamic Optimization: The Calculus of Variations and Optimal Control in Economics and Management*](http://rads.stackoverflow.com/amzn/click/048648856X) is pretty well known.

> **Comments:**
> **username_5**: It is a very good but note that the book is not really an introduction book for dynamic optimization.  

---

### Answer by username_4 on 2015-02-21. Score: 4

[Controlled Markov Processes and Viscosity Solutions](http://rads.stackoverflow.com/amzn/click/0387260455) by Fleming and Soner includes a number of applications to Finance and Differential Games.


---

### Answer by username_6 on 2015-02-21. Score: 3

A really nice methodology for approximating the HJB is the upwind scheme, which I learnt quite quickly using [Ben Moll et al's notes and codes](http://www.princeton.edu/%7Emoll/HACTproject.htm)

The examples are continuous time versions of familiar heterogenous agents economies models such as Hugget and Aiyagari.


---

### Answer by username_5 on 2015-09-24. Score: 2

[Applied Intertemporal Optimization](http://www.waelde.com/pdf/AIO.pdf) by Klaus Wälde is a very very nice book, even for those who are not really familiar with mathematics.

The book treats deterministic and stochastic models, both in discrete and continuous time.

I would really say for this book "Dynamic Optimization for dummies". I was not familiar at all with dynamic optimization but this book let me to get through.

> **Comments:**
> **username_0**: This book looks fantastic. I love the organization. The tables at the beginning really illustrate how thorough it is. The book covers most cases of interest and provides many examples. (Apologies for the 4 year delay in responding... :) )  
> **username_5**: Happy that it helps! I learned so much from this book :)  

---

