<p>What you want to maximize is the discounted stream of utility over time. To that end, you can set control variables over time (subject to some constraints, like technology, budget...). We are ultimately interested in the optimal setting of these variables (called the optimal policy).</p>

<p>Bellman's main insight is the principle of optimality (reminiscent of backward induction/subgame perfectness in game theory): suppose you choose some action today as part of an optimal policy, then the remaining action sequence must be part of an optimal policy from tomorrow on.</p>

<p>Before turning to the policy, we solve for the value function (this part is called the prediction problem). For a discrete action space, you could actually draw a complete decision tree and solve via backward induction. The more feasible, tractable, and elegant solution is to write the problem recursively with respect to the continuation value of the problem. Solving the Bellman Equation will give you that value function <span class="math-container">$V(\cdot)$</span>, which is the <em>optimal</em> discounted value of the objective function, given the state you are in and an optimal policy from that state onwards. Note that the solution is thus a function, not a value.</p>

<p>From there, you can back out the sequence of actions (policy) that produces the optimal outcome. This part is called the control problem. Having solved the prediction problem, this is easy: the value function takes today's state and action as input and gives you the continuation value. So for each state, compute the continuation value for each possible action and pick the action that has the highest continuation value.</p>

CC BY-SA 4.0
2019-06-14T00:18:32.460Z
Hi Frederic, can you elaborate more by editing your answer on when you say "you can back out the sequence of actions that produces the optimal outcome". Thanks,
Frank Swanton
CC BY-SA 4.0
2019-06-14T11:17:32.663Z
Hi Frank, you are right, my first answer was a bit terse, I have expanded it a bit for clarity. I hope this helps.
Frederic Schneider
CC BY-SA 4.0
2019-06-18T08:08:26.833Z
Hi Fred, thanks for the elaboration. It helped a lot. I am reading few notes on this as it has been quite a long time the last time I dealt with the dynamic programming. When we use the term, the current value function $v(k)$ and the continuation value function, $v(k')$, what is the difference? Is it merely that we are putting the future capital stock as argument? Also, why is it the case that the Bellman equation has to hold for all feasible values of $k$? Why is this emphasized?
Frank Swanton
CC BY-SA 4.0
2019-06-13T14:21:24.707Z
2019-06-14T11:16:54.827Z
2019-06-14T11:16:54.827Z
Frederic Schneider
<p>Not sure what problem you are solving.
But, the next state can be stochastic and the reward can be stochastic.</p>

<p>For example, In Inventory problem, your action is the number of items you purchase now. But next state can be calculated by <strong>current inventory(current state) + new purchase(action) - demand(unknown factor)</strong> therefore it will be stochastic. </p>

<p>In stochastic problem usually, you have Expectation in your objective function.</p>

CC BY-SA 4.0
2019-06-14T00:17:12.707Z
I am reviewing the deterministic optimal growth model, and this is not exactly the response I was looking for because stochastic growth model comes later. Thanks.
Frank Swanton
CC BY-SA 4.0
2019-06-13T16:53:18.267Z
2019-06-13T16:53:18.267Z
S. Phil Kim
<p>The original problem was probably of the form <span class="math-container">$$\max_{\{W_t\}_{t=1}^\infty}\sum_{t=0}^\infty \beta^t u(W_t-W_{t+1}),$$</span>
<span class="math-container">$$\mbox{s.t. } W_{t+1}\in[0,W_t] \ \forall \ t,  ~~ W_0 \mbox{ given}$$</span>
When formulated that way there is no need to solve for any function <span class="math-container">$V$</span>, but the infinite elements of the sequence <span class="math-container">${\{W_t\}}_{t=1}^\infty$</span> are all choice variables. Bellman's main insight is that (under some conditions) the problem can be simplified. Instead of solving for an infinite number of variables, you could solve for a single "policy function" that tells you what to do each period as a function of the current state variables, and a "value function" that summarizes what is the value (in terms of today) of inducing each possible state in the future. Even if there is no uncertainty, it is clear that this value function is an endogenous object, since it must somehow capture the future stream of utility as a single function of the future state. In your case, that function is  <span class="math-container">$V(W')$</span>. This is why the Bellman equation: 
<span class="math-container">$$V(W)=\max_{W'\in[0,W]} u(W-W')+\beta V(W')$$</span>
has two properties: </p>

<p>1) it lacks the subscripts on the <span class="math-container">$W$</span>'s, this reflects that you will solve for <span class="math-container">$W'$</span> as a function of <span class="math-container">$W$</span>, i.e. the policy function. </p>

<p>2) has a new object, <span class="math-container">$V$</span>, that was not in the original problem and it is a function of <span class="math-container">$W$</span> (or <span class="math-container">$W'$</span>) and just by eyeballing the two problems you can see that for them to be equivalent <span class="math-container">$V$</span> must be the discounted sum of future payoffs assuming future actions are also optimal. So it is not a choice variable, but definitely, an endogenous object that you need to find when solving the problem.</p>

<p>You can see that the problem of optimally choosing infinitely many objects is transformed into one of choosing a finite number (2) of functions, which are infinite-dimensional objects. Brilliant!</p>

CC BY-SA 4.0
2019-06-18T08:13:59.870Z
Hi Regio, thanks for the clear explanation. I am getting a firmer grasp on the DP. One additional question is that why does the Bellman equation usually emphasize that the equation must hold for all (feasible) values of $W$? Also, I see that $V(W)$ is the current value function and $V(W')$ is the continuation value function, but how do you mean when you say it is "an endogenous object that you need to... the problem?" Just because it appears on both sides of the Bellman equation? I think what I am confused is that under the max operator $W'$ is choice variable but we say $V(\cdot)$ is the sol
Frank Swanton
CC BY-SA 4.0
2019-06-18T17:03:14.393Z
Not sure I understand your question. But $V(W)$ is the value of currently having a stock of $W$ (that is why $\beta V(W')$ is the discounted value of having a stock of $W'$ next period). and for you to be able to know how much should you consume today, $u(W-W')$ it is necessary to know what payoff each of these consumptions will imply for tomorrow. Note that different consumptions induce different $W'$. That's why $V(\cdot)$ must be valid for all values of $W$.
Regio
CC BY-SA 4.0
2019-06-18T17:07:05.343Z
Maybe it will be easier for you to see the max operator as some function $F$. then the  Bellman equation is of the form $V(W)=F(\cdot, V(W'))$, where $V$ is the unique function that satisfies this equality. (the uniqueness requires some conditions, but in most economics problems you will have uniqueness. Because $F$ is a max operator it can be challenging to find $V$, but there are some standard techniques that can be helpful.
Regio
CC BY-SA 4.0
2019-06-20T02:09:59.603Z
Thanks for the comments Regio. It helps a lot. Do you have any reference material where I can see an example of a problem that has a full-fledged version of solving a DP program including the lagrangean, envelope conditions, and etc? I want to work out a full version if you have a credible reference? thanks
Frank Swanton
CC BY-SA 4.0
2019-06-20T20:47:12.807Z
A classic reference is Thomas Sargent's [book](https://www.amazon.com/Dynamic-Macroeconomic-Theory-Thomas-Sargent/dp/0674218779/ref=asc_df_0674218779/?tag=hyprod-20&linkCode=df0&hvadid=312111907622&hvpos=1o3&hvnetw=g&hvrand=6249537410562009453&hvpone=&hvptwo=&hvqmt=&hvdev=c&hvdvcmdl=&hvlocint=&hvlocphy=9018948&hvtargid=pla-448229355571&psc=1). However if you do a quick search with  "Dynamic programming macroeconomics" you can find many lecture notes of introductory macro courses at the graduate level that will likely be helpful.
Regio
CC BY-SA 4.0
2019-06-14T17:03:34.497Z
2019-06-14T17:03:34.497Z
Regio
<p><span class="math-container">\begin{align}
 V(W)=\max\limits_{W'\in[0,W]}\qquad&amp; u(W-W')+\beta V(W')\qquad\forall W
\end{align}</span></p>

<p><span class="math-container">$\textbf{My Question}$</span>: Why is the unknown in the Bellman equation <span class="math-container">$V(W)$</span> itself? Isn't the choice variable tomorrow's state, <span class="math-container">$W'$</span>?</p>

<macroeconomics><dynamic-programming><bellman-equations>
What is unknown in Bellman Equation?
Frank Swanton
# What is unknown in Bellman Equation?

(Asked by: username_0 on 2019-06-13)

\begin{align}
V(W)=\max\limits\_{W'\in[0,W]}\qquad& u(W-W')+\beta V(W')\qquad\forall W
\end{align}

$\textbf{My Question}$: Why is the unknown in the Bellman equation $V(W)$ itself? Isn't the choice variable tomorrow's state, $W'$?

**Tags:** macroeconomics, dynamic-programming, bellman-equations

---

## 3 answers

### Answer by username_3 on 2019-06-14. Score: 4 | Accepted answer

The original problem was probably of the form $$\max\_{\{W\_t\}\_{t=1}^\infty}\sum\_{t=0}^\infty \beta^t u(W\_t-W\_{t+1}),$$
$$\mbox{s.t. } W\_{t+1}\in[0,W\_t] \ \forall \ t, ~~ W\_0 \mbox{ given}$$
When formulated that way there is no need to solve for any function $V$, but the infinite elements of the sequence ${\{W\_t\}}\_{t=1}^\infty$ are all choice variables. Bellman's main insight is that (under some conditions) the problem can be simplified. Instead of solving for an infinite number of variables, you could solve for a single "policy function" that tells you what to do each period as a function of the current state variables, and a "value function" that summarizes what is the value (in terms of today) of inducing each possible state in the future. Even if there is no uncertainty, it is clear that this value function is an endogenous object, since it must somehow capture the future stream of utility as a single function of the future state. In your case, that function is $V(W')$. This is why the Bellman equation:
$$V(W)=\max\_{W'\in[0,W]} u(W-W')+\beta V(W')$$
has two properties:

1) it lacks the subscripts on the $W$'s, this reflects that you will solve for $W'$ as a function of $W$, i.e. the policy function.

2) has a new object, $V$, that was not in the original problem and it is a function of $W$ (or $W'$) and just by eyeballing the two problems you can see that for them to be equivalent $V$ must be the discounted sum of future payoffs assuming future actions are also optimal. So it is not a choice variable, but definitely, an endogenous object that you need to find when solving the problem.

You can see that the problem of optimally choosing infinitely many objects is transformed into one of choosing a finite number (2) of functions, which are infinite-dimensional objects. Brilliant!

> **Comments:**
> **username_0**: Hi Regio, thanks for the clear explanation. I am getting a firmer grasp on the DP. One additional question is that why does the Bellman equation usually emphasize that the equation must hold for all (feasible) values of $W$? Also, I see that $V(W)$ is the current value function and $V(W')$ is the continuation value function, but how do you mean when you say it is "an endogenous object that you need to... the problem?" Just because it appears on both sides of the Bellman equation? I think what I am confused is that under the max operator $W'$ is choice variable but we say $V(\cdot)$ is the sol  
> **username_3**: Not sure I understand your question. But $V(W)$ is the value of currently having a stock of $W$ (that is why $\beta V(W')$ is the discounted value of having a stock of $W'$ next period). and for you to be able to know how much should you consume today, $u(W-W')$ it is necessary to know what payoff each of these consumptions will imply for tomorrow. Note that different consumptions induce different $W'$. That's why $V(\cdot)$ must be valid for all values of $W$.  
> **username_3**: Maybe it will be easier for you to see the max operator as some function $F$. then the  Bellman equation is of the form $V(W)=F(\cdot, V(W'))$, where $V$ is the unique function that satisfies this equality. (the uniqueness requires some conditions, but in most economics problems you will have uniqueness. Because $F$ is a max operator it can be challenging to find $V$, but there are some standard techniques that can be helpful.  
> **username_0**: Thanks for the comments Regio. It helps a lot. Do you have any reference material where I can see an example of a problem that has a full-fledged version of solving a DP program including the lagrangean, envelope conditions, and etc? I want to work out a full version if you have a credible reference? thanks  
> **username_3**: A classic reference is Thomas Sargent's [book](https://www.amazon.com/Dynamic-Macroeconomic-Theory-Thomas-Sargent/dp/0674218779/ref=asc_df_0674218779/?tag=hyprod-20&linkCode=df0&hvadid=312111907622&hvpos=1o3&hvnetw=g&hvrand=6249537410562009453&hvpone=&hvptwo=&hvqmt=&hvdev=c&hvdvcmdl=&hvlocint=&hvlocphy=9018948&hvtargid=pla-448229355571&psc=1). However if you do a quick search with  "Dynamic programming macroeconomics" you can find many lecture notes of introductory macro courses at the graduate level that will likely be helpful.  

---

### Answer by username_1 on 2019-06-13. Score: 1

What you want to maximize is the discounted stream of utility over time. To that end, you can set control variables over time (subject to some constraints, like technology, budget...). We are ultimately interested in the optimal setting of these variables (called the optimal policy).

Bellman's main insight is the principle of optimality (reminiscent of backward induction/subgame perfectness in game theory): suppose you choose some action today as part of an optimal policy, then the remaining action sequence must be part of an optimal policy from tomorrow on.

Before turning to the policy, we solve for the value function (this part is called the prediction problem). For a discrete action space, you could actually draw a complete decision tree and solve via backward induction. The more feasible, tractable, and elegant solution is to write the problem recursively with respect to the continuation value of the problem. Solving the Bellman Equation will give you that value function $V(\cdot)$, which is the *optimal* discounted value of the objective function, given the state you are in and an optimal policy from that state onwards. Note that the solution is thus a function, not a value.

From there, you can back out the sequence of actions (policy) that produces the optimal outcome. This part is called the control problem. Having solved the prediction problem, this is easy: the value function takes today's state and action as input and gives you the continuation value. So for each state, compute the continuation value for each possible action and pick the action that has the highest continuation value.

> **Comments:**
> **username_0**: Hi Frederic, can you elaborate more by editing your answer on when you say "you can back out the sequence of actions that produces the optimal outcome". Thanks,  
> **username_1**: Hi Frank, you are right, my first answer was a bit terse, I have expanded it a bit for clarity. I hope this helps.  
> **username_0**: Hi Fred, thanks for the elaboration. It helped a lot. I am reading few notes on this as it has been quite a long time the last time I dealt with the dynamic programming. When we use the term, the current value function $v(k)$ and the continuation value function, $v(k')$, what is the difference? Is it merely that we are putting the future capital stock as argument? Also, why is it the case that the Bellman equation has to hold for all feasible values of $k$? Why is this emphasized?  

---

### Answer by username_2 on 2019-06-13. Score: 0

Not sure what problem you are solving.
But, the next state can be stochastic and the reward can be stochastic.

For example, In Inventory problem, your action is the number of items you purchase now. But next state can be calculated by **current inventory(current state) + new purchase(action) - demand(unknown factor)** therefore it will be stochastic.

In stochastic problem usually, you have Expectation in your objective function.

> **Comments:**
> **username_0**: I am reviewing the deterministic optimal growth model, and this is not exactly the response I was looking for because stochastic growth model comes later. Thanks.  

---

