<p>You need to distinguish two cases:</p>
<ol>
<li>Your analytical example solves the <a href="https://www.sciencedirect.com/science/article/abs/pii/0022053172901354" rel="nofollow noreferrer">Brock/Mirman</a> model, which is uses log utility and full depreciation <span class="math-container">$\delta=1$</span>. With log utility, income and substitution effects offset each other. Full depreciation shuts off additional wealth effects. As a consequence, there is certainty equivalence: the agent behaves as if expected values would occur with certainty. This implies that the specific structure of the stochastic process for <span class="math-container">$Z$</span> is irrelevant. However, it is a very special case that.</li>
<li>In pretty much all other setups (like the CRRA case you refer to), the agent must form expectations about the future using the complete stochastic process. That considerably enlarges the state space. You can use arbitrary ARMA processes (which are all covariance stationary) as you suggest above. Representing them as a VAR(1) simplifies notation and facilitates numerical solutions. These approaches usually involve discretizing the exogenous process to obtain a finite state Markov chain as in e.g. <a href="https://www.sciencedirect.com/science/article/abs/pii/0165176586901680" rel="nofollow noreferrer">Tauchen (1986)</a>. Here, it is convenient to only deal with a generic VAR(1) process as the input without having to adjust codes to the specific ARMA structure.</li>
</ol>

CC BY-SA 4.0
2024-10-02T09:31:04.240Z
2024-10-02T09:31:04.240Z
jpfeifer
<p>Consider a simple stochastic dynamic programming growth model Bellman with full depreciation,</p>
<p><span class="math-container">$$V(K_{t},Z_{t})=\ln(C_{t})+\beta\mathbb{E}[V(K_{t+1},Z_{t+1})|Z_{t}]$$</span>
where
<span class="math-container">$$C_{t}=Z_{t}K_{t}^\alpha+(1-\delta)K_{t}-K_{t+1}$$</span>
and
<span class="math-container">$$Z_{t+2}=\rho_1Z_{t+1}+\rho_2Z_t+\phi\varepsilon_{t+1}+\varepsilon_{t+2};\varepsilon_{t+2}\sim N(0,\sigma^2).$$</span></p>
<p>Notice that the innovations/shocks follow a higher order stochastic process, with a moving average component. This could really be any structure, and my question remains the same.</p>
<p>If we use a standard process to solve for the Euler equation (forgive my abuses of time subscript notation), we have
<span class="math-container">$$\frac{1}{C_{t}}=\beta\mathbb{E}_t[\frac{\alpha Z_{t+1}K_{t}^{\alpha-1}}{Z_{t+1}K_{t+1}^\alpha-K_{t+2}}].$$</span></p>
<p>And by the method of undetermined coefficients, we can conjecture the policy function to be <span class="math-container">$K_{t+1}=sZ_{t}K_{t}^\alpha \implies K_{t+2}=sZ_{t+1}K_{t+1}^\alpha $</span>. Substituting this into the EE we have,
<span class="math-container">$$\frac{1}{C_{t}}=\beta\mathbb{E}_t[\frac{\alpha Z_{t+1}K_{t}^{\alpha-1}}{Z_{t+1}K_{t+1}^\alpha-sZ_{t+1}K_{t+1}^\alpha}]$$</span></p>
<p>From which we can simplify to,
<span class="math-container">$$\frac{1}{C_{t}}=\beta\mathbb{E}_t[\frac{\alpha}{K_{t+1}(1-s)}]$$</span>
which is non stochastic in the expectation. I won't continue, as I think I can ask my question now.</p>
<p><strong>Question:</strong> Most if not all of the DP problems I have seen have assumed either AR1 or iid shocks. I know that with a higher-order AR process that you need to redefine the state space such as the one from above:
<span class="math-container">$$Z_{t+2}=\rho_1Z_{t+1}+\rho_2Z_t+\phi\varepsilon_{t+1}+\varepsilon_{t+2}$$</span>
becomes some first order vector autoregressive process of the AR(2) process
<span class="math-container">$$A_{t+1}=HA_t+\phi E_{t+1}.$$</span>
where <span class="math-container">$A_{t+1}=[Z_{t+1}\quad Z_t]'$</span> and <span class="math-container">$H$</span> contains the coefficients of past shocks.</p>
<p>I don't see in my short derivation above where I would actually need to complicate things with the VAR. We make no assumptions about the stochastic process before Z' cancels out and we would need to take an expected value of it. It seems to me that the standard approach may work with higher order processes, but my intuition (maybe false) is telling me that it can't be that simple. Am I making a false assumption about the stationarity of the value function? If I am fine to do it using the standard approach from advanced macro, then what assumptions must we make about the stochastic process for this to be valid? Any ideas would be great, Thanks!</p>
<p><strong>edit:</strong>
If I use CRRA rather than log, then the expectation still has future values of Z in it, so then do I just need to expand the state space to include these?
<span class="math-container">$$V(K_t,Z_t,Z_{t-1},\varepsilon_{t-1})=U(C_t)+\beta\mathbb{E}[V(K_{t+1},Z_{t+1},Z_t, \varepsilon_t)|Z_t,Z_{t-1},\varepsilon_{t-1}]$$</span></p>

CC BY-SA 4.0
2024-10-01T05:15:28.690Z
Hi: Your third expression in your question, involving $Z^{\prime \prime}$ does not have any time subscripts. So, from a modelling perspective, that's not a higher order AR. Should there be time subscripts ? You do use them later when describing how every AR process should be converted to a VAR so that it can remain markov. That part was clear but maybe you can add t subscripts to the rest of it. I'm not guaranteeing that I can help but I might be able to tell you where to look. The conversion to the VAR(1) is pretty standard but I did not follow what your question was.
mark leeds
CC BY-SA 4.0
2024-10-01T05:24:49.497Z
Oh, to answer your sub-question, "I know that with a higher order markov process" should read as "with a higher order AR process".  An AR(1) is markov. But, by definition, any higher order AR process is not markov. That's why they convert higher order AR processes to VAR(1). But, as I said, you'll get more replies if you change to t-subscripting rather than primes. But maybe you are not doing that ?
mark leeds
CC BY-SA 4.0
2024-10-01T07:36:18.463Z
It seems you solved the Brock/Mirman model with log utility and full depreciation. Only in that case does the exact process for Z not matter because the problem becomes linear in logs.
jpfeifer
CC BY-SA 4.0
2024-10-01T17:50:09.837Z
@markleeds thanks for pointing out the notation issues. I've updated the AR process with time subscripts. I should have been more careful with notation. I guess my first question was whether my reasoning is right in converting the higher order process into a VAR(1) process, but you seem to have addressed that well. My second question is what conditions must hold about a higher order process for it to still be trivial? If there is strong seasonality, or a non-stationary exogenous deterministic process e.g. $z_{t+1}=f(t+1)+\varepsilon_{t+1}$ where f is some generic function.
giff
CC BY-SA 4.0
2024-10-01T17:57:35.720Z
@jpfeifer can you help me understand why this is? What is it about the log function that makes any stochastic process irrelevant? I get that the math works out this way, but it feels wrong for some reason!
giff
CC BY-SA 4.0
2024-10-01T18:02:39.433Z
I would greatly appreciate any book recommendations on handling higher-order processes (with seasonality and or non-stationarity). If in SLP or recursive macro theory, then a chapter or page number. Thanks.
giff
CC BY-SA 4.0
2024-10-01T20:33:24.157Z
whydaho99: I'm not familiar with the Brock/Mirman model so I can't help you there. I'll read through your edit and see whether the edits make sense from a time-series perspective. I also don't think I gave you an adequate answer on why they modify to a VAR(1). They might be trying to remain Markov but I'm not 100 percent sure.
mark leeds
CC BY-SA 4.0
2024-10-01T20:37:10.937Z
It looks improved but the line where you say: "I think it's called this" should be modified to say "higher order auto-regressive process" rather than "higher order markov process". I'll try to find a discussion of the conversion from higher order AR to vector VAR(1).
mark leeds
CC BY-SA 4.0
2024-10-02T19:40:17.833Z
giff: clearly, jpfeifer knows this material in depth so I'll drop out in that respect. But I do plan on finding some discussion of the conversion to a VAR. I'll be in touch with whatever I find and good luck with your understanding.
mark leeds
CC BY-SA 4.0
2024-10-02T22:29:06.450Z
@markleeds: Thank you for the help, I think I used too simplified a model in order to play around with the AR process, but the oversimplification was hiding the structure that I needed to understand the problem.
giff
CC BY-SA 4.0
2024-10-04T13:24:17.197Z
okay, but I'll try to find an explanation anyway. I think it's ( the switching to a vector ) done to make things markov. you're welcome too but I didn't do much helping.
mark leeds
<dynamic-programming><dynamic-optimization><stochastic-processes><bellman-equations><autoregressive>
Stochastic dynamic programming with history dependent shocks
giff
# Stochastic dynamic programming with history dependent shocks

(Asked by: username_0 on 2024-10-01)

Consider a simple stochastic dynamic programming growth model Bellman with full depreciation,

$$V(K\_{t},Z\_{t})=\ln(C\_{t})+\beta\mathbb{E}[V(K\_{t+1},Z\_{t+1})|Z\_{t}]$$
where
$$C\_{t}=Z\_{t}K\_{t}^\alpha+(1-\delta)K\_{t}-K\_{t+1}$$
and
$$Z\_{t+2}=\rho\_1Z\_{t+1}+\rho\_2Z\_t+\phi\varepsilon\_{t+1}+\varepsilon\_{t+2};\varepsilon\_{t+2}\sim N(0,\sigma^2).$$

Notice that the innovations/shocks follow a higher order stochastic process, with a moving average component. This could really be any structure, and my question remains the same.

If we use a standard process to solve for the Euler equation (forgive my abuses of time subscript notation), we have
$$\frac{1}{C\_{t}}=\beta\mathbb{E}\_t[\frac{\alpha Z\_{t+1}K\_{t}^{\alpha-1}}{Z\_{t+1}K\_{t+1}^\alpha-K\_{t+2}}].$$

And by the method of undetermined coefficients, we can conjecture the policy function to be $K\_{t+1}=sZ\_{t}K\_{t}^\alpha \implies K\_{t+2}=sZ\_{t+1}K\_{t+1}^\alpha $. Substituting this into the EE we have,
$$\frac{1}{C\_{t}}=\beta\mathbb{E}\_t[\frac{\alpha Z\_{t+1}K\_{t}^{\alpha-1}}{Z\_{t+1}K\_{t+1}^\alpha-sZ\_{t+1}K\_{t+1}^\alpha}]$$

From which we can simplify to,
$$\frac{1}{C\_{t}}=\beta\mathbb{E}\_t[\frac{\alpha}{K\_{t+1}(1-s)}]$$
which is non stochastic in the expectation. I won't continue, as I think I can ask my question now.

**Question:** Most if not all of the DP problems I have seen have assumed either AR1 or iid shocks. I know that with a higher-order AR process that you need to redefine the state space such as the one from above:
$$Z\_{t+2}=\rho\_1Z\_{t+1}+\rho\_2Z\_t+\phi\varepsilon\_{t+1}+\varepsilon\_{t+2}$$
becomes some first order vector autoregressive process of the AR(2) process
$$A\_{t+1}=HA\_t+\phi E\_{t+1}.$$
where $A\_{t+1}=[Z\_{t+1}\quad Z\_t]'$ and $H$ contains the coefficients of past shocks.

I don't see in my short derivation above where I would actually need to complicate things with the VAR. We make no assumptions about the stochastic process before Z' cancels out and we would need to take an expected value of it. It seems to me that the standard approach may work with higher order processes, but my intuition (maybe false) is telling me that it can't be that simple. Am I making a false assumption about the stationarity of the value function? If I am fine to do it using the standard approach from advanced macro, then what assumptions must we make about the stochastic process for this to be valid? Any ideas would be great, Thanks!

**edit:**
If I use CRRA rather than log, then the expectation still has future values of Z in it, so then do I just need to expand the state space to include these?
$$V(K\_t,Z\_t,Z\_{t-1},\varepsilon\_{t-1})=U(C\_t)+\beta\mathbb{E}[V(K\_{t+1},Z\_{t+1},Z\_t, \varepsilon\_t)|Z\_t,Z\_{t-1},\varepsilon\_{t-1}]$$

**Tags:** dynamic-programming, dynamic-optimization, stochastic-processes, bellman-equations, autoregressive
> **Comments:**
> **username_1**: Hi: Your third expression in your question, involving $Z^{\prime \prime}$ does not have any time subscripts. So, from a modelling perspective, that's not a higher order AR. Should there be time subscripts ? You do use them later when describing how every AR process should be converted to a VAR so that it can remain markov. That part was clear but maybe you can add t subscripts to the rest of it. I'm not guaranteeing that I can help but I might be able to tell you where to look. The conversion to the VAR(1) is pretty standard but I did not follow what your question was.  
> **username_1**: Oh, to answer your sub-question, "I know that with a higher order markov process" should read as "with a higher order AR process".  An AR(1) is markov. But, by definition, any higher order AR process is not markov. That's why they convert higher order AR processes to VAR(1). But, as I said, you'll get more replies if you change to t-subscripting rather than primes. But maybe you are not doing that ?  
> **username_2**: It seems you solved the Brock/Mirman model with log utility and full depreciation. Only in that case does the exact process for Z not matter because the problem becomes linear in logs.  
> **username_0**: @markleeds thanks for pointing out the notation issues. I've updated the AR process with time subscripts. I should have been more careful with notation. I guess my first question was whether my reasoning is right in converting the higher order process into a VAR(1) process, but you seem to have addressed that well. My second question is what conditions must hold about a higher order process for it to still be trivial? If there is strong seasonality, or a non-stationary exogenous deterministic process e.g. $z_{t+1}=f(t+1)+\varepsilon_{t+1}$ where f is some generic function.  
> **username_0**: @jpfeifer can you help me understand why this is? What is it about the log function that makes any stochastic process irrelevant? I get that the math works out this way, but it feels wrong for some reason!  
> **username_0**: I would greatly appreciate any book recommendations on handling higher-order processes (with seasonality and or non-stationarity). If in SLP or recursive macro theory, then a chapter or page number. Thanks.  
> **username_1**: whydaho99: I'm not familiar with the Brock/Mirman model so I can't help you there. I'll read through your edit and see whether the edits make sense from a time-series perspective. I also don't think I gave you an adequate answer on why they modify to a VAR(1). They might be trying to remain Markov but I'm not 100 percent sure.  
> **username_1**: It looks improved but the line where you say: "I think it's called this" should be modified to say "higher order auto-regressive process" rather than "higher order markov process". I'll try to find a discussion of the conversion from higher order AR to vector VAR(1).  
> **username_1**: giff: clearly, jpfeifer knows this material in depth so I'll drop out in that respect. But I do plan on finding some discussion of the conversion to a VAR. I'll be in touch with whatever I find and good luck with your understanding.  
> **username_0**: @markleeds: Thank you for the help, I think I used too simplified a model in order to play around with the AR process, but the oversimplification was hiding the structure that I needed to understand the problem.  
> **username_1**: okay, but I'll try to find an explanation anyway. I think it's ( the switching to a vector ) done to make things markov. you're welcome too but I didn't do much helping.  

---

## 1 answers

### Answer by username_2 on 2024-10-02. Score: 1

You need to distinguish two cases:

1. Your analytical example solves the [Brock/Mirman](https://www.sciencedirect.com/science/article/abs/pii/0022053172901354) model, which is uses log utility and full depreciation $\delta=1$. With log utility, income and substitution effects offset each other. Full depreciation shuts off additional wealth effects. As a consequence, there is certainty equivalence: the agent behaves as if expected values would occur with certainty. This implies that the specific structure of the stochastic process for $Z$ is irrelevant. However, it is a very special case that.
2. In pretty much all other setups (like the CRRA case you refer to), the agent must form expectations about the future using the complete stochastic process. That considerably enlarges the state space. You can use arbitrary ARMA processes (which are all covariance stationary) as you suggest above. Representing them as a VAR(1) simplifies notation and facilitates numerical solutions. These approaches usually involve discretizing the exogenous process to obtain a finite state Markov chain as in e.g. [Tauchen (1986)](https://www.sciencedirect.com/science/article/abs/pii/0165176586901680). Here, it is convenient to only deal with a generic VAR(1) process as the input without having to adjust codes to the specific ARMA structure.


---

