<p><strong>Q1.</strong>
<strike>Assumptions 1 and 2 are independent of one another.
Assumption 2 is an assumption of the fundamentals of the game, the setup if you will, and have no restrictions of the solutions. Its merely a description of the game.</p>
<p>Assumption 1 on the other hand, is an assumption on the choice of the class of equilibria we are interested in. I agree that Assumption 2 makes it natural to look at Markov perfect equilibria, but there is no such inherent need to do so.</strike></p>
<p>EDIT: See Micheal's comment below for a better answer.</p>
<p><strong>Q2.</strong>
Your latter interpretation is the one I would go for.</p>
<p><strong>Q3.</strong>
A SPNE here would be a time and history dependent strategy. So
<span class="math-container">$$
a_{it}: \times_{i = 1}^{t} (\mathcal{A} \times \mathcal{X})_i \rightarrow \mathcal{A}
$$</span>
i.e. the stategy at period <span class="math-container">$t$</span> under state <span class="math-container">$x_t$</span> depends on the entire history of realised states and actions played.</p>
<p>Its clear a Markov strategy is a special case of the above.</p>

CC BY-SA 4.0
2022-02-22T22:19:06.647Z
Assumptions 1 and 2 are not quite independent. If the state variable follows a first-order Markov process and every other player plays according to a Markovian strategy, then the decision problem a player faces at each state will be Markovian too and allow for Markovian best responses. Without the state following a first-order Markov process, there is little hope that SPNE in Markovian strategies exist.
Michael Greinecker
CC BY-SA 4.0
2022-02-22T22:32:15.387Z
That is what I thought too.
Star
CC BY-SA 4.0
2022-02-22T23:32:06.290Z
I've edited my answer to reflect the better insight provided by Michael.
Walrasian Auctioneer
CC BY-SA 4.0
2022-02-22T22:11:52.000Z
2022-02-22T23:31:41.903Z
2022-02-22T23:31:41.903Z
Walrasian Auctioneer
<p>I am studying dynamic games and I'm fundamentally confused about the relation between Markov Perfect Nash Equilibrium and Markovian evolution of the state. Before illustrating my doubt, let me describe the basic framework.</p>
<p>Consider a game played by N players that we index by <span class="math-container">$i=1,...,N$</span>. Time is discrete and indexed by <span class="math-container">$t$</span>. In each period <span class="math-container">$t$</span>, every player <span class="math-container">$i$</span> chooses an action <span class="math-container">$a_{it}\in  \mathcal{A}$</span>, where <span class="math-container">$\mathcal{A}$</span> is finite and fixed across time/players for simplicity. <span class="math-container">$a_t\equiv (a_{1t},..., a_{Nt})$</span> is the vector of all players' actions in period t. In each period <span class="math-container">$t$</span>, each player <span class="math-container">$i$</span>  gets a payoff, <span class="math-container">$\pi_i(a_t, x_t)$</span>, that depends on <span class="math-container">$a_t$</span> and on a vector of common knowledge state variables, <span class="math-container">$x_t$</span>, with support <span class="math-container">$\mathcal{X}$</span>. Each player <span class="math-container">$i$</span> chooses the actions that maximise his expected and discounted flow of profits
<span class="math-container">$$
E_t(\sum_{s=0}^\infty \beta_i \pi(a_t, x_t))
$$</span>
where <span class="math-container">$\beta_i$</span> is the discount factor.</p>
<p>It is common to make two assumptions in applied work:</p>
<p><strong>1.</strong> The players play <strong>Markov Perfect Nash Equilibrium</strong> (MPNE). That is, their strategies in period <span class="math-container">$t$</span> are functions only of the payoff-relevant state variables at the same period.</p>
<p><strong>2.</strong> The vector of state variables, <span class="math-container">$x_t$</span>, follows a <strong>first order controlled Markov process</strong> with transition CDF <span class="math-container">$F(x_{t+1}| x_t, a_t)$</span>.</p>
<p>My understanding is that the payoff-relevant state variables (mentioned in ass. 1) are the state variables explicitly entering <span class="math-container">$\pi$</span>. In this case, just <span class="math-container">$x_t$</span>.
Therefore, the players' strategies are <span class="math-container">$\alpha\equiv \{\alpha_i(x_t): i=1,...,N \text{ and } x_t\in \mathcal{X}\}$</span>.</p>
<p><span class="math-container">$\alpha$</span> is a MPNE if it satisfies
<span class="math-container">$$
(*) \quad \alpha_i(x_t)= \text{argmax}_{a_{it}\in \mathcal{A}} \Big\{ \pi_i (a_{it}, \{\alpha_j(x_t)\}_{j\neq i}, x_t)+\beta_i \int V_i^\alpha(x_{t+1}) d F(x_{t+1}| x_t, a_{it}, \{\alpha_j(x_t)\}_{j\neq i}) \Big\}
$$</span>
for each player <span class="math-container">$i$</span> and state <span class="math-container">$x_t$</span>, where <span class="math-container">$V_i^\alpha$</span> is the value function uniquely solving the Bellman equation:
<span class="math-container">$$
(**) \quad  V_i^\alpha(x_t)=\max_{a_{it}\in \mathcal{A}} \Big\{ \pi_i (a_{it}, \{\alpha_j(x_t)\}_{j\neq i}, x_t)+\beta_i \int V_i^\alpha(x_{t+1}) d F(x_{t+1}| x_t, a_{it}, \{\alpha_j(x_t)\}_{j\neq i}) \Big\}
$$</span></p>
<p><strong>Questions:</strong></p>
<ul>
<li><p>I am confused about the relation between ass. 1 and 2. In particular, it seems to me that ass. 2 is &quot;<strong>necessary</strong>&quot; for ass. 1. In fact, suppose that <span class="math-container">$x_t$</span>  follows a <strong>second</strong> order controlled Markov process with transition CDF <span class="math-container">$F(x_{t+1}| x_t, a_t, x_{t-1}, a_{t-1})$</span>. Then, <span class="math-container">$x_{t-1}, a_{t-1}$</span>  would  appear in <span class="math-container">$(*)$</span> and <span class="math-container">$(**)$</span>, in addition to <span class="math-container">$x_t$</span>. In turn, this would invalidate the fact that the players' strategies can depend on <span class="math-container">$x_t$</span> only.</p>
</li>
<li><p>Here, perhaps, I'm misunderstanding the definition of &quot;payoff-relevant state variables&quot;. Are these the state variables explicitly entering <span class="math-container">$\pi$</span> (as I initially thought), or are these the state variables that matter for the evolution of the state? The latter interpretation, if correct, would clear any confusion: suppose, for instance, that <span class="math-container">$x_t$</span>  follows a <strong>second</strong> order controlled Markov process with transition CDF <span class="math-container">$F(x_{t+1}| x_t, a_t, x_{t-1}, a_{t-1})$</span>; in this case, the payoff-relevant state variables would be <span class="math-container">$x_t, x_{t-1}, a_{t-1}$</span>.
; in turn, a MPNE would be a strategy <span class="math-container">$\alpha_i(x_t, x_{t-1}, a_{t-1})$</span> for each player <span class="math-container">$i$</span>.</p>
</li>
<li><p>Suppose that 2 holds. Could you show me how does the definition of MPNE differ from a classic definition of <strong>subgame perfect Nash equilibrium</strong>? In particular, if the state evolves as a first order Markov, why should a player condition his strategies on  past history?</p>
</li>
</ul>

<dynamic-games><markov-chain>
Relation between Markov Perfect Nash Equilibrium and Markovian evolution of the state
Star
# Relation between Markov Perfect Nash Equilibrium and Markovian evolution of the state

(Asked by: username_0 on 2022-02-22)

I am studying dynamic games and I'm fundamentally confused about the relation between Markov Perfect Nash Equilibrium and Markovian evolution of the state. Before illustrating my doubt, let me describe the basic framework.

Consider a game played by N players that we index by $i=1,...,N$. Time is discrete and indexed by $t$. In each period $t$, every player $i$ chooses an action $a\_{it}\in \mathcal{A}$, where $\mathcal{A}$ is finite and fixed across time/players for simplicity. $a\_t\equiv (a\_{1t},..., a\_{Nt})$ is the vector of all players' actions in period t. In each period $t$, each player $i$ gets a payoff, $\pi\_i(a\_t, x\_t)$, that depends on $a\_t$ and on a vector of common knowledge state variables, $x\_t$, with support $\mathcal{X}$. Each player $i$ chooses the actions that maximise his expected and discounted flow of profits
$$
E\_t(\sum\_{s=0}^\infty \beta\_i \pi(a\_t, x\_t))
$$
where $\beta\_i$ is the discount factor.

It is common to make two assumptions in applied work:

**1.** The players play **Markov Perfect Nash Equilibrium** (MPNE). That is, their strategies in period $t$ are functions only of the payoff-relevant state variables at the same period.

**2.** The vector of state variables, $x\_t$, follows a **first order controlled Markov process** with transition CDF $F(x\_{t+1}| x\_t, a\_t)$.

My understanding is that the payoff-relevant state variables (mentioned in ass. 1) are the state variables explicitly entering $\pi$. In this case, just $x\_t$.
Therefore, the players' strategies are $\alpha\equiv \{\alpha\_i(x\_t): i=1,...,N \text{ and } x\_t\in \mathcal{X}\}$.

$\alpha$ is a MPNE if it satisfies
$$
(\*) \quad \alpha\_i(x\_t)= \text{argmax}\_{a\_{it}\in \mathcal{A}} \Big\{ \pi\_i (a\_{it}, \{\alpha\_j(x\_t)\}\_{j\neq i}, x\_t)+\beta\_i \int V\_i^\alpha(x\_{t+1}) d F(x\_{t+1}| x\_t, a\_{it}, \{\alpha\_j(x\_t)\}\_{j\neq i}) \Big\}
$$
for each player $i$ and state $x\_t$, where $V\_i^\alpha$ is the value function uniquely solving the Bellman equation:
$$
(\*\*) \quad V\_i^\alpha(x\_t)=\max\_{a\_{it}\in \mathcal{A}} \Big\{ \pi\_i (a\_{it}, \{\alpha\_j(x\_t)\}\_{j\neq i}, x\_t)+\beta\_i \int V\_i^\alpha(x\_{t+1}) d F(x\_{t+1}| x\_t, a\_{it}, \{\alpha\_j(x\_t)\}\_{j\neq i}) \Big\}
$$

**Questions:**

* I am confused about the relation between ass. 1 and 2. In particular, it seems to me that ass. 2 is "**necessary**" for ass. 1. In fact, suppose that $x\_t$ follows a **second** order controlled Markov process with transition CDF $F(x\_{t+1}| x\_t, a\_t, x\_{t-1}, a\_{t-1})$. Then, $x\_{t-1}, a\_{t-1}$ would appear in $(\*)$ and $(\*\*)$, in addition to $x\_t$. In turn, this would invalidate the fact that the players' strategies can depend on $x\_t$ only.
* Here, perhaps, I'm misunderstanding the definition of "payoff-relevant state variables". Are these the state variables explicitly entering $\pi$ (as I initially thought), or are these the state variables that matter for the evolution of the state? The latter interpretation, if correct, would clear any confusion: suppose, for instance, that $x\_t$ follows a **second** order controlled Markov process with transition CDF $F(x\_{t+1}| x\_t, a\_t, x\_{t-1}, a\_{t-1})$; in this case, the payoff-relevant state variables would be $x\_t, x\_{t-1}, a\_{t-1}$.
  ; in turn, a MPNE would be a strategy $\alpha\_i(x\_t, x\_{t-1}, a\_{t-1})$ for each player $i$.
* Suppose that 2 holds. Could you show me how does the definition of MPNE differ from a classic definition of **subgame perfect Nash equilibrium**? In particular, if the state evolves as a first order Markov, why should a player condition his strategies on past history?

**Tags:** dynamic-games, markov-chain

---

## 1 answers

### Answer by username_1 on 2022-02-22. Score: 1 | Accepted answer

**Q1.**
Assumptions 1 and 2 are independent of one another.
Assumption 2 is an assumption of the fundamentals of the game, the setup if you will, and have no restrictions of the solutions. Its merely a description of the game.

Assumption 1 on the other hand, is an assumption on the choice of the class of equilibria we are interested in. I agree that Assumption 2 makes it natural to look at Markov perfect equilibria, but there is no such inherent need to do so.

EDIT: See Micheal's comment below for a better answer.

**Q2.**
Your latter interpretation is the one I would go for.

**Q3.**
A SPNE here would be a time and history dependent strategy. So
$$
a\_{it}: \times\_{i = 1}^{t} (\mathcal{A} \times \mathcal{X})\_i \rightarrow \mathcal{A}
$$
i.e. the stategy at period $t$ under state $x\_t$ depends on the entire history of realised states and actions played.

Its clear a Markov strategy is a special case of the above.

> **Comments:**
> **username_2**: Assumptions 1 and 2 are not quite independent. If the state variable follows a first-order Markov process and every other player plays according to a Markovian strategy, then the decision problem a player faces at each state will be Markovian too and allow for Markovian best responses. Without the state following a first-order Markov process, there is little hope that SPNE in Markovian strategies exist.  
> **username_0**: That is what I thought too.  
> **username_1**: I've edited my answer to reflect the better insight provided by Michael.  

---

