<p>It appears that what you do is not a Taylor expansion, but an application of the mean-value theorem (as one should). If it was a Taylor expansion, apart from the remainder, the gradient would have to be evaluated at $\beta_0$. With the mean-value theorem, there is no remainder, and you evaluate the gradient at some $\bar \beta$ that always lies between $\beta$ and $\hat \beta$.  </p>

<p>As regards the issue of second derivatives/Hessian, officially speaking they/it only "temporarily" appear in the derivation of asymptotic normality of the non-linear least -squares estimator, but they vanish asymptotically (while in the Maximum likelihood estimator the Hessian stays there).  </p>

<p>Apart from that, we want to minimize the <em>sum</em> of squared residuals so, indexing the observations by $i$, and using much simpler notation (you will have to adjust it to vector-matrix notation), you set out to minimize $\sum_i[u_i(\beta)]^2 = \sum_i[y_i-h(\mathbf x_i,\beta)]^2$ w.r. to the vector $\beta$, in order to obtain a $\hat \beta$. So your FOC is (suppressing the regressors and passing the $i$ index to the function $h$)</p>

<p>$$\hat \beta : \sum_i\frac {\partial }{\partial \beta}[y_i-h_i(\beta)]^2 = 0 \implies \sum_i2[y_i-h_i(\hat \beta)] \frac {\partial h_i (\hat \beta)}{\partial \beta} =0, $$</p>

<p>Ignore "$2$" and apply the mean value theorem <em>to the whole expression</em>  to get </p>

<p>$$\sum_i[y_i-h_i(\beta)] \frac {\partial h_i (\hat \beta)}{\partial \beta} =  \sum_i[y_i-h_i(\beta_0)] \frac {\partial h_i (\beta_0)}{\partial \beta} \\ 
+ (\hat \beta -\beta) \sum_i\left [-\frac {\partial h_i (\bar \beta)}{\partial \beta}\frac {\partial h_i (\bar \beta)}{\partial \beta}+ [y_i-h_i(\bar \beta)]\frac {\partial^2 h_i (\bar \beta)}{\partial \beta^2}\right] =0$$</p>

<p>Note that $y_i-h_i(\beta_0) = u_i$, the true error, divide by $1/n$ and then multiply by $\sqrt n$ as you are allowed to and re-arrange to get </p>

<p>$$\sqrt n (\hat \beta -\beta) = -\left (\frac 1n\sum_i\left [-\frac {\partial h_i (\bar \beta)}{\partial \beta}\frac {\partial h_i (\bar \beta)}{\partial \beta}\right] + \frac 1n\sum_i\left [[y_i-h_i(\bar \beta)]\frac {\partial^2 h_i (\bar \beta)}{\partial \beta^2}\right]\right)^{-1} \cdot \left(\frac 1{\sqrt n} \sum_iu_i \frac {\partial h_i (\beta_0)}{\partial \beta} \right) $$</p>

<p>Now, we consider asymptotic normality, <em>given</em> that consistency holds,  $\hat \beta \xrightarrow{p} \beta_0$. Since $\bar \beta$ is sandwiched between $\hat \beta$ and $\beta_0$, it follows that $\bar \beta \xrightarrow{p} \beta_0$ also. This means that</p>

<p>$$\frac 1n\sum_i\left [[y_i-h_i(\bar \beta)]\frac {\partial^2 h_i (\bar \beta)}{\partial \beta^2}\right] \xrightarrow{p} \frac 1n\sum_i\left [[y_i-h_i(\beta_0)]\frac {\partial^2 h_i (\beta_0)}{\partial \beta^2}\right] \\= \frac 1n\sum_i\left [E(u_i) E\frac {\partial^2 h_i (\beta_0)}{\partial \beta^2}\right] =0$$</p>

<p>because $E(u_i) =0$.</p>

<p>So this term vanishes asymptotically and we are left with (cancelling also the negative sings)</p>

<p>$$\sqrt n (\hat \beta -\beta) \xrightarrow{d} \left (\text {plim}\frac 1n\sum_i\left [\frac {\partial h_i (\beta_0)}{\partial \beta}\frac {\partial h_i (\beta_0)}{\partial \beta}\right] \right)^{-1} \cdot \left(\frac 1{\sqrt n} \sum_iu_i \frac {\partial h_i (\beta_0)}{\partial \beta} \right) $$</p>

<p>You have to assume that the first sum converges to something positive definite, and the second converges in distribution to a normal random variable, and you do indeed make these assumptions (or deeper ones that lead to them). I don't have the specific book you mention, but you can compare the above to Davidson &amp; McKinnon "Econometric Theory and Methods"(2004) ch. 6, around eq. $(6.30)$.  </p>

CC BY-SA 3.0
2016-04-12T10:06:22.973Z
Many thanks for your answer Alecos. It has helped a lot. ;)
An old man in the sea.
CC BY-SA 3.0
2016-04-12T14:20:22.557Z
Alecos, would you mind checking if my answer below is correct? Maybe I could have used a more useful notation. Feel free to edit it, if you would like to. ;)
An old man in the sea.
CC BY-SA 3.0
2016-04-11T20:03:05.337Z
2016-04-11T20:03:05.337Z
Alecos Papadopoulos
<p>Based on Alecos Papadopoulos answer, I'm posting an answer with matrix notation.
I'll change the notation a bit, to make it easier to understand.</p>

<p>The FOC is $D_\mathbf{x}( \beta )^T(\mathbf{y}-\mathbf{x}(\beta))=0$. This gives a function in $\beta$, where $D_\mathbf{x}( \beta )$ is a matrix of dim $N\times K$, with element $\frac{\partial x_n}{\partial\beta_k}(\beta)$.</p>

<p>So, applying the Taylor expansion(1st order)/mean-value theorem to each component of the LHS of the equality, we obtain</p>

<p>\begin{equation}
D_\mathbf{x}(\beta_0)^T \mathbf{u} +\big[H_\mathbf{x}(\bar\beta)-D_\mathbf{x}(\bar\beta)^T D_\mathbf{x}(\bar\beta)\big](\hat\beta-\beta_0)=0 
\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \text{(a)}
\end{equation}</p>

<p>where $H_\mathbf{x}(\bar\beta)$ represents a matrix $K\times K$, with each element being $H_{ij}(\bar\beta_i)[\mathbf{y}-\mathbf{x}(\bar\beta_i)]$, where $H_{ij}(\bar\beta_i)=\begin{bmatrix}
    \frac{\partial^2 x_1}{\partial\beta_j\partial\beta_i}(\bar\beta_i) &amp; \dots  &amp; \frac{\partial^2 x_n}{\partial\beta_j\partial\beta_i}(\bar\beta_i) 
\end{bmatrix}$. 
Similarly, $D_\mathbf{x}(\bar\beta)^T D_\mathbf{x}(\bar\beta)$ is the Gramian of $D_\mathbf{x}(\bar\beta)$, with $ij$-th element $\sum_l^n\frac{\partial x_l}{\partial\beta_j}(\bar\beta_i)\frac{\partial x_l}{\partial\beta_i}(\bar\beta_i)$.</p>

<p>From (a), $n^{\frac{1}{2}}(\hat\beta-\beta_0)=-\left(\frac{1}{n}H_\mathbf{x}(\bar\beta)-\frac{1}{n}D_\mathbf{x}(\bar\beta)^T D_\mathbf{x}(\bar\beta)\right)^{-1}n^{-\frac{1}{2}}D_\mathbf{x}(\beta_0)^T \mathbf{u}$.</p>

<p>We know from the non-linear model that $n^{-\frac{1}{2}}D_\mathbf{x}(\beta_0)^T \mathbf{u}=n^{-\frac{1}{2}}\sum_i^n u_i D_{\mathbf{x}i}(\beta_0)^T\rightarrow^d N\left(\underbrace{E(u_i D_{\mathbf{x}i}(\beta_0)^T)}_{=\mathbf{0}},\underbrace{\lim 1/n \sum E(u_i^2 D_{\mathbf{x}i}(\beta_0)D_{\mathbf{x}i}(\beta_0)^T)}_{\sigma_0^2 S_{D_0^TD_0}}\right)$</p>

<p>by a CLT.</p>

<p>Since we know that $\bar \beta\rightarrow^p \beta_0$, we have $\frac{1}{n}H_\mathbf{x}(\bar\beta)\rightarrow^p \left[\frac{1}{n}\sum \underbrace{E\left( \underbrace{(y_i-x_i\beta_0)}_{=u_i}\frac{\partial^2 x_1}{\partial\beta_j\partial\beta_i}(\beta_0)\right)}_{=0}\right]_{K\times K}=\mathbf{0}$.</p>

<p>Similarly, we have that $\frac{1}{n}D_\mathbf{x}(\bar\beta)^T D_\mathbf{x}(\bar\beta)\rightarrow^p S_{D_0^TD_0}$.</p>

<p>Thus, we can conclude that $n^{\frac{1}{2}}(\hat\beta-\beta_0)\rightarrow^p N(\mathbf{0},\sigma_0^2 S_{D_0^TD_0}^{-1})$.</p>

CC BY-SA 3.0
2016-04-12T15:05:46.413Z
Looks ok.The Asymptotic variance depends on the error being homoskedastic and non-autocorrelated of course.
Alecos Papadopoulos
CC BY-SA 3.0
2016-04-12T15:07:36.347Z
Opportunity taken, please consider also upvoting answers you find helpful (apart from the green mark). I think various statistics related to the site take into account only votes and not green marks.
Alecos Papadopoulos
CC BY-SA 3.0
2016-04-12T15:11:59.983Z
@AlecosPapadopoulos Done! and Thanks for the review. ;)
An old man in the sea.
CC BY-SA 3.0
2016-04-12T14:18:32.890Z
2016-04-12T15:05:41.820Z
2016-04-12T15:05:41.820Z
An old man in the sea.
<p>Our model is $Y=X(\beta_0)+u$, where $u\sim IID(0,\sigma_0^2I)$, and $X(\beta)$ is a non-linear function of the beta.</p>

<p>When trying to minimize the $SSR(\beta)$ we get the following FOC:</p>

<p>$\nabla X(\beta)^T(Y-X(\beta))=0$, where $\nabla X(\beta)$ is the gradient.</p>

<p>Well, the FOC is equivalent to $n^{(-1/2)}(\nabla X(\beta)^T(X(\beta_0)+u-X(\beta))=0$.</p>

<p>If we apply a taylor expansion of the first order to each component $X_t(\beta)$ of $X(\beta)$, we obtain $X_t(\beta)=X_t(\beta_0)+\nabla X(\bar\beta_{(t)})^T(\beta-\beta_0)$, where $\bar\beta_{(t)}$ is a point in the line segment that joins $\beta$ and $\beta_0$. This point may be different for each taylor expansion we do, and that's why it's indexed by $t$.</p>

<p>Inserting the taylor expansion in the FOC:
$n^{(-1/2)}(\nabla X(\beta)^T(u-\nabla \bar X^T(\beta-\beta_0))=0$, where $\nabla \bar X$ is the matrix with $\nabla X(\bar\beta_{(i)})$ as each i-th column.</p>

<p>Are all of the above calculations correct? I ask this because in this <a href="http://www.amazon.co.uk/Econometric-Theory-Methods-International-Edition/dp/0195391055?ie=UTF8&amp;ref_=asap_bc" rel="nofollow">book</a>, the authors state in page 225 that we should obtain a term with second derivatives of $X(\beta)$... I do not understand why this is.</p>

<p>Any help would be appreciated</p>

<econometrics>
Proving the asymptotic normality of th Non-linear Least Squares estimator
An old man in the sea.
# Proving the asymptotic normality of th Non-linear Least Squares estimator

(Asked by: username_0 on 2016-04-11)

Our model is $Y=X(\beta\_0)+u$, where $u\sim IID(0,\sigma\_0^2I)$, and $X(\beta)$ is a non-linear function of the beta.

When trying to minimize the $SSR(\beta)$ we get the following FOC:

$\nabla X(\beta)^T(Y-X(\beta))=0$, where $\nabla X(\beta)$ is the gradient.

Well, the FOC is equivalent to $n^{(-1/2)}(\nabla X(\beta)^T(X(\beta\_0)+u-X(\beta))=0$.

If we apply a taylor expansion of the first order to each component $X\_t(\beta)$ of $X(\beta)$, we obtain $X\_t(\beta)=X\_t(\beta\_0)+\nabla X(\bar\beta\_{(t)})^T(\beta-\beta\_0)$, where $\bar\beta\_{(t)}$ is a point in the line segment that joins $\beta$ and $\beta\_0$. This point may be different for each taylor expansion we do, and that's why it's indexed by $t$.

Inserting the taylor expansion in the FOC:
$n^{(-1/2)}(\nabla X(\beta)^T(u-\nabla \bar X^T(\beta-\beta\_0))=0$, where $\nabla \bar X$ is the matrix with $\nabla X(\bar\beta\_{(i)})$ as each i-th column.

Are all of the above calculations correct? I ask this because in this [book](http://www.amazon.co.uk/Econometric-Theory-Methods-International-Edition/dp/0195391055?ie=UTF8&ref_=asap_bc), the authors state in page 225 that we should obtain a term with second derivatives of $X(\beta)$... I do not understand why this is.

Any help would be appreciated

**Tags:** econometrics

---

## 2 answers

### Answer by username_1 on 2016-04-11. Score: 2 | Accepted answer

It appears that what you do is not a Taylor expansion, but an application of the mean-value theorem (as one should). If it was a Taylor expansion, apart from the remainder, the gradient would have to be evaluated at $\beta\_0$. With the mean-value theorem, there is no remainder, and you evaluate the gradient at some $\bar \beta$ that always lies between $\beta$ and $\hat \beta$.

As regards the issue of second derivatives/Hessian, officially speaking they/it only "temporarily" appear in the derivation of asymptotic normality of the non-linear least -squares estimator, but they vanish asymptotically (while in the Maximum likelihood estimator the Hessian stays there).

Apart from that, we want to minimize the *sum* of squared residuals so, indexing the observations by $i$, and using much simpler notation (you will have to adjust it to vector-matrix notation), you set out to minimize $\sum\_i[u\_i(\beta)]^2 = \sum\_i[y\_i-h(\mathbf x\_i,\beta)]^2$ w.r. to the vector $\beta$, in order to obtain a $\hat \beta$. So your FOC is (suppressing the regressors and passing the $i$ index to the function $h$)

$$\hat \beta : \sum\_i\frac {\partial }{\partial \beta}[y\_i-h\_i(\beta)]^2 = 0 \implies \sum\_i2[y\_i-h\_i(\hat \beta)] \frac {\partial h\_i (\hat \beta)}{\partial \beta} =0, $$

Ignore "$2$" and apply the mean value theorem *to the whole expression* to get

$$\sum\_i[y\_i-h\_i(\beta)] \frac {\partial h\_i (\hat \beta)}{\partial \beta} = \sum\_i[y\_i-h\_i(\beta\_0)] \frac {\partial h\_i (\beta\_0)}{\partial \beta} \\
+ (\hat \beta -\beta) \sum\_i\left [-\frac {\partial h\_i (\bar \beta)}{\partial \beta}\frac {\partial h\_i (\bar \beta)}{\partial \beta}+ [y\_i-h\_i(\bar \beta)]\frac {\partial^2 h\_i (\bar \beta)}{\partial \beta^2}\right] =0$$

Note that $y\_i-h\_i(\beta\_0) = u\_i$, the true error, divide by $1/n$ and then multiply by $\sqrt n$ as you are allowed to and re-arrange to get

$$\sqrt n (\hat \beta -\beta) = -\left (\frac 1n\sum\_i\left [-\frac {\partial h\_i (\bar \beta)}{\partial \beta}\frac {\partial h\_i (\bar \beta)}{\partial \beta}\right] + \frac 1n\sum\_i\left [[y\_i-h\_i(\bar \beta)]\frac {\partial^2 h\_i (\bar \beta)}{\partial \beta^2}\right]\right)^{-1} \cdot \left(\frac 1{\sqrt n} \sum\_iu\_i \frac {\partial h\_i (\beta\_0)}{\partial \beta} \right) $$

Now, we consider asymptotic normality, *given* that consistency holds, $\hat \beta \xrightarrow{p} \beta\_0$. Since $\bar \beta$ is sandwiched between $\hat \beta$ and $\beta\_0$, it follows that $\bar \beta \xrightarrow{p} \beta\_0$ also. This means that

$$\frac 1n\sum\_i\left [[y\_i-h\_i(\bar \beta)]\frac {\partial^2 h\_i (\bar \beta)}{\partial \beta^2}\right] \xrightarrow{p} \frac 1n\sum\_i\left [[y\_i-h\_i(\beta\_0)]\frac {\partial^2 h\_i (\beta\_0)}{\partial \beta^2}\right] \\= \frac 1n\sum\_i\left [E(u\_i) E\frac {\partial^2 h\_i (\beta\_0)}{\partial \beta^2}\right] =0$$

because $E(u\_i) =0$.

So this term vanishes asymptotically and we are left with (cancelling also the negative sings)

$$\sqrt n (\hat \beta -\beta) \xrightarrow{d} \left (\text {plim}\frac 1n\sum\_i\left [\frac {\partial h\_i (\beta\_0)}{\partial \beta}\frac {\partial h\_i (\beta\_0)}{\partial \beta}\right] \right)^{-1} \cdot \left(\frac 1{\sqrt n} \sum\_iu\_i \frac {\partial h\_i (\beta\_0)}{\partial \beta} \right) $$

You have to assume that the first sum converges to something positive definite, and the second converges in distribution to a normal random variable, and you do indeed make these assumptions (or deeper ones that lead to them). I don't have the specific book you mention, but you can compare the above to Davidson & McKinnon "Econometric Theory and Methods"(2004) ch. 6, around eq. $(6.30)$.

> **Comments:**
> **username_0**: Many thanks for your answer Alecos. It has helped a lot. ;)  
> **username_0**: Alecos, would you mind checking if my answer below is correct? Maybe I could have used a more useful notation. Feel free to edit it, if you would like to. ;)  

---

### Answer by username_0 on 2016-04-12. Score: 2

Based on Alecos Papadopoulos answer, I'm posting an answer with matrix notation.
I'll change the notation a bit, to make it easier to understand.

The FOC is $D\_\mathbf{x}( \beta )^T(\mathbf{y}-\mathbf{x}(\beta))=0$. This gives a function in $\beta$, where $D\_\mathbf{x}( \beta )$ is a matrix of dim $N\times K$, with element $\frac{\partial x\_n}{\partial\beta\_k}(\beta)$.

So, applying the Taylor expansion(1st order)/mean-value theorem to each component of the LHS of the equality, we obtain

\begin{equation}
D\_\mathbf{x}(\beta\_0)^T \mathbf{u} +\big[H\_\mathbf{x}(\bar\beta)-D\_\mathbf{x}(\bar\beta)^T D\_\mathbf{x}(\bar\beta)\big](\hat\beta-\beta\_0)=0
\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \text{(a)}
\end{equation}

where $H\_\mathbf{x}(\bar\beta)$ represents a matrix $K\times K$, with each element being $H\_{ij}(\bar\beta\_i)[\mathbf{y}-\mathbf{x}(\bar\beta\_i)]$, where $H\_{ij}(\bar\beta\_i)=\begin{bmatrix}
\frac{\partial^2 x\_1}{\partial\beta\_j\partial\beta\_i}(\bar\beta\_i) & \dots & \frac{\partial^2 x\_n}{\partial\beta\_j\partial\beta\_i}(\bar\beta\_i)
\end{bmatrix}$.
Similarly, $D\_\mathbf{x}(\bar\beta)^T D\_\mathbf{x}(\bar\beta)$ is the Gramian of $D\_\mathbf{x}(\bar\beta)$, with $ij$-th element $\sum\_l^n\frac{\partial x\_l}{\partial\beta\_j}(\bar\beta\_i)\frac{\partial x\_l}{\partial\beta\_i}(\bar\beta\_i)$.

From (a), $n^{\frac{1}{2}}(\hat\beta-\beta\_0)=-\left(\frac{1}{n}H\_\mathbf{x}(\bar\beta)-\frac{1}{n}D\_\mathbf{x}(\bar\beta)^T D\_\mathbf{x}(\bar\beta)\right)^{-1}n^{-\frac{1}{2}}D\_\mathbf{x}(\beta\_0)^T \mathbf{u}$.

We know from the non-linear model that $n^{-\frac{1}{2}}D\_\mathbf{x}(\beta\_0)^T \mathbf{u}=n^{-\frac{1}{2}}\sum\_i^n u\_i D\_{\mathbf{x}i}(\beta\_0)^T\rightarrow^d N\left(\underbrace{E(u\_i D\_{\mathbf{x}i}(\beta\_0)^T)}\_{=\mathbf{0}},\underbrace{\lim 1/n \sum E(u\_i^2 D\_{\mathbf{x}i}(\beta\_0)D\_{\mathbf{x}i}(\beta\_0)^T)}\_{\sigma\_0^2 S\_{D\_0^TD\_0}}\right)$

by a CLT.

Since we know that $\bar \beta\rightarrow^p \beta\_0$, we have $\frac{1}{n}H\_\mathbf{x}(\bar\beta)\rightarrow^p \left[\frac{1}{n}\sum \underbrace{E\left( \underbrace{(y\_i-x\_i\beta\_0)}\_{=u\_i}\frac{\partial^2 x\_1}{\partial\beta\_j\partial\beta\_i}(\beta\_0)\right)}\_{=0}\right]\_{K\times K}=\mathbf{0}$.

Similarly, we have that $\frac{1}{n}D\_\mathbf{x}(\bar\beta)^T D\_\mathbf{x}(\bar\beta)\rightarrow^p S\_{D\_0^TD\_0}$.

Thus, we can conclude that $n^{\frac{1}{2}}(\hat\beta-\beta\_0)\rightarrow^p N(\mathbf{0},\sigma\_0^2 S\_{D\_0^TD\_0}^{-1})$.

> **Comments:**
> **username_1**: Looks ok.The Asymptotic variance depends on the error being homoskedastic and non-autocorrelated of course.  
> **username_1**: Opportunity taken, please consider also upvoting answers you find helpful (apart from the green mark). I think various statistics related to the site take into account only votes and not green marks.  
> **username_0**: @AlecosPapadopoulos Done! and Thanks for the review. ;)  

---

