<p>Suppose the relationship between <span class="math-container">$y_{i,t}\equiv\log w_{i,t}$</span> and <span class="math-container">$z_{i,t}\equiv\left(s_{i,t},j_{i,t},e_{i,t},x_{i,t}\right)^{\top}$</span> is given by</p>
<p><span class="math-container">$$y_{i,t}=g\left(z_{i,t}\right)+\varepsilon_{i,t},$$</span></p>
<p>where <span class="math-container">$g\left(\cdot\right)$</span> can be nonlinear. There can be two cases for this problem</p>
<ul>
<li><p><span class="math-container">$g\left(\cdot\right)$</span> is unknown. For example, if one use sieve estimation, which may lead to the high dimension problem you mentioned, to address this problem, he can apply some regulation methods like LASSO. However, this is not the case in this question, so I won't talk too much.</p>
</li>
<li><p><span class="math-container">$g\left(\cdot\right)=m\left(z_{i,t};\beta\right)$</span> where the form of <span class="math-container">$m\left(\cdot\right)$</span> is known, like in this question, and <span class="math-container">$\beta=\left(D,\Pi,\lambda,\theta,\alpha\right)^{\top}$</span>. This happens when there exists some economic theory or model that yields this kind of relationship, or the user needs to identify and analyze how the covariates affects the outcome. In this case, the NLS estimator is easier to analyze since it can be seen a special case of extreme estimator:</p>
</li>
</ul>
<p><span class="math-container">$$\hat{\beta}=\mathop{\arg\min}_{\beta\in\mathcal{B}}\hat{S}\left(\beta\right),$$</span>
where <span class="math-container">$\hat{S}\left(\beta\right)\equiv\frac{1}{NT}\sum_{i=1}^{N}\sum_{t=1}^{T}\left(y_{i,t}-m\left(z_{i,t};\beta\right)\right)^{2}$</span>. Following the similar discussion as in extreme estimators, if <span class="math-container">$\mathcal{B}$</span> is compact and <span class="math-container">$\hat{S}\left(\beta\right)$</span> converge uniformly in probability to some nonrandom function which has a unique minimizer, then <span class="math-container">$\hat{\beta}$</span> is consistent, and will be asymptotically normal under additional conditions.</p>
<p>To solve <span class="math-container">$\hat{\beta}$</span>, you can first write the FOC of this problem
<span class="math-container">$$\frac{1}{NT}\sum_{i=1}^{N}\sum_{t=1}^{T}\left(y_{i,t}-m\left(z_{i,t};\hat{\beta}\right)\right)\frac{\partial m\left(z_{i,t};\hat{\beta}\right)}{\partial \beta}=0.$$</span>
Generally there is no closed form solution for <span class="math-container">$\hat{\beta}$</span> hence it must be found by numerical methods. There are two iteration algorithms that can be used.</p>
<ul>
<li>Gauss-Newton method</li>
</ul>
<p><span class="math-container">$$\beta_{r+1}=\beta_{r}-\frac{1}{2}\left(\sum_{i=1}^{N}\sum_{t=1}^{T}\frac{\partial m\left(z_{i,t};\beta_{r}\right)}{\partial\beta}\frac{\partial m\left(z_{i,t};\beta_{r}\right)}{\partial\beta^{\top}}\right)^{-1}\frac{\partial\hat{S}\left(\beta_{r}\right)}{\partial \beta}.$$</span></p>
<ul>
<li>Newton-Raphson Algorithm
<span class="math-container">$$\beta_{r+1}=\beta_{r}-\left(\frac{\partial^{2}\hat{S}\left(\beta_{r}\right)}{\partial\beta\partial\beta^{\top}}\right)^{-1}\frac{\partial \hat{S}\left(\beta_{r}\right)}{\partial\beta},$$</span>
which requires the matrix in bracket being positive definite to converge.</li>
</ul>
<p>As for programming, I'm not sure if any statistics software has <code>NLS</code> function. You can google it and check the help document if have one. If there's no existing package, you can use <code>optimize</code> to solve the origin minimization problem, or <code>fsolve</code> to find numerical solution to above FOC.</p>
<hr />
<p>Update for std. Suppose <span class="math-container">$\sqrt{NT}\left(\hat{\beta}-\beta_{0}\right)\overset{d}{\to}\mathcal{N}\left(0,V\right)$</span>, then <span class="math-container">$V$</span> can be estimated by
<span class="math-container">$$\begin{align*}\hat{V}&amp;\equiv\left(\frac{1}{NT}\sum_{i=1}^{N}\sum_{t=1}^{T}\frac{\partial m\left(z_{i,t};\hat{\beta}\right)}{\partial\beta}\frac{\partial m\left(z_{i,t};\hat{\beta}\right)}{\partial\beta^{\top}}\right)^{-1}\\
&amp;\left(\frac{1}{NT}\sum_{i=1}^{N}\sum_{t=1}^{T}\frac{\partial m\left(z_{i,t};\hat{\beta}\right)}{\partial\beta}\frac{\partial m\left(z_{i,t};\hat{\beta}\right)}{\partial\beta^{\top}}\hat{\varepsilon}_{i,t}^{2}\right)\left(\frac{1}{NT}\sum_{i=1}^{N}\sum_{t=1}^{T}\frac{\partial m\left(z_{i,t};\hat{\beta}\right)}{\partial\beta}\frac{\partial m\left(z_{i,t};\hat{\beta}\right)}{\partial\beta^{\top}}\right)^{-1}.\end{align*}$$</span></p>

CC BY-SA 4.0
2021-08-10T17:48:59.420Z
Thanks for the  nice derivations. R offers many different packages for doing maximization of non-linear functions. You can look at the optimization task view  on cran for details.
mark leeds
CC BY-SA 4.0
2021-08-10T11:26:52.110Z
2021-08-10T11:35:28.800Z
2021-08-10T11:35:28.800Z
Q9y5
<p>I am reading <a href="https://www.aeaweb.org/articles?id=10.1257/aer.20080841" rel="nofollow noreferrer">Jeong, Kim, Manovskii 2015</a> and in the paper they apply &quot;a nonlinear least-squares method&quot; to estimate a log-wage equation, <a href="https://i.sstatic.net/R6QYG.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/R6QYG.png" alt="enter image description here" /></a>, where <span class="math-container">$D, \Pi$</span>, all <span class="math-container">$\lambda$</span>s <span class="math-container">$\theta$</span>s, and <span class="math-container">$\alpha$</span> are coefficients, <span class="math-container">$j$</span>, <span class="math-container">$e$</span> are age and experience, and <span class="math-container">$s$</span>, <span class="math-container">$x$</span> means sex and education group.</p>
<p>I understand the &quot;nonlinear least-squares method&quot; by simply doing <span class="math-container">$$\min_{D, \Pi, \lambda, \theta, \alpha} \sum_{i,t}\hat{\epsilon}_{i,t}^{2}$$</span>. But I have no idea what are the common approaches (both in terms of theory and in terms of statistic packages or programming) to estimate and are there any important issues or cautions in such estimations (e.g. high dimension, or in the case here the age and the experience is likely to be positively correlated)?</p>
<p>An additional question specific to the case here is that do we need panel data like PSID data for the estimation here? Put it in a different way, can we use repeated cross-sectional data like CPS data to estimate this log wage equation?</p>

CC BY-SA 4.0
2021-08-10T10:55:58.300Z
Hi: This not a specific comment to your particular problem but, in general, if you're using optimization software that allows for gradients, then it's best to use them when you call the function. The algorithm will have an easier time finding a solution and won't have to rely on numerical approximations for the gradients.
mark leeds
CC BY-SA 4.0
2021-08-10T11:00:19.267Z
@markleeds Thanks for comment. If in general the method is to use optimization packages, how can I get the standard deviations? Bootstrapping?
Alalalalaki
CC BY-SA 4.0
2021-08-10T17:46:48.917Z
you can use the resulting hessian as an asymptotic approximation. For reasonable sample size, that should be okay.
mark leeds
<labor-economics><regression><wages><reduced-form-estimation>
Nonlinear Least-Squares Estimation in Practice
Alalalalaki
# Nonlinear Least-Squares Estimation in Practice

(Asked by: username_0 on 2021-08-09)

I am reading [Jeong, Kim, Manovskii 2015](https://www.aeaweb.org/articles?id=10.1257/aer.20080841) and in the paper they apply "a nonlinear least-squares method" to estimate a log-wage equation, [![enter image description here](https://i.sstatic.net/R6QYG.png)](https://i.sstatic.net/R6QYG.png), where $D, \Pi$, all $\lambda$s $\theta$s, and $\alpha$ are coefficients, $j$, $e$ are age and experience, and $s$, $x$ means sex and education group.

I understand the "nonlinear least-squares method" by simply doing $$\min\_{D, \Pi, \lambda, \theta, \alpha} \sum\_{i,t}\hat{\epsilon}\_{i,t}^{2}$$. But I have no idea what are the common approaches (both in terms of theory and in terms of statistic packages or programming) to estimate and are there any important issues or cautions in such estimations (e.g. high dimension, or in the case here the age and the experience is likely to be positively correlated)?

An additional question specific to the case here is that do we need panel data like PSID data for the estimation here? Put it in a different way, can we use repeated cross-sectional data like CPS data to estimate this log wage equation?

**Tags:** labor-economics, regression, wages, reduced-form-estimation
> **Comments:**
> **username_1**: Hi: This not a specific comment to your particular problem but, in general, if you're using optimization software that allows for gradients, then it's best to use them when you call the function. The algorithm will have an easier time finding a solution and won't have to rely on numerical approximations for the gradients.  
> **username_0**: @markleeds Thanks for comment. If in general the method is to use optimization packages, how can I get the standard deviations? Bootstrapping?  
> **username_1**: you can use the resulting hessian as an asymptotic approximation. For reasonable sample size, that should be okay.  

---

## 1 answers

### Answer by username_2 on 2021-08-10. Score: 5

Suppose the relationship between $y\_{i,t}\equiv\log w\_{i,t}$ and $z\_{i,t}\equiv\left(s\_{i,t},j\_{i,t},e\_{i,t},x\_{i,t}\right)^{\top}$ is given by

$$y\_{i,t}=g\left(z\_{i,t}\right)+\varepsilon\_{i,t},$$

where $g\left(\cdot\right)$ can be nonlinear. There can be two cases for this problem

* $g\left(\cdot\right)$ is unknown. For example, if one use sieve estimation, which may lead to the high dimension problem you mentioned, to address this problem, he can apply some regulation methods like LASSO. However, this is not the case in this question, so I won't talk too much.
* $g\left(\cdot\right)=m\left(z\_{i,t};\beta\right)$ where the form of $m\left(\cdot\right)$ is known, like in this question, and $\beta=\left(D,\Pi,\lambda,\theta,\alpha\right)^{\top}$. This happens when there exists some economic theory or model that yields this kind of relationship, or the user needs to identify and analyze how the covariates affects the outcome. In this case, the NLS estimator is easier to analyze since it can be seen a special case of extreme estimator:

$$\hat{\beta}=\mathop{\arg\min}\_{\beta\in\mathcal{B}}\hat{S}\left(\beta\right),$$
where $\hat{S}\left(\beta\right)\equiv\frac{1}{NT}\sum\_{i=1}^{N}\sum\_{t=1}^{T}\left(y\_{i,t}-m\left(z\_{i,t};\beta\right)\right)^{2}$. Following the similar discussion as in extreme estimators, if $\mathcal{B}$ is compact and $\hat{S}\left(\beta\right)$ converge uniformly in probability to some nonrandom function which has a unique minimizer, then $\hat{\beta}$ is consistent, and will be asymptotically normal under additional conditions.

To solve $\hat{\beta}$, you can first write the FOC of this problem
$$\frac{1}{NT}\sum\_{i=1}^{N}\sum\_{t=1}^{T}\left(y\_{i,t}-m\left(z\_{i,t};\hat{\beta}\right)\right)\frac{\partial m\left(z\_{i,t};\hat{\beta}\right)}{\partial \beta}=0.$$
Generally there is no closed form solution for $\hat{\beta}$ hence it must be found by numerical methods. There are two iteration algorithms that can be used.

* Gauss-Newton method

$$\beta\_{r+1}=\beta\_{r}-\frac{1}{2}\left(\sum\_{i=1}^{N}\sum\_{t=1}^{T}\frac{\partial m\left(z\_{i,t};\beta\_{r}\right)}{\partial\beta}\frac{\partial m\left(z\_{i,t};\beta\_{r}\right)}{\partial\beta^{\top}}\right)^{-1}\frac{\partial\hat{S}\left(\beta\_{r}\right)}{\partial \beta}.$$

* Newton-Raphson Algorithm
  $$\beta\_{r+1}=\beta\_{r}-\left(\frac{\partial^{2}\hat{S}\left(\beta\_{r}\right)}{\partial\beta\partial\beta^{\top}}\right)^{-1}\frac{\partial \hat{S}\left(\beta\_{r}\right)}{\partial\beta},$$
  which requires the matrix in bracket being positive definite to converge.

As for programming, I'm not sure if any statistics software has `NLS` function. You can google it and check the help document if have one. If there's no existing package, you can use `optimize` to solve the origin minimization problem, or `fsolve` to find numerical solution to above FOC.

---

Update for std. Suppose $\sqrt{NT}\left(\hat{\beta}-\beta\_{0}\right)\overset{d}{\to}\mathcal{N}\left(0,V\right)$, then $V$ can be estimated by
$$\begin{align\*}\hat{V}&\equiv\left(\frac{1}{NT}\sum\_{i=1}^{N}\sum\_{t=1}^{T}\frac{\partial m\left(z\_{i,t};\hat{\beta}\right)}{\partial\beta}\frac{\partial m\left(z\_{i,t};\hat{\beta}\right)}{\partial\beta^{\top}}\right)^{-1}\\
&\left(\frac{1}{NT}\sum\_{i=1}^{N}\sum\_{t=1}^{T}\frac{\partial m\left(z\_{i,t};\hat{\beta}\right)}{\partial\beta}\frac{\partial m\left(z\_{i,t};\hat{\beta}\right)}{\partial\beta^{\top}}\hat{\varepsilon}\_{i,t}^{2}\right)\left(\frac{1}{NT}\sum\_{i=1}^{N}\sum\_{t=1}^{T}\frac{\partial m\left(z\_{i,t};\hat{\beta}\right)}{\partial\beta}\frac{\partial m\left(z\_{i,t};\hat{\beta}\right)}{\partial\beta^{\top}}\right)^{-1}.\end{align\*}$$

> **Comments:**
> **username_1**: Thanks for the  nice derivations. R offers many different packages for doing maximization of non-linear functions. You can look at the optimization task view  on cran for details.  

---

