<p>The variance of <span class="math-container">$\hat{\beta}_0$</span> is given by:
<span class="math-container">$$
\operatorname{Var}\left(\hat{\beta}_0\right)=\sigma^2\left(\frac{1}{n}+\frac{\bar{x}^2}{s_{x x}}\right)
$$</span>
where <span class="math-container">$s_{x x}=\sum_{i=1}^n\left(x_i-\bar{x}\right)^2$</span>.
The estimator <span class="math-container">$\hat{\beta}_0$</span> is normally distributed because the errors <span class="math-container">$\epsilon_i$</span> are normally distributed. Hence:
<span class="math-container">$$
\hat{\beta}_0 \sim N\left(\beta_0, \sigma^2\left(\frac{1}{n}+\frac{\bar{x}^2}{s_{x x}}\right)\right) .
$$</span></p>
<p>To standardize <span class="math-container">$\hat{\beta}_0$</span>, we consider:
<span class="math-container">$$
\frac{\hat{\beta}_0-\beta_0}{\sqrt{\operatorname{Var}\left(\hat{\beta}_0\right)}}=\frac{\hat{\beta}_0-\beta_0}{\sqrt{\sigma^2\left(\frac{1}{n}+\frac{\bar{x}^2}{s x x}\right)}} \sim N(0,1) \text {. }
$$</span></p>
<p>Since we do not know <span class="math-container">$\sigma^2$</span>, we replace it with the unbiased estimator <span class="math-container">$s^2$</span>, where: <span class="math-container">$s^2=\frac{1}{n-2}\left(s_{y y}-\frac{s_{x y}^2}{s_{x x}}\right)$</span>
and <span class="math-container">$s_{y y}=\sum_{i=1}^n\left(Y_i-\bar{Y}\right)^2, s_{x y}=\sum_{i=1}^n\left(x_i-\bar{x}\right)\left(Y_i-\bar{Y}\right)$</span>.
Thus, the standardized form becomes:
<span class="math-container">$$
\frac{\hat{\beta}_0-\beta_0}{\sqrt{s^2\left(\frac{1}{n}+\frac{\bar{x}^2}{s_{x x}}\right)}} .
$$</span></p>
<p>Under the assumption of normally distributed errors, this follows a <span class="math-container">$t$</span>-distribution with <span class="math-container">$n-2$</span> degrees of freedom:
<span class="math-container">$$
\frac{\hat{\beta}_0-\beta_0}{\sqrt{s^2\left(\frac{1}{n}+\frac{\bar{x}^2}{s_{x x}}\right)}} \sim t_{n-2}
$$</span></p>
<p>A <span class="math-container">$(1-\alpha) 100 \%$</span> confidence interval for <span class="math-container">$\beta_0$</span> is derived from the <span class="math-container">$t$</span>-distribution:
<span class="math-container">$$
\hat{\beta}_0 \pm t_{\alpha / 2, n-2} \cdot \sqrt{s^2\left(\frac{1}{n}+\frac{\bar{x}^2}{s_{x x}}\right)}
$$</span>
where <span class="math-container">$t_{\alpha / 2, n-2}$</span> is the critical value of the <span class="math-container">$t$</span>-distribution with <span class="math-container">$n-2$</span> degrees of freedom at the <span class="math-container">$\alpha / 2$</span> significance level.</p>
<p>Thus, the confidence interval for <span class="math-container">$\beta_0$</span> is:
<span class="math-container">$$
\left[\hat{\beta}_0-t_{\alpha / 2, n-2} \cdot \sqrt{s^2\left(\frac{1}{n}+\frac{\bar{x}^2}{s_{x x}}\right)}, \hat{\beta}_0+t_{\alpha / 2, n-2} \cdot \sqrt{s^2\left(\frac{1}{n}+\frac{\bar{x}^2}{s_{x x}}\right)}\right] .
$$</span></p>
<p>This provides the desired <span class="math-container">$(1-\alpha) 100 \%$</span> confidence interval for <span class="math-container">$\beta_0$</span>.</p>

CC BY-SA 4.0
2024-05-24T12:24:57.300Z
2024-05-24T12:24:57.300Z
bruno
<p>The distribution will be <span class="math-container">$t_{n-2}$</span>.</p>
<p>Below, I assume the sample <span class="math-container">$\{x_i\}_{i=1}^n$</span> is treated as fixed. That is, I do all computations as if we were conditioning on <span class="math-container">$\{x_i\}_{i=1}^n$</span>. I relax this assumption at the end.</p>
<p>I think the easiest way to get the distribution of the regressors is by first transitioning the setup into matrix notation. This avoids (most of) the need for partitioned regression. That is, we write <span class="math-container">$$Y=X\beta+\varepsilon$$</span> where <span class="math-container">$$Y=\begin{bmatrix} y_1 \\
y_2 \\
\vdots \\
y_n \end{bmatrix}, \qquad 
X=\begin{bmatrix}
1 &amp; x_1 \\
1 &amp; x_2 \\ 
\vdots &amp; \vdots \\
1 &amp; x_n \end{bmatrix}, \qquad 
\beta = \begin{bmatrix} \beta_1 &amp; \beta_1 &amp; \dots \ \  \beta_1 \\
\beta_2 &amp; \beta_2 &amp; \dots \ \  \beta_2
  \end{bmatrix}, \qquad 
\varepsilon=\begin{bmatrix} \epsilon_1 \\
\epsilon_2 \\
\vdots \\
\epsilon_n \end{bmatrix}$$</span>
and <span class="math-container">$\varepsilon \sim N(0,\sigma^2I_n)$</span> since <span class="math-container">$\{\epsilon_i\}_{i=1}^n$</span> is an i.i.d. sequence of normal random variables.</p>
<p>In this notation, our OLSE is <span class="math-container">$\hat{\beta}=(X'X)^{-1}X'Y$</span>. Plugging in <span class="math-container">$Y=X\beta +\varepsilon$</span> we find <span class="math-container">$$\hat{\beta}=(X'X)^{-1}X'(X\beta+\varepsilon)=\beta+(X'X)^{-1}X'\varepsilon$$</span></p>
<p>Therefore, <span class="math-container">$\hat{\beta}-\beta=(X'X)^{-1}X'\varepsilon$</span>. Hence, since the sample <span class="math-container">$\{x_i\}_{i=1}^n$</span> is treated as fixed and <span class="math-container">$\varepsilon \sim N(0,\sigma^2 I_n)$</span>, we have <span class="math-container">$$\begin{pmatrix} \hat{\beta}_0 \\
\hat{\beta}_1 \end{pmatrix}-\begin{pmatrix} \beta_0 \\
\beta_1 \end{pmatrix}=\hat{\beta}-\beta\sim N(0,\sigma^2 (X'X)^{-1})$$</span></p>
<p>Any linear combination of jointly normally distributed random variables is normal. Define <span class="math-container">$\lambda=\begin{bmatrix} 1 \\ 0 \end{bmatrix}$</span>, then <span class="math-container">$$\hat{\beta}_0-\beta_0 =\lambda'(\hat{\beta}-\beta)\sim  N(0,\sigma^2\lambda'(X'X)^{-1}\lambda)$$</span> Therefore, <span class="math-container">$$\frac{\hat{\beta}_0-\beta_0}{\sqrt{\sigma^2\lambda'(X'X)^{-1}\lambda}}\sim N(0,1)$$</span> Consider, <span class="math-container">$$\begin{align*} \lambda'(X'X)^{-1} \lambda &amp;= \lambda' \begin{bmatrix} n &amp; \sum_{i=1}^n x_i \\ \sum_{i=1}^n x_i &amp; \sum_{i=1}^n x_i^2\end{bmatrix}^{-1} \lambda \\
&amp;= \frac{1}{n\sum_{i=1}^n x_i^2 -\left(\sum_{i=1}^n x_i \right)^2} \lambda' \begin{bmatrix} \sum_{i=1}^n x_i^2 &amp; -\sum_{i=1}^n x_i \\
-\sum_{i=1}^n x_i &amp; n \end{bmatrix}\lambda \\
&amp;= \frac{\sum_{i=1}^n x_i^2}{n\sum_{i=1}^n x_i^2 -\left(\sum_{i=1}^n x_i \right)^2} \\
&amp;= \frac{1}{n}\left[\frac{\frac{1}{n}\sum_{i=1}^n x_i^2}{\frac{1}{n}\sum_{i=1}^n x_i^2 -\left(\frac{1}{n}\sum_{i=1}^n x_i \right)^2}\right] \\
&amp;= \frac{1}{n}\left[\frac{\frac{1}{n}\sum_{i=1}^n x_i^2}{\frac{1}{n}\sum_{i=1}^n x_i^2 -\bar{x}^2}\right]  \\
&amp;= \frac{1}{n}\left[\frac{\frac{1}{n}\sum_{i=1}^n x_i^2-\bar{x}^2+\bar{x}^2}{\frac{1}{n}\sum_{i=1}^n x_i^2 -\bar{x}^2}\right]  \\
&amp;= \frac{1}{n}\left[1+ \frac{\bar{x}^2}{\frac{1}{n}\sum_{i=1}^n x_i^2 -\bar{x}^2}\right] \\
&amp;=\frac{1}{n}\left[1+ \frac{\bar{x}^2}{\frac{1}{n}\sum_{i=1}^n (x_i -\bar{x})^2}\right] \\
&amp;= \frac{1}{n}+\frac{\bar{x}^2}{s_{xx}}
\end{align*}$$</span></p>
<p>Hence, <span class="math-container">$$\frac{\hat{\beta}_0-\beta}{\sqrt{\sigma^2\left(\frac{1}{n}+\frac{\bar{x}^2}{s_{xx}}\right)}}\sim N(0,1)$$</span></p>
<p>What we want to do now is remove <span class="math-container">$\sigma^2$</span> from our expression since it is unknown. For this, we can use the standard t-statistic approach.</p>
<p>Notice your expression for <span class="math-container">$s^2$</span> can be expressed in terms of the residuals <span class="math-container">$\{\hat{\varepsilon}_i\}_{i=1}^n$</span>. Using that <span class="math-container">$\hat{\beta}_1=\frac{s_{xy}}{s_{xx}}$</span> (by partitioned regression), and <span class="math-container">$\hat{\beta}_0=\bar{y}-\hat{\beta}_1\bar{x}$</span>, we have <span class="math-container">$$\begin{align*} 
\sum_{i=1}^n \hat{\varepsilon}^2_i &amp;= \sum_{i=1}^n (y_i-\hat{y}_i)^2 \\
&amp;=\sum_{i=1}^n(y_i-(\hat{\beta}_0+\hat{\beta}_1x_i))^2 \\
&amp;=\sum_{i=1}^n(y_i-(\bar{y}-\frac{s_{xy}}{s_{xx}}\bar{x}+\frac{s_{xy}}{s_{xx}}x_i))^2 \\ 
&amp;= \sum_{i=1}^n ((y_i-\bar{y})-\frac{s_{xy}}{s_{xx}}(x_i-\bar{x}))^2 \\
&amp;= s_{yy}-2\frac{s_{xy}^2}{s_{xx}}+\frac{s_{xy}^2}{s_{xx}} \\
&amp;=s_{yy}-\frac{s_{xy}^2}{s_{xx}}\end{align*}$$</span>
This is advantageous as residuals are the projection of the errors (whose distribution we do know) onto the space orthogonal to the regressors. That is, we can use the annihilator matrix <span class="math-container">$M=I_n-X'(X'X)^{-1}X$</span>, which is the projection matrix onto the space <span class="math-container">$\text{col}(X)^{\perp}$</span>, to relate the residuals to the errors. This matrix is useful for us as <span class="math-container">$M\varepsilon=\hat{\varepsilon}$</span>. We can calculate, <span class="math-container">$$\text{tr}(M)=n-\text{tr}(X'(X'X)^{-1}X)=n-\text{tr}((X'X)^{-1}(X'X))=n-2$$</span> Therefore, since <span class="math-container">$\frac{\varepsilon}{\sigma}\sim N(0,I_n)$</span>, by Cochran's theorem <span class="math-container">$$\frac{1}{\sigma^2}\sum_{i=1}^n \hat{\epsilon}_i=\frac{1}{\sigma^2}\hat{\varepsilon}'\hat{\varepsilon}=\frac{\varepsilon'}{\sigma} M \frac{\varepsilon}{\sigma}\sim \chi_{n-2} $$</span> So, <span class="math-container">$$(n-2) \frac{s^2}{\sigma^2}\sim \chi_{n-2}$$</span> Additionally, <span class="math-container">$\hat{\beta}_0$</span> and <span class="math-container">$\varepsilon$</span> are independent (see <a href="https://stats.stackexchange.com/questions/173396/estimators-independence-in-simple-linear-regression">this question</a>). Therefore, the two statistics <span class="math-container">$$\begin{align*} T_1 &amp;=\frac{\hat{\beta}_0-\beta}{\sqrt{\sigma^2\left(\frac{1}{n}+\frac{\bar{x}^2}{s_{xx}}\right)}} \sim N(0,1) \\
T_2 &amp;= (n-2) \frac{s^2}{\sigma^2}\sim \chi_{n-2}\end{align*}$$</span> are independent. Therefore, <span class="math-container">$\textit{by definition}$</span> we have <span class="math-container">$$\frac{T_1}{\sqrt{T_2/(n-2)}}\sim t_{n-2}$$</span> where <span class="math-container">$t_{n-2}$</span> is the t-distribution with <span class="math-container">$n-2$</span> degrees of freedom. Cancelling terms gives <span class="math-container">$$\frac{\hat{\beta}_0-\beta_0}{\sqrt{s^2\left(\frac{1}{n}+\frac{\bar{x}^2}{s_{xx}}\right)}}\sim t_{n-2}$$</span>
this is not normal as we have to account for the estimation error of <span class="math-container">$\sigma^2$</span> by <span class="math-container">$s^2$</span>.</p>
<p>We can conclude something more. Everything I have done has been implicitly conditioning on the sample and treating it as if it were the population. This could be a problematic assumption. However, since the distribution, <span class="math-container">$t_{n-2}$</span>, does not depend on the give sample data, we can remove the conditioning on <span class="math-container">$\{x_i\}_{i=1}^n$</span>. Therefore, even for stochastic regressors we have <span class="math-container">$$\frac{\hat{\beta}_0-\beta_0}{\sqrt{s^2\left(\frac{1}{n}+\frac{\bar{x}^2}{s_{xx}}\right)}}\sim t_{n-2}$$</span> so we do not need to worry about whether we conditioning on the sample or not for the result.</p>
<p>To derive a two-sided confidence interval of size <span class="math-container">$1-\alpha$</span>, let the  <span class="math-container">$\frac{\alpha}{2}$</span> and <span class="math-container">$1-\frac{\alpha}{2}$</span> quantiles of the <span class="math-container">$t_{n-2}$</span> distribution be <span class="math-container">$t_{\frac{\alpha}{2},n-2}$</span> and <span class="math-container">$t_{1-\frac{\alpha}{2},n-2}$</span>, respectively. Then, defining <span class="math-container">$\hat{se}(\hat{\beta}_0)=\sqrt{s^2\left(\frac{1}{n}+\frac{\bar{x}^2}{s_{xx}}\right)}$</span>, we have  <span class="math-container">$$1-\alpha = \mathbb{P}\left(t_{\frac{\alpha}{2}}\leq \frac{\hat{\beta}_0-\beta_0}{\hat{se}(\hat{\beta}_0)} \leq t_{1-\frac{\alpha}{2},n-2} \right)$$</span> Rearranging, we obtain <span class="math-container">$$1-\alpha =\mathbb{P}\left(\hat{\beta}_0-\hat{se}(\hat{\beta}_0)t_{\frac{\alpha}{2},n-2}\geq \beta_0 \geq \hat{\beta}_0-\hat{se}(\hat{\beta}_0)t_{1-\frac{\alpha}{2},n-2}  \right)$$</span> Therefore, <span class="math-container">$$CI(1-\alpha)=\left[\hat{\beta}_0-\hat{se}(\hat{\beta}_0)t_{1-\frac{\alpha}{2},n-2},\hat{\beta}_0-\hat{se}(\hat{\beta}_0)t_{\frac{\alpha}{2},n-2}\right]$$</span></p>

CC BY-SA 4.0
2024-05-19T01:39:22.853Z
2024-05-27T06:10:59.263Z
2024-05-27T06:10:59.263Z
Joseph Basford
<p>I have the following linear regression model:
<span class="math-container">$$
Y_i = \beta_0 + \beta_1 x_i + \epsilon_i
$$</span>
where <span class="math-container">$\epsilon_i$</span> are independent <span class="math-container">$N(0, \sigma^2)$</span> random variables. Let <span class="math-container">$\hat{\beta}_i$</span> denote the ordinary least squares estimators of the parameters <span class="math-container">$\beta_0$</span> and <span class="math-container">$\beta_1$</span> for <span class="math-container">$i=0,1$</span>. I want to determine the distribution of
<span class="math-container">$$
\frac{\hat{\beta}_0 - \beta_0}{\sqrt{s^2 \left( \frac{1}{n} + \frac{\bar{x}^2}{s_{xx}} \right)}}
$$</span>
where
<span class="math-container">$$
s^2 = \frac{1}{n-2} \left( s_{yy} - \frac{s_{xy}^2}{s_{xx}} \right)
$$</span>
and <span class="math-container">$s_{xx}$</span>, <span class="math-container">$s_{yy}$</span>, and <span class="math-container">$s_{xy}$</span> are the sums of squares and cross-products.</p>
<p>Additionally, I want to derive a <span class="math-container">$(1-\alpha)100\%$</span> confidence interval for <span class="math-container">$\beta_0$</span>. Could someone help me with the steps and derivation?</p>

<econometrics><regression>
Finding the distribution in OLS
bruno
# Finding the distribution in OLS

(Asked by: username_0 on 2024-05-18)

I have the following linear regression model:
$$
Y\_i = \beta\_0 + \beta\_1 x\_i + \epsilon\_i
$$
where $\epsilon\_i$ are independent $N(0, \sigma^2)$ random variables. Let $\hat{\beta}\_i$ denote the ordinary least squares estimators of the parameters $\beta\_0$ and $\beta\_1$ for $i=0,1$. I want to determine the distribution of
$$
\frac{\hat{\beta}\_0 - \beta\_0}{\sqrt{s^2 \left( \frac{1}{n} + \frac{\bar{x}^2}{s\_{xx}} \right)}}
$$
where
$$
s^2 = \frac{1}{n-2} \left( s\_{yy} - \frac{s\_{xy}^2}{s\_{xx}} \right)
$$
and $s\_{xx}$, $s\_{yy}$, and $s\_{xy}$ are the sums of squares and cross-products.

Additionally, I want to derive a $(1-\alpha)100\%$ confidence interval for $\beta\_0$. Could someone help me with the steps and derivation?

**Tags:** econometrics, regression

---

## 2 answers

### Answer by username_1 on 2024-05-19. Score: 6 | Accepted answer

The distribution will be $t\_{n-2}$.

Below, I assume the sample $\{x\_i\}\_{i=1}^n$ is treated as fixed. That is, I do all computations as if we were conditioning on $\{x\_i\}\_{i=1}^n$. I relax this assumption at the end.

I think the easiest way to get the distribution of the regressors is by first transitioning the setup into matrix notation. This avoids (most of) the need for partitioned regression. That is, we write $$Y=X\beta+\varepsilon$$ where $$Y=\begin{bmatrix} y\_1 \\
y\_2 \\
\vdots \\
y\_n \end{bmatrix}, \qquad
X=\begin{bmatrix}
1 & x\_1 \\
1 & x\_2 \\
\vdots & \vdots \\
1 & x\_n \end{bmatrix}, \qquad
\beta = \begin{bmatrix} \beta\_1 & \beta\_1 & \dots \ \ \beta\_1 \\
\beta\_2 & \beta\_2 & \dots \ \ \beta\_2
\end{bmatrix}, \qquad
\varepsilon=\begin{bmatrix} \epsilon\_1 \\
\epsilon\_2 \\
\vdots \\
\epsilon\_n \end{bmatrix}$$
and $\varepsilon \sim N(0,\sigma^2I\_n)$ since $\{\epsilon\_i\}\_{i=1}^n$ is an i.i.d. sequence of normal random variables.

In this notation, our OLSE is $\hat{\beta}=(X'X)^{-1}X'Y$. Plugging in $Y=X\beta +\varepsilon$ we find $$\hat{\beta}=(X'X)^{-1}X'(X\beta+\varepsilon)=\beta+(X'X)^{-1}X'\varepsilon$$

Therefore, $\hat{\beta}-\beta=(X'X)^{-1}X'\varepsilon$. Hence, since the sample $\{x\_i\}\_{i=1}^n$ is treated as fixed and $\varepsilon \sim N(0,\sigma^2 I\_n)$, we have $$\begin{pmatrix} \hat{\beta}\_0 \\
\hat{\beta}\_1 \end{pmatrix}-\begin{pmatrix} \beta\_0 \\
\beta\_1 \end{pmatrix}=\hat{\beta}-\beta\sim N(0,\sigma^2 (X'X)^{-1})$$

Any linear combination of jointly normally distributed random variables is normal. Define $\lambda=\begin{bmatrix} 1 \\ 0 \end{bmatrix}$, then $$\hat{\beta}\_0-\beta\_0 =\lambda'(\hat{\beta}-\beta)\sim N(0,\sigma^2\lambda'(X'X)^{-1}\lambda)$$ Therefore, $$\frac{\hat{\beta}\_0-\beta\_0}{\sqrt{\sigma^2\lambda'(X'X)^{-1}\lambda}}\sim N(0,1)$$ Consider, $$\begin{align\*} \lambda'(X'X)^{-1} \lambda &= \lambda' \begin{bmatrix} n & \sum\_{i=1}^n x\_i \\ \sum\_{i=1}^n x\_i & \sum\_{i=1}^n x\_i^2\end{bmatrix}^{-1} \lambda \\
&= \frac{1}{n\sum\_{i=1}^n x\_i^2 -\left(\sum\_{i=1}^n x\_i \right)^2} \lambda' \begin{bmatrix} \sum\_{i=1}^n x\_i^2 & -\sum\_{i=1}^n x\_i \\
-\sum\_{i=1}^n x\_i & n \end{bmatrix}\lambda \\
&= \frac{\sum\_{i=1}^n x\_i^2}{n\sum\_{i=1}^n x\_i^2 -\left(\sum\_{i=1}^n x\_i \right)^2} \\
&= \frac{1}{n}\left[\frac{\frac{1}{n}\sum\_{i=1}^n x\_i^2}{\frac{1}{n}\sum\_{i=1}^n x\_i^2 -\left(\frac{1}{n}\sum\_{i=1}^n x\_i \right)^2}\right] \\
&= \frac{1}{n}\left[\frac{\frac{1}{n}\sum\_{i=1}^n x\_i^2}{\frac{1}{n}\sum\_{i=1}^n x\_i^2 -\bar{x}^2}\right] \\
&= \frac{1}{n}\left[\frac{\frac{1}{n}\sum\_{i=1}^n x\_i^2-\bar{x}^2+\bar{x}^2}{\frac{1}{n}\sum\_{i=1}^n x\_i^2 -\bar{x}^2}\right] \\
&= \frac{1}{n}\left[1+ \frac{\bar{x}^2}{\frac{1}{n}\sum\_{i=1}^n x\_i^2 -\bar{x}^2}\right] \\
&=\frac{1}{n}\left[1+ \frac{\bar{x}^2}{\frac{1}{n}\sum\_{i=1}^n (x\_i -\bar{x})^2}\right] \\
&= \frac{1}{n}+\frac{\bar{x}^2}{s\_{xx}}
\end{align\*}$$

Hence, $$\frac{\hat{\beta}\_0-\beta}{\sqrt{\sigma^2\left(\frac{1}{n}+\frac{\bar{x}^2}{s\_{xx}}\right)}}\sim N(0,1)$$

What we want to do now is remove $\sigma^2$ from our expression since it is unknown. For this, we can use the standard t-statistic approach.

Notice your expression for $s^2$ can be expressed in terms of the residuals $\{\hat{\varepsilon}\_i\}\_{i=1}^n$. Using that $\hat{\beta}\_1=\frac{s\_{xy}}{s\_{xx}}$ (by partitioned regression), and $\hat{\beta}\_0=\bar{y}-\hat{\beta}\_1\bar{x}$, we have $$\begin{align\*}
\sum\_{i=1}^n \hat{\varepsilon}^2\_i &= \sum\_{i=1}^n (y\_i-\hat{y}\_i)^2 \\
&=\sum\_{i=1}^n(y\_i-(\hat{\beta}\_0+\hat{\beta}\_1x\_i))^2 \\
&=\sum\_{i=1}^n(y\_i-(\bar{y}-\frac{s\_{xy}}{s\_{xx}}\bar{x}+\frac{s\_{xy}}{s\_{xx}}x\_i))^2 \\
&= \sum\_{i=1}^n ((y\_i-\bar{y})-\frac{s\_{xy}}{s\_{xx}}(x\_i-\bar{x}))^2 \\
&= s\_{yy}-2\frac{s\_{xy}^2}{s\_{xx}}+\frac{s\_{xy}^2}{s\_{xx}} \\
&=s\_{yy}-\frac{s\_{xy}^2}{s\_{xx}}\end{align\*}$$
This is advantageous as residuals are the projection of the errors (whose distribution we do know) onto the space orthogonal to the regressors. That is, we can use the annihilator matrix $M=I\_n-X'(X'X)^{-1}X$, which is the projection matrix onto the space $\text{col}(X)^{\perp}$, to relate the residuals to the errors. This matrix is useful for us as $M\varepsilon=\hat{\varepsilon}$. We can calculate, $$\text{tr}(M)=n-\text{tr}(X'(X'X)^{-1}X)=n-\text{tr}((X'X)^{-1}(X'X))=n-2$$ Therefore, since $\frac{\varepsilon}{\sigma}\sim N(0,I\_n)$, by Cochran's theorem $$\frac{1}{\sigma^2}\sum\_{i=1}^n \hat{\epsilon}\_i=\frac{1}{\sigma^2}\hat{\varepsilon}'\hat{\varepsilon}=\frac{\varepsilon'}{\sigma} M \frac{\varepsilon}{\sigma}\sim \chi\_{n-2} $$ So, $$(n-2) \frac{s^2}{\sigma^2}\sim \chi\_{n-2}$$ Additionally, $\hat{\beta}\_0$ and $\varepsilon$ are independent (see [this question](https://stats.stackexchange.com/questions/173396/estimators-independence-in-simple-linear-regression)). Therefore, the two statistics $$\begin{align\*} T\_1 &=\frac{\hat{\beta}\_0-\beta}{\sqrt{\sigma^2\left(\frac{1}{n}+\frac{\bar{x}^2}{s\_{xx}}\right)}} \sim N(0,1) \\
T\_2 &= (n-2) \frac{s^2}{\sigma^2}\sim \chi\_{n-2}\end{align\*}$$ are independent. Therefore, $\textit{by definition}$ we have $$\frac{T\_1}{\sqrt{T\_2/(n-2)}}\sim t\_{n-2}$$ where $t\_{n-2}$ is the t-distribution with $n-2$ degrees of freedom. Cancelling terms gives $$\frac{\hat{\beta}\_0-\beta\_0}{\sqrt{s^2\left(\frac{1}{n}+\frac{\bar{x}^2}{s\_{xx}}\right)}}\sim t\_{n-2}$$
this is not normal as we have to account for the estimation error of $\sigma^2$ by $s^2$.

We can conclude something more. Everything I have done has been implicitly conditioning on the sample and treating it as if it were the population. This could be a problematic assumption. However, since the distribution, $t\_{n-2}$, does not depend on the give sample data, we can remove the conditioning on $\{x\_i\}\_{i=1}^n$. Therefore, even for stochastic regressors we have $$\frac{\hat{\beta}\_0-\beta\_0}{\sqrt{s^2\left(\frac{1}{n}+\frac{\bar{x}^2}{s\_{xx}}\right)}}\sim t\_{n-2}$$ so we do not need to worry about whether we conditioning on the sample or not for the result.

To derive a two-sided confidence interval of size $1-\alpha$, let the $\frac{\alpha}{2}$ and $1-\frac{\alpha}{2}$ quantiles of the $t\_{n-2}$ distribution be $t\_{\frac{\alpha}{2},n-2}$ and $t\_{1-\frac{\alpha}{2},n-2}$, respectively. Then, defining $\hat{se}(\hat{\beta}\_0)=\sqrt{s^2\left(\frac{1}{n}+\frac{\bar{x}^2}{s\_{xx}}\right)}$, we have $$1-\alpha = \mathbb{P}\left(t\_{\frac{\alpha}{2}}\leq \frac{\hat{\beta}\_0-\beta\_0}{\hat{se}(\hat{\beta}\_0)} \leq t\_{1-\frac{\alpha}{2},n-2} \right)$$ Rearranging, we obtain $$1-\alpha =\mathbb{P}\left(\hat{\beta}\_0-\hat{se}(\hat{\beta}\_0)t\_{\frac{\alpha}{2},n-2}\geq \beta\_0 \geq \hat{\beta}\_0-\hat{se}(\hat{\beta}\_0)t\_{1-\frac{\alpha}{2},n-2} \right)$$ Therefore, $$CI(1-\alpha)=\left[\hat{\beta}\_0-\hat{se}(\hat{\beta}\_0)t\_{1-\frac{\alpha}{2},n-2},\hat{\beta}\_0-\hat{se}(\hat{\beta}\_0)t\_{\frac{\alpha}{2},n-2}\right]$$


---

### Answer by username_0 on 2024-05-24. Score: 0

The variance of $\hat{\beta}\_0$ is given by:
$$
\operatorname{Var}\left(\hat{\beta}\_0\right)=\sigma^2\left(\frac{1}{n}+\frac{\bar{x}^2}{s\_{x x}}\right)
$$
where $s\_{x x}=\sum\_{i=1}^n\left(x\_i-\bar{x}\right)^2$.
The estimator $\hat{\beta}\_0$ is normally distributed because the errors $\epsilon\_i$ are normally distributed. Hence:
$$
\hat{\beta}\_0 \sim N\left(\beta\_0, \sigma^2\left(\frac{1}{n}+\frac{\bar{x}^2}{s\_{x x}}\right)\right) .
$$

To standardize $\hat{\beta}\_0$, we consider:
$$
\frac{\hat{\beta}\_0-\beta\_0}{\sqrt{\operatorname{Var}\left(\hat{\beta}\_0\right)}}=\frac{\hat{\beta}\_0-\beta\_0}{\sqrt{\sigma^2\left(\frac{1}{n}+\frac{\bar{x}^2}{s x x}\right)}} \sim N(0,1) \text {. }
$$

Since we do not know $\sigma^2$, we replace it with the unbiased estimator $s^2$, where: $s^2=\frac{1}{n-2}\left(s\_{y y}-\frac{s\_{x y}^2}{s\_{x x}}\right)$
and $s\_{y y}=\sum\_{i=1}^n\left(Y\_i-\bar{Y}\right)^2, s\_{x y}=\sum\_{i=1}^n\left(x\_i-\bar{x}\right)\left(Y\_i-\bar{Y}\right)$.
Thus, the standardized form becomes:
$$
\frac{\hat{\beta}\_0-\beta\_0}{\sqrt{s^2\left(\frac{1}{n}+\frac{\bar{x}^2}{s\_{x x}}\right)}} .
$$

Under the assumption of normally distributed errors, this follows a $t$-distribution with $n-2$ degrees of freedom:
$$
\frac{\hat{\beta}\_0-\beta\_0}{\sqrt{s^2\left(\frac{1}{n}+\frac{\bar{x}^2}{s\_{x x}}\right)}} \sim t\_{n-2}
$$

A $(1-\alpha) 100 \%$ confidence interval for $\beta\_0$ is derived from the $t$-distribution:
$$
\hat{\beta}\_0 \pm t\_{\alpha / 2, n-2} \cdot \sqrt{s^2\left(\frac{1}{n}+\frac{\bar{x}^2}{s\_{x x}}\right)}
$$
where $t\_{\alpha / 2, n-2}$ is the critical value of the $t$-distribution with $n-2$ degrees of freedom at the $\alpha / 2$ significance level.

Thus, the confidence interval for $\beta\_0$ is:
$$
\left[\hat{\beta}\_0-t\_{\alpha / 2, n-2} \cdot \sqrt{s^2\left(\frac{1}{n}+\frac{\bar{x}^2}{s\_{x x}}\right)}, \hat{\beta}\_0+t\_{\alpha / 2, n-2} \cdot \sqrt{s^2\left(\frac{1}{n}+\frac{\bar{x}^2}{s\_{x x}}\right)}\right] .
$$

This provides the desired $(1-\alpha) 100 \%$ confidence interval for $\beta\_0$.


---

