<p>I'm not a statistician... and this is the economics forum, so forgive me for the possible mistakes. I got most of this information from these <a href="https://myweb.uiowa.edu/pbreheny/uk/teaching/621/notes/9-6.pdf" rel="nofollow noreferrer">slides</a>.</p>
<p>Consider a statistic of interest  <span class="math-container">$\theta$</span> with a consistent but biased estimate <span class="math-container">$\hat \theta$</span>. The Jacknife estimates <span class="math-container">$\hat \theta_{(i)}$</span> with is the same statistic, based on the sample that is obtained from removing observation <span class="math-container">$i$</span>. Doing this for all <span class="math-container">$i$</span> and taking the mean over the values <span class="math-container">$\hat \theta_{(i)}$</span> gives:
<span class="math-container">$$
\hat \theta_{(.)} = \frac{1}{n} \sum_{i = 1}^n \hat \theta_{(i)}
$$</span>
As stated above, the estimator <span class="math-container">$\hat \theta$</span> is biased, so let:
<span class="math-container">$$
\mathbb{E}(\hat \theta) = \theta + \frac{b}{n} + O\left(\frac{1}{n^2}\right)
$$</span>
Where <span class="math-container">$b$</span> is the first order bias of the estimator <span class="math-container">$\hat \theta$</span>. Then for the Jacknife estimates, we have a similar expression:
<span class="math-container">$$
\mathbb{E}(\hat \theta_{(i)}) = \theta + \frac{b}{n-1} + O\left(\frac{1}{n^2}\right).
$$</span>
So, ignoring the higher order terms, we get:
<span class="math-container">$$
\mathbb{E}(\hat \theta_{(i)}) - \frac{b}{n-1} =  \mathbb{E}(\hat \theta) - \frac{b}{n},\\
\to (n-1)\mathbb{E}(\hat \theta_{(i)} - \hat \theta) = \frac{b}{n}
$$</span>
This shows that: <span class="math-container">$(n-1) (\hat \theta_{(i)} - \hat \theta)$</span> is an `unbiased' estimator for <span class="math-container">$\dfrac{b}{n}$</span> (at least when we are ignoring all other higher order bias terms). Then using this correction, we have the following 'bias-corrected estimate':
<span class="math-container">$$
pv_{(i)} = \hat \theta + (n-1)(\hat \theta - \hat \theta_{(i)}) = n \hat \theta + (n-1) \hat \theta_{(i)}.
$$</span>
This is called the pseudovalue.</p>
<p>Now, of course it would be stupid to only use <span class="math-container">$pv_{(i)}$</span> for one particular value of <span class="math-container">$i$</span> . So a better bias-corrected estimate is the one that averages over all these, the jacknife estimator:
<span class="math-container">$$
\hat \theta_{jack} = \frac{1}{n}\sum_{i = }^n pv_i =  n \hat \theta + (n-1) \hat \theta_{(.)}.
$$</span>
The pseudovalues <span class="math-container">$pv_{(i)}$</span> are not necessarily i.i.d. but assume they are anyway. Then, their variance is given by:
<span class="math-container">$$
\begin{align*}
{\rm var}(pv_i) &amp;= \frac{1}{n-1} \sum_{i = 1}^n (pv_i - \hat \theta_{jack})^2,\\
&amp;= \frac{1}{n-1} \sum_{i = 1}^n (n \hat \theta - (n-1)\hat \theta_{(.)})^2,\\
&amp;= \frac{1}{n-1} \sum_{i = 1}^n (n \hat \theta + (n-1) \hat \theta_{(i)} - n \hat \theta - (n-1) \hat \theta_{(.)} )^2,\\
&amp;= \frac{(n-1)^2}{n-1} \sum_{i = 1}^n (\hat \theta_{(i)}- \hat \theta_{(.)})^2,\\
&amp;= (n-1) \sum_{i = 1}^n (\hat \theta_{(i)} - \hat \theta_{(.)})^2
\end{align*}
$$</span>
Assuming a central limit theorem holds for <span class="math-container">$\hat \theta_{jack}$</span>, then:
<span class="math-container">$$
\frac{\theta_{jack} - \theta}{s_n} \to^d N(0,1)
$$</span>
where <span class="math-container">$s_n$</span> is the variance of <span class="math-container">$pv_i$</span> divided by <span class="math-container">$n$</span>.
<span class="math-container">$$
s_n^2 = \frac{n-1}{n} \sum_{i = 1}^n \left(\hat \theta_{(i)} - \hat \theta_{(.)}\right)^2.
$$</span>
This 'variance', <span class="math-container">$s^2_n$</span>, is the estimator that you give in your question.</p>

CC BY-SA 4.0
2021-05-25T09:09:30.263Z
Thanks for this, though I have 2 questions. 1. When you say things like "biased estimator $\hat{\theta}$", I guess you mean "possibly biased estimator $\hat{\theta}$ "? One can use the jackknife even if one's estimator is unbiased (right?) 2. Shouldn't your pseudo-value be $pv_{(i)} = \hat \theta - (n-1)(\hat \theta - \hat \theta_{(i)})$ (minus not plus)?
afreelunch
CC BY-SA 4.0
2021-05-25T10:37:30.483Z
@afreelunch as far as I know, the jacknife is mainly used as a bias correction (but probably you can also use it if the estimator is unbiased). I think the pseudovalues are correct. you compute $\hat \theta - b/n \approx \hat \theta - (n-1)(\hat \theta_{(i)} - \hat \theta)$ which equals $\hat \theta + (n-1)(\hat \theta - \hat \theta_{(i)})$ which in turn equals $n \hat \theta - (n-1) \hat \theta_{(i)}$ as on slide 7 [here](https://myweb.uiowa.edu/pbreheny/uk/teaching/621/notes/9-6.pdf).
tdm
CC BY-SA 4.0
2021-05-25T10:55:00.870Z
Apologies I misread your equation (didn't see you had interchanged $\hat {\theta}$ and $\hat {\theta}_{(i)}$)!
afreelunch
CC BY-SA 4.0
2021-05-22T18:06:12.947Z
2021-05-22T18:20:34.043Z
2021-05-22T18:20:34.043Z
tdm
<p>Suppose that I am trying to use the jack-knife to estimate the variance of some estimator <span class="math-container">$E$</span>. If I have <span class="math-container">$n$</span> data points, I begin by computing <span class="math-container">$n$</span> estimates (call them <span class="math-container">$B_1$</span>, ..., <span class="math-container">$B_n$</span>), each obtained from leaving one of the <span class="math-container">$n$</span> data points out of my sample. If I understand it correctly, <a href="https://en.wikipedia.org/wiki/Jackknife_resampling#cite_note-FOOTNOTEEfron198214-3" rel="nofollow noreferrer">the Wikipedia page</a> then suggests that my estimate of the variance should be</p>
<p><span class="math-container">$$\hat{Var(E)} = \frac{\sum_{i = 1}^n (B_i - \bar{B_i})^2}{n-1}$$</span></p>
<p>where <span class="math-container">$\bar{B_i}$</span> is the average of my <span class="math-container">$n$</span> estimates. The standard error is then <span class="math-container">$\sqrt(\hat{Var(E)})$</span>.</p>
<p>On the other hand, the <a href="http://people.bu.edu/aimcinto/jackknife.pdf" rel="nofollow noreferrer">notes here</a> seem to suggest</p>
<p><span class="math-container">$$\hat{Var(E)} = \sum_{i = 1}^n (B_i - \bar{B_i})^2\left(\frac{n-1}{n}\right)$$</span></p>
<p>What am I missing here? And which formula should I use? Many thanks in advance!</p>
<p>Extra question: if the answer is indeed the second formula, will standard errors indeed get smaller (as one would expect) as <span class="math-container">$n$</span> gets large? I ask because <span class="math-container">$\sum_{i = 1}^n (B_i - \bar{B_i})^2/n \approx Var(B_i)$</span> for large <span class="math-container">$n$</span>; so it appears that the second estimate is roughly <span class="math-container">$Var(B_i)(n-1)$</span>. It is not clear to me that this is decreasing in <span class="math-container">$n$</span> (even if <span class="math-container">$Var(B_i)$</span> goes to zero as <span class="math-container">$n$</span> gets large).</p>

<statistics>
Confusion about the jack-knife
afreelunch
# Confusion about the jack-knife

(Asked by: username_0 on 2021-05-21)

Suppose that I am trying to use the jack-knife to estimate the variance of some estimator $E$. If I have $n$ data points, I begin by computing $n$ estimates (call them $B\_1$, ..., $B\_n$), each obtained from leaving one of the $n$ data points out of my sample. If I understand it correctly, [the Wikipedia page](https://en.wikipedia.org/wiki/Jackknife_resampling#cite_note-FOOTNOTEEfron198214-3) then suggests that my estimate of the variance should be

$$\hat{Var(E)} = \frac{\sum\_{i = 1}^n (B\_i - \bar{B\_i})^2}{n-1}$$

where $\bar{B\_i}$ is the average of my $n$ estimates. The standard error is then $\sqrt(\hat{Var(E)})$.

On the other hand, the [notes here](http://people.bu.edu/aimcinto/jackknife.pdf) seem to suggest

$$\hat{Var(E)} = \sum\_{i = 1}^n (B\_i - \bar{B\_i})^2\left(\frac{n-1}{n}\right)$$

What am I missing here? And which formula should I use? Many thanks in advance!

Extra question: if the answer is indeed the second formula, will standard errors indeed get smaller (as one would expect) as $n$ gets large? I ask because $\sum\_{i = 1}^n (B\_i - \bar{B\_i})^2/n \approx Var(B\_i)$ for large $n$; so it appears that the second estimate is roughly $Var(B\_i)(n-1)$. It is not clear to me that this is decreasing in $n$ (even if $Var(B\_i)$ goes to zero as $n$ gets large).

**Tags:** statistics

---

## 1 answers

### Answer by username_1 on 2021-05-22. Score: 2 | Accepted answer

I'm not a statistician... and this is the economics forum, so forgive me for the possible mistakes. I got most of this information from these [slides](https://myweb.uiowa.edu/pbreheny/uk/teaching/621/notes/9-6.pdf).

Consider a statistic of interest $\theta$ with a consistent but biased estimate $\hat \theta$. The Jacknife estimates $\hat \theta\_{(i)}$ with is the same statistic, based on the sample that is obtained from removing observation $i$. Doing this for all $i$ and taking the mean over the values $\hat \theta\_{(i)}$ gives:
$$
\hat \theta\_{(.)} = \frac{1}{n} \sum\_{i = 1}^n \hat \theta\_{(i)}
$$
As stated above, the estimator $\hat \theta$ is biased, so let:
$$
\mathbb{E}(\hat \theta) = \theta + \frac{b}{n} + O\left(\frac{1}{n^2}\right)
$$
Where $b$ is the first order bias of the estimator $\hat \theta$. Then for the Jacknife estimates, we have a similar expression:
$$
\mathbb{E}(\hat \theta\_{(i)}) = \theta + \frac{b}{n-1} + O\left(\frac{1}{n^2}\right).
$$
So, ignoring the higher order terms, we get:
$$
\mathbb{E}(\hat \theta\_{(i)}) - \frac{b}{n-1} = \mathbb{E}(\hat \theta) - \frac{b}{n},\\
\to (n-1)\mathbb{E}(\hat \theta\_{(i)} - \hat \theta) = \frac{b}{n}
$$
This shows that: $(n-1) (\hat \theta\_{(i)} - \hat \theta)$ is an `unbiased' estimator for $\dfrac{b}{n}$ (at least when we are ignoring all other higher order bias terms). Then using this correction, we have the following 'bias-corrected estimate':
$$
pv\_{(i)} = \hat \theta + (n-1)(\hat \theta - \hat \theta\_{(i)}) = n \hat \theta + (n-1) \hat \theta\_{(i)}.
$$
This is called the pseudovalue.

Now, of course it would be stupid to only use $pv\_{(i)}$ for one particular value of $i$ . So a better bias-corrected estimate is the one that averages over all these, the jacknife estimator:
$$
\hat \theta\_{jack} = \frac{1}{n}\sum\_{i = }^n pv\_i = n \hat \theta + (n-1) \hat \theta\_{(.)}.
$$
The pseudovalues $pv\_{(i)}$ are not necessarily i.i.d. but assume they are anyway. Then, their variance is given by:
$$
\begin{align\*}
{\rm var}(pv\_i) &= \frac{1}{n-1} \sum\_{i = 1}^n (pv\_i - \hat \theta\_{jack})^2,\\
&= \frac{1}{n-1} \sum\_{i = 1}^n (n \hat \theta - (n-1)\hat \theta\_{(.)})^2,\\
&= \frac{1}{n-1} \sum\_{i = 1}^n (n \hat \theta + (n-1) \hat \theta\_{(i)} - n \hat \theta - (n-1) \hat \theta\_{(.)} )^2,\\
&= \frac{(n-1)^2}{n-1} \sum\_{i = 1}^n (\hat \theta\_{(i)}- \hat \theta\_{(.)})^2,\\
&= (n-1) \sum\_{i = 1}^n (\hat \theta\_{(i)} - \hat \theta\_{(.)})^2
\end{align\*}
$$
Assuming a central limit theorem holds for $\hat \theta\_{jack}$, then:
$$
\frac{\theta\_{jack} - \theta}{s\_n} \to^d N(0,1)
$$
where $s\_n$ is the variance of $pv\_i$ divided by $n$.
$$
s\_n^2 = \frac{n-1}{n} \sum\_{i = 1}^n \left(\hat \theta\_{(i)} - \hat \theta\_{(.)}\right)^2.
$$
This 'variance', $s^2\_n$, is the estimator that you give in your question.

> **Comments:**
> **username_0**: Thanks for this, though I have 2 questions. 1. When you say things like "biased estimator $\hat{\theta}$", I guess you mean "possibly biased estimator $\hat{\theta}$ "? One can use the jackknife even if one's estimator is unbiased (right?) 2. Shouldn't your pseudo-value be $pv_{(i)} = \hat \theta - (n-1)(\hat \theta - \hat \theta_{(i)})$ (minus not plus)?  
> **username_1**: @afreelunch as far as I know, the jacknife is mainly used as a bias correction (but probably you can also use it if the estimator is unbiased). I think the pseudovalues are correct. you compute $\hat \theta - b/n \approx \hat \theta - (n-1)(\hat \theta_{(i)} - \hat \theta)$ which equals $\hat \theta + (n-1)(\hat \theta - \hat \theta_{(i)})$ which in turn equals $n \hat \theta - (n-1) \hat \theta_{(i)}$ as on slide 7 [here](https://myweb.uiowa.edu/pbreheny/uk/teaching/621/notes/9-6.pdf).  
> **username_0**: Apologies I misread your equation (didn't see you had interchanged $\hat {\theta}$ and $\hat {\theta}_{(i)}$)!  

---

