<p>There are multiple ways to perform various test that can be defended, depending on the context, but they are not, necessarily, equivalent.</p>
<p>The typical way to calculate the standard errors in an OLS linear model assumes equal variances in the two groups (constant-variance error term for the general model). This is the default calculation in most software, and then the confidence intervals and p-values are derived from that standard error.</p>
<p>There is a variant of the t-test called Welch's t-test that assumes (or at least allows for) unequal variances in to two groups. It sounds like you might have used this calculation for what you are calling the &quot;t-test&quot;. For example, this is the default calculation in the <code>t.test</code> function in R software.</p>
<p>While the usual t-test will agree with the usual OLS linear model coefficient test, Welch's test does not agree with the usual t-test.</p>
<p>It sounds like you used some variant of the usual t-test, such as Welch's test, to calculate the standard error and resulting t-statistic and p-value in your &quot;t-test&quot; and then used a different method in the explicit regression.</p>
<p>I give two examples below where the <code>var.equal = F</code> Welch test has a different p-value than the regression and <code>var.equal = T</code> t-test, which have p-values that do agree.</p>
<pre><code>set.seed(2023)
N &lt;- 5
x &lt;- rep(c(0, 1), N)
e1 &lt;- rnorm(length(x), 0, 1)     # Equal variances
e2 &lt;- rnorm(length(x), 0, x + 1) # Unequal variances
y1 &lt;- x + e1
y2 &lt;- x + e2
summary(lm(y1 ~ x))$coef[2, 4]                      # p = 0.006424082
t.test(y1[x == 0], y1[x == 1], var.equal = T)$p.val # p = 0.006424082
t.test(y1[x == 0], y1[x == 1], var.equal = F)$p.val # p = 0.00736509
#
summary(lm(y2 ~ x))$coef[2, 4]                      # p = 0.01837531
t.test(y2[x == 0], y2[x == 1], var.equal = T)$p.val # p = 0.01837531
t.test(y2[x == 0], y2[x == 1], var.equal = F)$p.val # p = 0.02251352
</code></pre>

CC BY-SA 4.0
2023-11-20T19:11:00.083Z
What you so nicely explained is EXACTLY why I asked him to show his work and specifically show what he was doing. But it's been a while now so maybe he worked it out ?
mark leeds
CC BY-SA 4.0
2023-11-20T15:52:29.977Z
2023-11-20T15:52:29.977Z
Dave
<p>While it is true that the regression you apparently run provides as coefficient of the non-constant regressor the difference of the two means, the t-statistic here uses the variance of the OLS estimator, which could be different from the variance used in the statistical two-sample t-test, hence probably the different statistical conclusion.</p>
<p>But without providing the mathematical details of your work, this is all that can be said.</p>
<p><strong>ADDENDUM</strong></p>
<p>This is a response to the post by @Dave. Let two batches of observations, <span class="math-container">$\{y_i \in \mathbf y_1\}$</span>, and <span class="math-container">$\{y_i\in \mathbf y_2\}$</span>. Let <span class="math-container">$\bar y_1$</span> and <span class="math-container">$s^2_1$</span> be the mean and variance of batch 1, and <span class="math-container">$\bar y_2$</span> and <span class="math-container">$s^2_1$</span> correspondingly for batch 2. Let <span class="math-container">$n_1$</span> be the size of batch 1 and <span class="math-container">$n_2$</span> be the size of batch 2, and <span class="math-container">$n=n_1+n_2$</span>.
If we put the batches together and consider the regression equation
<span class="math-container">$$y_i = \alpha + \beta\cdot I\{y_i \in \mathbf y_2\}$$</span>,</p>
<p>and we apply Ordinary Least Squares (as the OP did), we will get</p>
<p><span class="math-container">$$\hat \alpha = \bar y_1, \quad \hat \beta = \bar y_2 - \bar y_1$$</span></p>
<p>while the variance of <span class="math-container">$\hat \beta$</span> turns out to be</p>
<p><span class="math-container">$${\rm Var}(\hat \beta) = \frac {s^2_1}{n_2} + \frac {s^2_2} {n_1}.$$</span></p>
<p>Note how the sample size of batch 2 divides the variance of batch 1, and the sample size of batch 1 divides the variance of batch 2. So the t-test in the context of OLS regression will use</p>
<p><span class="math-container">$$\frac{\hat \beta}{{\rm SD}(\hat \beta)} = \frac{\bar y_2 - \bar y_1}{\left(\frac {s^2_1}{n_2} + \frac {s^2_2} {n_1}\right)^{1/2}}.$$</span>
No assumption is made about equal/unequal sample sizes, and equal/unequal variances.</p>
<p>This is the statistic that must be compared to what stand-alone t-test the OP used.</p>

CC BY-SA 4.0
2023-11-20T04:15:13.183Z
Equal-variance vs unequal-variance standard error, you mean?
Dave
CC BY-SA 4.0
2023-11-20T15:25:20.280Z
@Dave ...compared also to the standard error of a regression coefficient.
Alecos Papadopoulos
CC BY-SA 4.0
2023-11-20T15:31:41.623Z
The usual standard error for an OLS coefficient coincides with the standard error in an equal-variance two-sample t-test. Not all software uses the equal-variance test by default (`t.test` in R uses Welch's test by default, for example), so I could believe that the difference is because the regression calculation used an equal-variance assumption while the standard t-test used an unequal-variance calculation.
Dave
CC BY-SA 4.0
2023-11-20T15:39:11.493Z
@Dave This is why everybody is telling the OP to post their math,
Alecos Papadopoulos
CC BY-SA 4.0
2023-11-19T03:22:43.140Z
2023-11-21T00:45:45.047Z
2023-11-21T00:45:45.047Z
Alecos Papadopoulos
<p>I have two batches of data and I want to check whether the response variable is different across the two batches. I first did a t-test and found that the means of the response variables in the two batches differ significantly. I then run a regression with an intercept and a Group 2 dummy. The coefficient of the Group 2 dummy is insignificant. This regression result seems to be inconsistent with the t-test result. Could anyone please explain this happens and provide a simple example? Thanks.</p>

CC BY-SA 4.0
2023-11-17T08:03:51.760Z
Hi: This won't answer your question but what did you use for the denominator of your t-test ? By this I mean, how did you calculate the sample variance of the difference estimate  Once that expression is  known, it could  possibly explain some of the the difference  Actually, now that I think about it, my advice to you write out exactly what you did in both cases notationally by using equations in latex. ( or whatever it is called here. MathJax ? ) It can kind of be roughly guessed by a reader but it's easier to see it in lights :).
mark leeds
CC BY-SA 4.0
2023-11-17T11:20:57.733Z
Adding to @mark leeds suggestionsss: could you add how you computed the regression (e.g. Python or R syntax and what the dataframe looks like (a table of headers and a few entries). 

Otherwise it's mainly guessing what you may or may not have done.
AKdemy
CC BY-SA 4.0
2023-11-19T16:31:44.407Z
Thanks for your answers. In fact, I used ChatGPT to do this so I don't know the math models behind these. I will dig deeper into the models.
Justin
<econometrics>
t-test result inconsistent with regression
Justin
# t-test result inconsistent with regression

(Asked by: username_0 on 2023-11-17)

I have two batches of data and I want to check whether the response variable is different across the two batches. I first did a t-test and found that the means of the response variables in the two batches differ significantly. I then run a regression with an intercept and a Group 2 dummy. The coefficient of the Group 2 dummy is insignificant. This regression result seems to be inconsistent with the t-test result. Could anyone please explain this happens and provide a simple example? Thanks.

**Tags:** econometrics
> **Comments:**
> **username_1**: Hi: This won't answer your question but what did you use for the denominator of your t-test ? By this I mean, how did you calculate the sample variance of the difference estimate  Once that expression is  known, it could  possibly explain some of the the difference  Actually, now that I think about it, my advice to you write out exactly what you did in both cases notationally by using equations in latex. ( or whatever it is called here. MathJax ? ) It can kind of be roughly guessed by a reader but it's easier to see it in lights :).  
> **username_2**: Adding to @mark leeds suggestionsss: could you add how you computed the regression (e.g. Python or R syntax and what the dataframe looks like (a table of headers and a few entries).   Otherwise it's mainly guessing what you may or may not have done.  
> **username_0**: Thanks for your answers. In fact, I used ChatGPT to do this so I don't know the math models behind these. I will dig deeper into the models.  

---

## 2 answers

### Answer by username_3 on 2023-11-20. Score: 1

There are multiple ways to perform various test that can be defended, depending on the context, but they are not, necessarily, equivalent.
The typical way to calculate the standard errors in an OLS linear model assumes equal variances in the two groups (constant-variance error term for the general model). This is the default calculation in most software, and then the confidence intervals and p-values are derived from that standard error.
There is a variant of the t-test called Welch's t-test that assumes (or at least allows for) unequal variances in to two groups. It sounds like you might have used this calculation for what you are calling the "t-test". For example, this is the default calculation in the t.test function in R software.
While the usual t-test will agree with the usual OLS linear model coefficient test, Welch's test does not agree with the usual t-test.
It sounds like you used some variant of the usual t-test, such as Welch's test, to calculate the standard error and resulting t-statistic and p-value in your "t-test" and then used a different method in the explicit regression.
I give two examples below where the var.equal = F Welch test has a different p-value than the regression and var.equal = T t-test, which have p-values that do agree.
set.seed(2023)
N <- 5
x <- rep(c(0, 1), N)
e1 <- rnorm(length(x), 0, 1)     # Equal variances
e2 <- rnorm(length(x), 0, x + 1) # Unequal variances
y1 <- x + e1
y2 <- x + e2
summary(lm(y1 ~ x))$coef[2, 4]                      # p = 0.006424082
t.test(y1[x == 0], y1[x == 1], var.equal = T)$p.val # p = 0.006424082
t.test(y1[x == 0], y1[x == 1], var.equal = F)$p.val # p = 0.00736509
#
summary(lm(y2 ~ x))$coef[2, 4]                      # p = 0.01837531
t.test(y2[x == 0], y2[x == 1], var.equal = T)$p.val # p = 0.01837531
t.test(y2[x == 0], y2[x == 1], var.equal = F)$p.val # p = 0.02251352

> **Comments:**
> **username_1**: What you so nicely explained is EXACTLY why I asked him to show his work and specifically show what he was doing. But it's been a while now so maybe he worked it out ?  

---

### Answer by username_4 on 2023-11-19. Score: 1

While it is true that the regression you apparently run provides as coefficient of the non-constant regressor the difference of the two means, the t-statistic here uses the variance of the OLS estimator, which could be different from the variance used in the statistical two-sample t-test, hence probably the different statistical conclusion.

But without providing the mathematical details of your work, this is all that can be said.

**ADDENDUM**

This is a response to the post by @Dave. Let two batches of observations, $\{y\_i \in \mathbf y\_1\}$, and $\{y\_i\in \mathbf y\_2\}$. Let $\bar y\_1$ and $s^2\_1$ be the mean and variance of batch 1, and $\bar y\_2$ and $s^2\_1$ correspondingly for batch 2. Let $n\_1$ be the size of batch 1 and $n\_2$ be the size of batch 2, and $n=n\_1+n\_2$.
If we put the batches together and consider the regression equation
$$y\_i = \alpha + \beta\cdot I\{y\_i \in \mathbf y\_2\}$$,

and we apply Ordinary Least Squares (as the OP did), we will get

$$\hat \alpha = \bar y\_1, \quad \hat \beta = \bar y\_2 - \bar y\_1$$

while the variance of $\hat \beta$ turns out to be

$${\rm Var}(\hat \beta) = \frac {s^2\_1}{n\_2} + \frac {s^2\_2} {n\_1}.$$

Note how the sample size of batch 2 divides the variance of batch 1, and the sample size of batch 1 divides the variance of batch 2. So the t-test in the context of OLS regression will use

$$\frac{\hat \beta}{{\rm SD}(\hat \beta)} = \frac{\bar y\_2 - \bar y\_1}{\left(\frac {s^2\_1}{n\_2} + \frac {s^2\_2} {n\_1}\right)^{1/2}}.$$
No assumption is made about equal/unequal sample sizes, and equal/unequal variances.

This is the statistic that must be compared to what stand-alone t-test the OP used.

> **Comments:**
> **username_3**: Equal-variance vs unequal-variance standard error, you mean?  
> **username_4**: @Dave ...compared also to the standard error of a regression coefficient.  
> **username_3**: The usual standard error for an OLS coefficient coincides with the standard error in an equal-variance two-sample t-test. Not all software uses the equal-variance test by default (`t.test` in R uses Welch's test by default, for example), so I could believe that the difference is because the regression calculation used an equal-variance assumption while the standard t-test used an unequal-variance calculation.  
> **username_4**: @Dave This is why everybody is telling the OP to post their math,  

---

