<p>No, this should not be a major issue. We will very carefully take into account the different sample sizes. Allow me to continue your example:</p>
<p>Assume &quot;Resume A&quot; is the the treatment resume and &quot;Resume B&quot; is the control resume, where the treatment has a less advantaged ethnic name (Jamal, Beyonce) and the control resume contains a anglicized name (James, Sophia). One should be careful about gender here as well.</p>
<p>We then compare the mean acceptance rate between the two resumes.
Suppose <span class="math-container">$N_a = 300$</span> and <span class="math-container">$N_b = 310$</span>.
The accepted resumes are: <span class="math-container">$Accepted_a = 30$</span> and <span class="math-container">$Accepted_b = 62$</span>.
For shortcuts, the mean rates of acceptance are <span class="math-container">$r_a = 30/300 = 0.1$</span> and <span class="math-container">$r_b = 62/310 = 0.2$</span>. Notice the mean rate of acceptance normalizes each group, so size is partially accounted for here.</p>
<p>The variances are then:</p>
<p><span class="math-container">$\sigma_a^2 = (1-r_a)*r_a = 0.090$</span></p>
<p><span class="math-container">$\sigma_b^2 = (1-r_b)*r_b = 0.160$</span></p>
<p>And more importantly, the variance of the <em>means</em> of each distribution are then <span class="math-container">$\sigma_a^2/N_a$</span> and <span class="math-container">$\sigma_b^2/N_b$</span>, respectively.  We are comparing the means of the two distributions with a simple mean comparison test. Notice this step explicitly uses the sample size to account for expected variation between the two distributions.</p>
<p>So we now compare the two to see if there is a difference, under the null hypotheses there is no difference. (In statistics, you nearly always always assume no difference/no relevance/no effect as the baseline.)</p>
<p><span class="math-container">$t=\frac{r_a-r_b}{\sqrt{\sigma_a^2/N_a+\sigma_b^2/N_b}} = 3.572$</span></p>
<p>The reasoning can be very explicitly laid out as follows:</p>
<ul>
<li>The difference between the two samples is 3.572 standard deviations
away from 0.</li>
<li>Under the assumptions we have made above, it seems very
unlikely that this would occur by chance if we repeated the process
(looking this up on the t-table as a one-tailed test has a p-value of less than 0.001). Bayesians will point out there are some other silent assumptions one may have made, so language is particularly important at this step, but this generally passes the intuition.</li>
<li>Therefore, I reject the idea that these two samples came from the same distribution. It seems implausible.</li>
<li>So, I am left to conclude that there is a difference between the two samples. It seems that indeed people are more accepting of the Anglicized names rather than the ethnic names.</li>
</ul>

CC BY-SA 4.0
2020-10-02T03:44:57.980Z
Thanks for the really great explanation. Assuming that we work instead with numbers of acceptances rather than normalised rates (as much as it is a poor approach), sample sizes would pose an issue then? Here, I presume that a t-test would be used?
kékszajkók
CC BY-SA 4.0
2020-10-02T08:04:26.423Z
@jo_1 How do you "...work...with numbers of acceptances"? What would be the statistic? The population parameter of interest is the probability of acceptance, conditional on control/treatment. Taking the sample rate is not some "fix" that's supposed to be remedy for small sample size. It's just estimating the population moment by sample moment. (If sample size is a problem and one would like to use the same model, only fix is to get a larger sample.)
Michael
CC BY-SA 4.0
2020-10-02T10:14:36.210Z
Point taken. Just realized that we would simply not be able to come up with a statistic based on numbers of acceptances.
kékszajkók
CC BY-SA 4.0
2020-10-02T14:39:31.243Z
Note though: when the sample sizes in the treatment arms get very different, total uncertainty in the difference becomes dominated by the size of the smaller treatment arm.
cbeleites
CC BY-SA 4.0
2020-10-02T19:59:07.570Z
@cbeleitesunhappywithSX relatedly, equal sized groups are most often chosen because it gives the highest efficiency. In other words if you were doing a clinical trial on a covid vaccine vs placebo and had n people, the most optimal experiment would be to assign each group n/2 people.  there are other considerations though, as Joe's answer points out.
eps
CC BY-SA 4.0
2020-10-01T15:14:50.103Z
2020-10-01T15:21:25.293Z
2020-10-01T15:21:25.293Z
RegressForward
<p>They generally measure the probability of getting an interview, not the number of interviews, so that that the number of applications is normalized out. For example, consider <a href="https://www.aeaweb.org/articles?id=10.1257/0002828042002561" rel="noreferrer">Are Emily and Greg More Employable Than Lakisha and Jamal? A Field Experiment on Labor Market Discrimination (AER 2004 with ~5000 citations)</a>.</p>
<blockquote>
<p>We study race in the labor market by sending fictitious resumes to
help-wanted ads in Boston and Chicago newspapers. To manipulate
perceived race, resumes are randomly assigned African-American- or
White-sounding names. White names receive 50 percent more callbacks
for interviews. Callbacks are also more respon- sive to resume quality
for White names than for African-American ones. The racial gap is
uniform across occupation, industry, and employer size. We also find
little evidence that employers are inferring social class from the
names. Differential treatment by race still appears to still be
prominent in the U.S. labor market</p>
</blockquote>
<p>That paper uses &quot;Likelihood of a Callback&quot; and &quot;Callback Rate&quot; as the primary variables of interest, and these measures use callbacks normalized by applications which means that they are not sensitive to the number of applications (in the levels as least, but the number of applications influences the standard errors).</p>

CC BY-SA 4.0
2020-10-01T15:08:18.763Z
Thanks for the reply. This means that an experiment that uses raw numbers as its variable of interest is necessarily sensitive to the number of applications?
kékszajkók
CC BY-SA 4.0
2020-10-01T15:20:45.267Z
Yes, unless you are careful to balance the sizes of the groups and sub-populations. Normalization is generally going to be the superior and more flexible approach.
BKay
CC BY-SA 4.0
2020-10-01T14:56:25.397Z
2020-10-01T14:56:25.397Z
BKay
<p>There are legitimate reasons for wanting to send out more to one group than the other - for example, if you need a sample size of at least 100 to have sufficient power to use certain tests, and you wanted to be able to do some in-group comparisons, you might oversample one or the other.</p>
<p>Example: You have 1000 jobs available to sample.  You want to test the following:</p>
<ul>
<li>English names vs. Hispanic names</li>
<li>English names vs. Black names</li>
<li>English names vs. Black and Hispanic names</li>
<li>Black names vs. Hispanic names</li>
<li>Black nonhispanic names vs. Black hispanic names</li>
<li>Black male names vs. Black female names</li>
</ul>
<p>And the desired power requires at least n=100.  So you might take:</p>
<ul>
<li>300 English (gender irrelevant)</li>
<li>250 Black Male inc. 100 Black Hispanic Male</li>
<li>250 Black Female inc. 100 Black Hispanic Female</li>
<li>200 White Hispanic (gender irrelevant)</li>
</ul>
<p>That gives us a sufficient power to test quite a few things within groups, while still having the larger out-of-group samples with better power (n=300 vs n=500 for English vs. Black, for example.)</p>
<p>When we do a statistical test, then, the n-sizes of both groups do factor into the test, but they're constructed to consider that.  As such, the difference in n sizes is not a reason in and of itself to invalidate a study; it may be for <em>good</em> reasons, or it may simply be a minor irrelevant difference.</p>

CC BY-SA 4.0
2020-10-02T18:50:10.220Z
2020-10-02T18:50:10.220Z
Joe
<p>In traditional field experiments on racial discrimination, two identical resumes differing only by name or ID photo are sent to randomly selected firms. A statistical test is then run on the number of interview offers received by each resume to determine if any difference in the number of interview offers received between the resumes holds true in the population.</p>
<p>I am curious whether, if in the event that each resume is sent to a different number of companies (e.g. one is sent to 300 firms and the other 310), results that show a statistically significant difference in the number of interview offers received on one resume compared to the other will be invalidated. My intuition is that there may be a tendency for bias to be introduced as the resume that is assigned to a greater number of firms has a higher chance of being selected. Would appreciate comments on whether this is accurate.</p>

<econometrics><labor-economics>
Field experiments on labour market discrimination: do sizes of randomly assigned groups matter?
kékszajkók
# Field experiments on labour market discrimination: do sizes of randomly assigned groups matter?

(Asked by: username_0 on 2020-10-01)

In traditional field experiments on racial discrimination, two identical resumes differing only by name or ID photo are sent to randomly selected firms. A statistical test is then run on the number of interview offers received by each resume to determine if any difference in the number of interview offers received between the resumes holds true in the population.

I am curious whether, if in the event that each resume is sent to a different number of companies (e.g. one is sent to 300 firms and the other 310), results that show a statistically significant difference in the number of interview offers received on one resume compared to the other will be invalidated. My intuition is that there may be a tendency for bias to be introduced as the resume that is assigned to a greater number of firms has a higher chance of being selected. Would appreciate comments on whether this is accurate.

**Tags:** econometrics, labor-economics

---

## 3 answers

### Answer by username_1 on 2020-10-01. Score: 9

No, this should not be a major issue. We will very carefully take into account the different sample sizes. Allow me to continue your example:

Assume "Resume A" is the the treatment resume and "Resume B" is the control resume, where the treatment has a less advantaged ethnic name (Jamal, Beyonce) and the control resume contains a anglicized name (James, Sophia). One should be careful about gender here as well.

We then compare the mean acceptance rate between the two resumes.
Suppose $N\_a = 300$ and $N\_b = 310$.
The accepted resumes are: $Accepted\_a = 30$ and $Accepted\_b = 62$.
For shortcuts, the mean rates of acceptance are $r\_a = 30/300 = 0.1$ and $r\_b = 62/310 = 0.2$. Notice the mean rate of acceptance normalizes each group, so size is partially accounted for here.

The variances are then:

$\sigma\_a^2 = (1-r\_a)\*r\_a = 0.090$

$\sigma\_b^2 = (1-r\_b)\*r\_b = 0.160$

And more importantly, the variance of the *means* of each distribution are then $\sigma\_a^2/N\_a$ and $\sigma\_b^2/N\_b$, respectively. We are comparing the means of the two distributions with a simple mean comparison test. Notice this step explicitly uses the sample size to account for expected variation between the two distributions.

So we now compare the two to see if there is a difference, under the null hypotheses there is no difference. (In statistics, you nearly always always assume no difference/no relevance/no effect as the baseline.)

$t=\frac{r\_a-r\_b}{\sqrt{\sigma\_a^2/N\_a+\sigma\_b^2/N\_b}} = 3.572$

The reasoning can be very explicitly laid out as follows:

* The difference between the two samples is 3.572 standard deviations
  away from 0.
* Under the assumptions we have made above, it seems very
  unlikely that this would occur by chance if we repeated the process
  (looking this up on the t-table as a one-tailed test has a p-value of less than 0.001). Bayesians will point out there are some other silent assumptions one may have made, so language is particularly important at this step, but this generally passes the intuition.
* Therefore, I reject the idea that these two samples came from the same distribution. It seems implausible.
* So, I am left to conclude that there is a difference between the two samples. It seems that indeed people are more accepting of the Anglicized names rather than the ethnic names.

> **Comments:**
> **username_0**: Thanks for the really great explanation. Assuming that we work instead with numbers of acceptances rather than normalised rates (as much as it is a poor approach), sample sizes would pose an issue then? Here, I presume that a t-test would be used?  
> **username_2**: @jo_1 How do you "...work...with numbers of acceptances"? What would be the statistic? The population parameter of interest is the probability of acceptance, conditional on control/treatment. Taking the sample rate is not some "fix" that's supposed to be remedy for small sample size. It's just estimating the population moment by sample moment. (If sample size is a problem and one would like to use the same model, only fix is to get a larger sample.)  
> **username_0**: Point taken. Just realized that we would simply not be able to come up with a statistic based on numbers of acceptances.  
> **username_3**: Note though: when the sample sizes in the treatment arms get very different, total uncertainty in the difference becomes dominated by the size of the smaller treatment arm.  
> **username_4**: @cbeleitesunhappywithSX relatedly, equal sized groups are most often chosen because it gives the highest efficiency. In other words if you were doing a clinical trial on a covid vaccine vs placebo and had n people, the most optimal experiment would be to assign each group n/2 people.  there are other considerations though, as Joe's answer points out.  

---

### Answer by username_5 on 2020-10-01. Score: 7

They generally measure the probability of getting an interview, not the number of interviews, so that that the number of applications is normalized out. For example, consider [Are Emily and Greg More Employable Than Lakisha and Jamal? A Field Experiment on Labor Market Discrimination (AER 2004 with ~5000 citations)](https://www.aeaweb.org/articles?id=10.1257/0002828042002561).

> We study race in the labor market by sending fictitious resumes to
> help-wanted ads in Boston and Chicago newspapers. To manipulate
> perceived race, resumes are randomly assigned African-American- or
> White-sounding names. White names receive 50 percent more callbacks
> for interviews. Callbacks are also more respon- sive to resume quality
> for White names than for African-American ones. The racial gap is
> uniform across occupation, industry, and employer size. We also find
> little evidence that employers are inferring social class from the
> names. Differential treatment by race still appears to still be
> prominent in the U.S. labor market

That paper uses "Likelihood of a Callback" and "Callback Rate" as the primary variables of interest, and these measures use callbacks normalized by applications which means that they are not sensitive to the number of applications (in the levels as least, but the number of applications influences the standard errors).

> **Comments:**
> **username_0**: Thanks for the reply. This means that an experiment that uses raw numbers as its variable of interest is necessarily sensitive to the number of applications?  
> **username_5**: Yes, unless you are careful to balance the sizes of the groups and sub-populations. Normalization is generally going to be the superior and more flexible approach.  

---

### Answer by username_6 on 2020-10-02. Score: 1

There are legitimate reasons for wanting to send out more to one group than the other - for example, if you need a sample size of at least 100 to have sufficient power to use certain tests, and you wanted to be able to do some in-group comparisons, you might oversample one or the other.

Example: You have 1000 jobs available to sample. You want to test the following:

* English names vs. Hispanic names
* English names vs. Black names
* English names vs. Black and Hispanic names
* Black names vs. Hispanic names
* Black nonhispanic names vs. Black hispanic names
* Black male names vs. Black female names

And the desired power requires at least n=100. So you might take:

* 300 English (gender irrelevant)
* 250 Black Male inc. 100 Black Hispanic Male
* 250 Black Female inc. 100 Black Hispanic Female
* 200 White Hispanic (gender irrelevant)

That gives us a sufficient power to test quite a few things within groups, while still having the larger out-of-group samples with better power (n=300 vs n=500 for English vs. Black, for example.)

When we do a statistical test, then, the n-sizes of both groups do factor into the test, but they're constructed to consider that. As such, the difference in n sizes is not a reason in and of itself to invalidate a study; it may be for *good* reasons, or it may simply be a minor irrelevant difference.


---

