<p>Without knowing more about your specific project, you probably want to look at difference-in-difference methodologies. It's the bread and butter of this sort of psudo-experiment. </p>

<p>I see you use the phrases "train" and "test".  Be very careful, typically machine learning approaches are designed to get very good <span class="math-container">$\hat{Y}$</span>s.  Economists in general <em>don't care about <span class="math-container">$\hat{Y}$</span>s much at all</em>. We care about having good <span class="math-container">$\hat{\beta}$</span>'s. It will take some time to get used to that idea.</p>

<p>Naive use of traditional ML techniques will lead to highly biased estimators and very misleading estimates of <span class="math-container">$\hat{Y}$</span> that often lead to bad policy decisions.</p>

CC BY-SA 4.0
2018-10-03T22:51:28.037Z
Prediction vs. inference and the bias-variance tradeoff, in other words?
Kenneth Rios
CC BY-SA 4.0
2018-10-04T18:23:14.137Z
I would say those often-quoted statements do not properly imply how likely it is that economic data is catastrophically biased.  In practice, if you are using unmodified ML techniques in a standard area of economic research (at least any I am familiar with) you are likely to generate poor estimates of the effects of the very policies you wish to understand. These naive effect estimates are often highly biased in unanticipated directions, and may lead to catastrophic misinterpretations of the effects of a given policy. Blending the ML and econometric toolsets is a new area of active research.
RegressForward
CC BY-SA 4.0
2019-07-23T04:06:20.843Z
Hey RF, I know this is a bit of a blast from the past, but I was wondering if you could forward along some of the "active research" you mention blending metrics and ML. Would be much appreciated.
Kenneth Rios
CC BY-SA 4.0
2019-07-24T15:33:04.620Z
Sure! I have an article coming out in August that should give a good crossover of the econometrics side of things in non-economics settings. "Resolving Simultaneity Bias: Using Features to Estimate Causal Effects in Competitive Games." IEEE Conference on Games.   I also strongly recommend Susan Athey's work, she is cited several times, with good reason.
RegressForward
CC BY-SA 4.0
2019-07-25T03:41:26.910Z
Thank you for the resource, and I am quite familiar with Athey's work.
Kenneth Rios
CC BY-SA 4.0
2018-10-03T22:24:24.543Z
2018-10-03T22:24:24.543Z
RegressForward
<p>Not sure what information the training data gives you, but in this study it appears that you want to estimate the effect of knowledge of a (one-shot) increase in gas prices on gasoline consumption. From that research question, it is clear that the two relevant groups of comparison are</p>

<ul>
<li>people who have not noticed a​ gas-price increase (<strong>control</strong> group), and</li>
<li>people who have recently noticed a​ gas-price increase (<strong>treatment</strong> group).</li>
</ul>

<p>Depending on how rich the training data is, you could run a variety of robust models. It would also be important to not overstate your findings to your boss -- ideally you would have many such instances of gasoline price hikes, not just one: it is plausible that during the time period in question there was an external factor that made one group much more or much less likely to purchase gas, independent of the price hike. A panel set would put you in a better position to control for such factors and make more substantial claims.</p>

CC BY-SA 4.0
2018-10-03T16:08:45.323Z
2018-10-03T23:18:27.413Z
2018-10-03T23:18:27.413Z
Kenneth Rios
<p>I'd like to have your opinion on the following; </p>

<p>Assuming that I'm doing a research on setting the optimal price for gasoline for a company and my client wants me to perform a study on​ customers' gasoline purchasing habits when they notice an increase in the price of gas. </p>

<p>I have a training data on these categories: </p>

<ul>
<li>People who have not noticed a​ gas-price increase
  </li>
<li>People who drive electric cars
  </li>
<li>People who have recently noticed a​ gas-price increase
  </li>
<li>People who ride their bikes to work
  </li>
<li>People who ride the bus</li>
</ul>

<p>Where I am going to form a treatment and a control group. Though I'm not sure how to pick the categories I'll be including in the test. </p>

<p>My opinion (definitely not an expert), I would ignore the people who ride their bikes to work and who drive electric cars, as they are simply out of the interest of the research. And randomly distribute the other categories' members. Thing I'm not sure is about the people who ride the bus; should I include them or not? </p>

<p>However my other opinion is that there might be also some switches for EVs, bikes or public bus. In this case I'd better include every category in the data. </p>

<p>Thanks in advance for every opinion. </p>

CC BY-SA 4.0
2018-10-03T20:20:44.680Z
Is the experiment hypothetical or real (i.e. are you going to simply send people surveys, or the chosen participants would be involved in a program where they'll have to pay for real gas at varying prices)?
Herr K.
<microeconomics>
Regarding the group formation in an oil-price experiment
EGE
# Regarding the group formation in an oil-price experiment

(Asked by: username_0 on 2018-10-03)

I'd like to have your opinion on the following;

Assuming that I'm doing a research on setting the optimal price for gasoline for a company and my client wants me to perform a study on​ customers' gasoline purchasing habits when they notice an increase in the price of gas.

I have a training data on these categories:

* People who have not noticed a​ gas-price increase
* People who drive electric cars
* People who have recently noticed a​ gas-price increase
* People who ride their bikes to work
* People who ride the bus

Where I am going to form a treatment and a control group. Though I'm not sure how to pick the categories I'll be including in the test.

My opinion (definitely not an expert), I would ignore the people who ride their bikes to work and who drive electric cars, as they are simply out of the interest of the research. And randomly distribute the other categories' members. Thing I'm not sure is about the people who ride the bus; should I include them or not?

However my other opinion is that there might be also some switches for EVs, bikes or public bus. In this case I'd better include every category in the data.

Thanks in advance for every opinion.

**Tags:** microeconomics
> **Comments:**
> **username_1**: Is the experiment hypothetical or real (i.e. are you going to simply send people surveys, or the chosen participants would be involved in a program where they'll have to pay for real gas at varying prices)?  

---

## 2 answers

### Answer by username_2 on 2018-10-03. Score: 1

Without knowing more about your specific project, you probably want to look at difference-in-difference methodologies. It's the bread and butter of this sort of psudo-experiment.

I see you use the phrases "train" and "test". Be very careful, typically machine learning approaches are designed to get very good $\hat{Y}$s. Economists in general *don't care about $\hat{Y}$s much at all*. We care about having good $\hat{\beta}$'s. It will take some time to get used to that idea.

Naive use of traditional ML techniques will lead to highly biased estimators and very misleading estimates of $\hat{Y}$ that often lead to bad policy decisions.

> **Comments:**
> **username_3**: Prediction vs. inference and the bias-variance tradeoff, in other words?  
> **username_2**: I would say those often-quoted statements do not properly imply how likely it is that economic data is catastrophically biased.  In practice, if you are using unmodified ML techniques in a standard area of economic research (at least any I am familiar with) you are likely to generate poor estimates of the effects of the very policies you wish to understand. These naive effect estimates are often highly biased in unanticipated directions, and may lead to catastrophic misinterpretations of the effects of a given policy. Blending the ML and econometric toolsets is a new area of active research.  
> **username_3**: Hey RF, I know this is a bit of a blast from the past, but I was wondering if you could forward along some of the "active research" you mention blending metrics and ML. Would be much appreciated.  
> **username_2**: Sure! I have an article coming out in August that should give a good crossover of the econometrics side of things in non-economics settings. "Resolving Simultaneity Bias: Using Features to Estimate Causal Effects in Competitive Games." IEEE Conference on Games.   I also strongly recommend Susan Athey's work, she is cited several times, with good reason.  
> **username_3**: Thank you for the resource, and I am quite familiar with Athey's work.  

---

### Answer by username_3 on 2018-10-03. Score: 0

Not sure what information the training data gives you, but in this study it appears that you want to estimate the effect of knowledge of a (one-shot) increase in gas prices on gasoline consumption. From that research question, it is clear that the two relevant groups of comparison are

* people who have not noticed a​ gas-price increase (**control** group), and
* people who have recently noticed a​ gas-price increase (**treatment** group).

Depending on how rich the training data is, you could run a variety of robust models. It would also be important to not overstate your findings to your boss -- ideally you would have many such instances of gasoline price hikes, not just one: it is plausible that during the time period in question there was an external factor that made one group much more or much less likely to purchase gas, independent of the price hike. A panel set would put you in a better position to control for such factors and make more substantial claims.


---

