<p>Why don't we use model averaging? Sure, the clever econometricians will say that the coefficients are not directly comparable and in theory there is one correct model with that exact set of covariates. The reality is that all applications have huge degrees of freedom choosing covariates and functional forms. We don't know which one is correct, and everybody chooses the one they like because either it gives stars or it fits nicely into the story they want to write.</p>
<p>Wouldn't it be more honest to have some systematic approach of reporting an average across 100 possible specifications?</p>
<p>** Clarification, further thoughts. **
I mean averaging across models of the same family, but across all the less discussed decisions which make up the <a href="https://en.wikipedia.org/wiki/Researcher_degrees_of_freedom" rel="nofollow noreferrer">researcher degrees of freedom</a> like covariates or functional form assumptions.</p>
<p>One issue is, of course, that even the little decisions imply different models. For example, a DiD in logs has a different identifying assumption than a DiD in levels. If one is correct, what is the informativeness of showing the average of both? But, on the other hand, do we really want to make policy decisions based on a DiD result which holds only in logs?</p>
<p>For covariates, I think the issue is more straightforward. It really annoys me how people present their regression equation as if the choice of covariates was the only feasible one. Usually there are many different possible sets we could include, we have much weaker priors on which set of covariates is necessary to ensure exogeneity and the choice makes HUGE differences. I think it would be sensible to force authors to show average and variance across a large set of possible covariate sets.</p>

CC BY-SA 4.0
2024-03-14T14:19:06.897Z
An average of what? (what values, across what population of models)
Giskard
CC BY-SA 4.0
2024-03-14T14:43:55.930Z
Model averaging is used in central banks or other institutions for forecasting. Why would you use it outside forecasting application? Averaging results between let’s say OLS and Dif-in-Dif or IV does not make sense
1muflon1
CC BY-SA 4.0
2024-03-14T17:40:39.893Z
I mean averaging across 50 plausible DiD specifications. Right now everybody conduct "robustness" by showing the changes under which results hold without showing specifications where results change. I want to average across all specifications.
Papayapap
CC BY-SA 4.0
2024-03-14T17:54:17.677Z
Papayapap, I think Andrew Gelman advocates something similar to your approach of reporting all models (with *all* defined within some sensible bounds). He has mentioned this a few times in his blog. @1muflon1, I think your phrasing of *results* is a little imprecise. If you take a result to mean a forecast, you said yourself that model averaging is being used. Also, I think we can define a common estimand of interest and use multiple models to estimate it. The models will use different assumptions, but the estimand can stay the same across the models.
Richard Hardy
CC BY-SA 4.0
2024-03-14T17:58:58.197Z
...In case of forecasting, the common estimand may be the expected value of a future observation. In case of causal estimation, it may be a well defined causal effect of $X$ on $Y$.
Richard Hardy
CC BY-SA 4.0
2024-03-15T06:27:29.473Z
@Papayapap Your [comment](https://economics.stackexchange.com/questions/57882/why-dont-we-use-model-averaging#comment96014_57882) includes a lot of details that your Q does not. As currently phrased, the Q is very broad (what industry, what purpose, averaging what, over which population of models, etc.) Do you think you could maybe go over it and edit in more details? E.g. those in your comment and any others you would like to add.
Giskard
CC BY-SA 4.0
2024-03-15T10:37:36.740Z
@Papayapap It is still not clear to me what it is that you are averaging, since one cannot "average a model". Are you averaging forecast values? Variable coefficients? Something else?
Giskard
CC BY-SA 4.0
2024-03-15T10:39:57.790Z
Sorry! I am thinking about regression coefficients, mostly in causal analysis
Papayapap
<econometrics>
Why don't we use model averaging?
Papayapap
# Why don't we use model averaging?

(Asked by: username_0 on 2024-03-14)

Why don't we use model averaging? Sure, the clever econometricians will say that the coefficients are not directly comparable and in theory there is one correct model with that exact set of covariates. The reality is that all applications have huge degrees of freedom choosing covariates and functional forms. We don't know which one is correct, and everybody chooses the one they like because either it gives stars or it fits nicely into the story they want to write.

Wouldn't it be more honest to have some systematic approach of reporting an average across 100 possible specifications?

\*\* Clarification, further thoughts. \*\*
I mean averaging across models of the same family, but across all the less discussed decisions which make up the [researcher degrees of freedom](https://en.wikipedia.org/wiki/Researcher_degrees_of_freedom) like covariates or functional form assumptions.

One issue is, of course, that even the little decisions imply different models. For example, a DiD in logs has a different identifying assumption than a DiD in levels. If one is correct, what is the informativeness of showing the average of both? But, on the other hand, do we really want to make policy decisions based on a DiD result which holds only in logs?

For covariates, I think the issue is more straightforward. It really annoys me how people present their regression equation as if the choice of covariates was the only feasible one. Usually there are many different possible sets we could include, we have much weaker priors on which set of covariates is necessary to ensure exogeneity and the choice makes HUGE differences. I think it would be sensible to force authors to show average and variance across a large set of possible covariate sets.

**Tags:** econometrics
> **Comments:**
> **username_1**: An average of what? (what values, across what population of models)  
> **username_2**: Model averaging is used in central banks or other institutions for forecasting. Why would you use it outside forecasting application? Averaging results between let’s say OLS and Dif-in-Dif or IV does not make sense  
> **username_0**: I mean averaging across 50 plausible DiD specifications. Right now everybody conduct "robustness" by showing the changes under which results hold without showing specifications where results change. I want to average across all specifications.  
> **username_3**: Papayapap, I think Andrew Gelman advocates something similar to your approach of reporting all models (with *all* defined within some sensible bounds). He has mentioned this a few times in his blog. @1muflon1, I think your phrasing of *results* is a little imprecise. If you take a result to mean a forecast, you said yourself that model averaging is being used. Also, I think we can define a common estimand of interest and use multiple models to estimate it. The models will use different assumptions, but the estimand can stay the same across the models.  
> **username_3**: ...In case of forecasting, the common estimand may be the expected value of a future observation. In case of causal estimation, it may be a well defined causal effect of $X$ on $Y$.  
> **username_1**: @Papayapap Your [comment](https://economics.stackexchange.com/questions/57882/why-dont-we-use-model-averaging#comment96014_57882) includes a lot of details that your Q does not. As currently phrased, the Q is very broad (what industry, what purpose, averaging what, over which population of models, etc.) Do you think you could maybe go over it and edit in more details? E.g. those in your comment and any others you would like to add.  
> **username_1**: @Papayapap It is still not clear to me what it is that you are averaging, since one cannot "average a model". Are you averaging forecast values? Variable coefficients? Something else?  
> **username_0**: Sorry! I am thinking about regression coefficients, mostly in causal analysis  

---

## 0 answers

