<p>This is an excellent question, but unfortunately with no (yet) agreed upon answer. </p>

<p>Many authors such as in those seminal papers use cross-validation for this. This is basically what is implied in the <a href="http://economics.mit.edu/files/11858" rel="nofollow noreferrer">ADH (2015)</a> paper on German unification. If the synthetic control is good the MAPE (mean absolute prediction error) and RMSPE (root mean square prediction error) should be quite low. Additional cross-validation where the pre-treatment period is split into training and testing period is so far the common way to go, although there is no agreed upon value of how low the prediction errors should be, just that lower is always better. </p>

<p>Next it is also argued that examining if the trend in the countries from donor pool that are assigned non-zero weights and seeing if the pre-treatment trend in those donor countries seem to match the pre-treatment trend of treated country similar way as you would do for DiD (Difference in Differences) as argued <a href="https://www.urban.org/sites/default/files/publication/89246/the_synthetic_control_method_as_a_tool_0.pdf" rel="nofollow noreferrer">here</a>. </p>

<p>Lastly, as crazy as it may seem, yes visual inspection is actually recommended as well. </p>

<p>Beside the above you should also conduct sensitivity analysis by systematically trying to vary the number of countries in the donor pool and see if the resulting synthetic control has still a good fit. That's again also done by ADH 2015 and you will find it in almost all studies as well. Publishing your weights is also considered good practice. </p>

<p>Of course, doing a proper cross-validation is difficult if you have small pre-treatment sample. I was once working on a synthetic control study at one of the euro-system member's central bank with few colleagues where we had very short pre-treatment period due to data constraints. Instead of doing cross-validation we decided to just test whether the real pre-treatment series is statistically significantly different from the synthetic series using couple of non-parametric tests. When we presented the results and methodology at a workshop no objections to that were raised, and there were some very excellent econometricians present in the room. However, beside central bank's brief the study was not published because the results were not very interesting as placebo test showed results were not significant and we decided to abandon the project since we judged chance of publication to be low so the idea never got proper peer review, so take that as you will. </p>

<hr>

<p><strong>Response to edit:</strong></p>

<p>as with classical DiD there is always an issue of potentially delayed effects or anticipation effects that would bias the result. However, in this case the authors argue that the increase after the reunification was actually part of the effect and not delayed effect. As the authors say:</p>

<blockquote>
  <p>Our estimate of the effect of the German reunification on per capita GDP in West Germany is given by the
  difference between the actual West Germany and its synthetic version, visualized in Figure 3. We estimate that the
  German reunification did not have much of an effect on
  West German per capita GDP in the first two years immediately following reunification. In this initial period, per
  capita GDP in the synthetic West Germany is even slightly
  lower than in the actual West Germany, which is broadly
  in line with arguments about an initial demand boom
  (see, e.g., Meinbardt et al. 1995). From 1992 onward,
  however, the two lines diverge substantially. While per
  capita GDP growth decelerates in West Germany, for the
  synthetic West Germany per capita GDP keeps ascending
  at a pace similar to that of the prereunification period.</p>
</blockquote>

<p>However, arguably this is a weak point of the paper.</p>

<p>Otherwise, if there would be a genuine delay in the effect you would have a problem that could be solved similarly as with DiD by identifying the 'correct' date where the treatment started to have an effect and identify the pre and post treatment period based on that.  </p>

CC BY-SA 4.0
2020-06-08T20:11:45.667Z
Thanks for the quick reply! also, can you provide more details on the non-parametric tests you mentioned?
deeyuan
CC BY-SA 4.0
2020-06-08T20:46:12.990Z
@deeyuan we did wilcoxon rank sum test permutation test and bootstrap - we did only these because our pre-treatment samples were only about 20-24 observations depending on variable, I don’t want to say that doing non-parametric test here is the way to go necessary if you have some nice sample size there some better ways how to compare two time series for sure
1muflon1
CC BY-SA 4.0
2020-06-07T21:25:33.513Z
2020-06-08T22:33:53.500Z
2020-06-08T22:33:53.500Z
1muflon1
<p>I'm using the synthetic control method to evaluate a state-level policy (using synth package in Stata). I've read the seminal ADH papers on CA tobacco &amp; Germany reunification, but I still don't know exactly how to decide if the constructed synthetic state is a good fit. What we do at the moment is looking at the graph - to see if there's overlap between the synthetic and the real state. However, I feel this is not a very "robust" way... </p>

<p>Is there any measure that we can use or test on?</p>

<p><strong>Edit</strong>:</p>

<p>After reading through reply (and papers mentioned again), I have another related (but maybe a bit nuanced) question on what to expect at the time of (or immediate after) intervention. </p>

<p>As shown in the two following figures from <a href="https://economics.mit.edu/files/11859" rel="nofollow noreferrer">ADH (2010)</a> and <a href="http://economics.mit.edu/files/11858" rel="nofollow noreferrer">ADH (2015)</a>, there is a small effect in the same year of the intervention for CA tobacco Propostion 99, while in the Germany reunification study, the gap only becomes noticeable 2 years later. In some other papers using SC, there is an immediate effect (such as <a href="https://papers.ssrn.com/sol3/papers.cfm?abstract_id=1597507" rel="nofollow noreferrer">Cavallo et al. (2013)</a>). I suppose as long as there is a consistent story to support the results, the short-term effect size of the intervention will not invalidate the fitness of synthetic control?    </p>

<p><a href="https://i.sstatic.net/hlq7Pm.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/hlq7Pm.png" alt="ADH (2010) CA tobacco"></a>
<a href="https://i.sstatic.net/KN66sm.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/KN66sm.png" alt="ADH (2015) Germany reunification"></a></p>

CC BY-SA 4.0
2020-06-08T22:31:26.343Z
I edited my question in response to your edit, but if you have multiple questions its better to ask them separately rather than keep extending original question. Especially, when the second question, although related, is distinct from the original one. Questions here should not become too broad since such questions are not good fit for the site. See the sites guidance on asking question: https://economics.stackexchange.com/help/on-topic
1muflon1
<applied-econometrics>
How to evaluate whether the synthetic control is good enough
deeyuan
# How to evaluate whether the synthetic control is good enough

(Asked by: username_0 on 2020-06-07)

I'm using the synthetic control method to evaluate a state-level policy (using synth package in Stata). I've read the seminal ADH papers on CA tobacco & Germany reunification, but I still don't know exactly how to decide if the constructed synthetic state is a good fit. What we do at the moment is looking at the graph - to see if there's overlap between the synthetic and the real state. However, I feel this is not a very "robust" way...

Is there any measure that we can use or test on?

**Edit**:

After reading through reply (and papers mentioned again), I have another related (but maybe a bit nuanced) question on what to expect at the time of (or immediate after) intervention.

As shown in the two following figures from [ADH (2010)](https://economics.mit.edu/files/11859) and [ADH (2015)](http://economics.mit.edu/files/11858), there is a small effect in the same year of the intervention for CA tobacco Propostion 99, while in the Germany reunification study, the gap only becomes noticeable 2 years later. In some other papers using SC, there is an immediate effect (such as [Cavallo et al. (2013)](https://papers.ssrn.com/sol3/papers.cfm?abstract_id=1597507)). I suppose as long as there is a consistent story to support the results, the short-term effect size of the intervention will not invalidate the fitness of synthetic control?

[![ADH (2010) CA tobacco](https://i.sstatic.net/hlq7Pm.png)](https://i.sstatic.net/hlq7Pm.png)
[![ADH (2015) Germany reunification](https://i.sstatic.net/KN66sm.png)](https://i.sstatic.net/KN66sm.png)

**Tags:** applied-econometrics
> **Comments:**
> **username_1**: I edited my question in response to your edit, but if you have multiple questions its better to ask them separately rather than keep extending original question. Especially, when the second question, although related, is distinct from the original one. Questions here should not become too broad since such questions are not good fit for the site. See the sites guidance on asking question: https://economics.stackexchange.com/help/on-topic  

---

## 1 answers

### Answer by username_1 on 2020-06-07. Score: 3 | Accepted answer

This is an excellent question, but unfortunately with no (yet) agreed upon answer.

Many authors such as in those seminal papers use cross-validation for this. This is basically what is implied in the [ADH (2015)](http://economics.mit.edu/files/11858) paper on German unification. If the synthetic control is good the MAPE (mean absolute prediction error) and RMSPE (root mean square prediction error) should be quite low. Additional cross-validation where the pre-treatment period is split into training and testing period is so far the common way to go, although there is no agreed upon value of how low the prediction errors should be, just that lower is always better.

Next it is also argued that examining if the trend in the countries from donor pool that are assigned non-zero weights and seeing if the pre-treatment trend in those donor countries seem to match the pre-treatment trend of treated country similar way as you would do for DiD (Difference in Differences) as argued [here](https://www.urban.org/sites/default/files/publication/89246/the_synthetic_control_method_as_a_tool_0.pdf).

Lastly, as crazy as it may seem, yes visual inspection is actually recommended as well.

Beside the above you should also conduct sensitivity analysis by systematically trying to vary the number of countries in the donor pool and see if the resulting synthetic control has still a good fit. That's again also done by ADH 2015 and you will find it in almost all studies as well. Publishing your weights is also considered good practice.

Of course, doing a proper cross-validation is difficult if you have small pre-treatment sample. I was once working on a synthetic control study at one of the euro-system member's central bank with few colleagues where we had very short pre-treatment period due to data constraints. Instead of doing cross-validation we decided to just test whether the real pre-treatment series is statistically significantly different from the synthetic series using couple of non-parametric tests. When we presented the results and methodology at a workshop no objections to that were raised, and there were some very excellent econometricians present in the room. However, beside central bank's brief the study was not published because the results were not very interesting as placebo test showed results were not significant and we decided to abandon the project since we judged chance of publication to be low so the idea never got proper peer review, so take that as you will.

---

**Response to edit:**

as with classical DiD there is always an issue of potentially delayed effects or anticipation effects that would bias the result. However, in this case the authors argue that the increase after the reunification was actually part of the effect and not delayed effect. As the authors say:

> Our estimate of the effect of the German reunification on per capita GDP in West Germany is given by the
> difference between the actual West Germany and its synthetic version, visualized in Figure 3. We estimate that the
> German reunification did not have much of an effect on
> West German per capita GDP in the first two years immediately following reunification. In this initial period, per
> capita GDP in the synthetic West Germany is even slightly
> lower than in the actual West Germany, which is broadly
> in line with arguments about an initial demand boom
> (see, e.g., Meinbardt et al. 1995). From 1992 onward,
> however, the two lines diverge substantially. While per
> capita GDP growth decelerates in West Germany, for the
> synthetic West Germany per capita GDP keeps ascending
> at a pace similar to that of the prereunification period.

However, arguably this is a weak point of the paper.

Otherwise, if there would be a genuine delay in the effect you would have a problem that could be solved similarly as with DiD by identifying the 'correct' date where the treatment started to have an effect and identify the pre and post treatment period based on that.

> **Comments:**
> **username_0**: Thanks for the quick reply! also, can you provide more details on the non-parametric tests you mentioned?  
> **username_1**: @deeyuan we did wilcoxon rank sum test permutation test and bootstrap - we did only these because our pre-treatment samples were only about 20-24 observations depending on variable, I don’t want to say that doing non-parametric test here is the way to go necessary if you have some nice sample size there some better ways how to compare two time series for sure  

---

