<p>Let us assume a balanced panel of companies <span class="math-container">$i=1,...,N$</span> observed in time periods <span class="math-container">$t=1,...,T$</span>.</p>
<p>Let <span class="math-container">$s_{it}$</span> denote R&amp;D expenses for company <span class="math-container">$i$</span> time <span class="math-container">$t$</span> and let</p>
<p><span class="math-container">$$\bar s_t :=\frac{1}{N} \sum_i s_{it},$$</span></p>
<p>denote the time <span class="math-container">$t$</span> average.</p>
<p>Consider now a simple regression model
<span class="math-container">$$(1)\ \ s_{it} = \beta_0 + \beta_1I[t\geq 2016] + e_{it}$$</span>
where <span class="math-container">$I[t\geq 2016]$</span> is indicator for the time being after policy implementation. I assume the policy is intended to stimulate R&amp;D investments by companies and hence we assume <span class="math-container">$\beta_1&gt;0$</span> if the policy has effect.</p>
<p>Using the normal equations for POOLED OLS estimator it is easy to show that</p>
<p><span class="math-container">$$\hat \beta_0 = \frac{1}{NT_1} \sum_i \sum_t s_{it}I[t&lt; 2016],$$</span>
where <span class="math-container">$T_1$</span> is number of time periods before treatment. Hence <span class="math-container">$\hat \beta_1$</span> is roughly the average R&amp;D expense level BEFORE treatment/policy implementation.</p>
<p>If model (1) is true then <span class="math-container">$plim \ \ \hat \beta_0 = \beta_0 = \mathbb E[s_{it} \lvert t&lt;2016]$</span> which is the expected level of R&amp;D expenses for any company for any year before treatment.</p>
<p>You can also show that</p>
<p><span class="math-container">$$\hat \beta_1 = \frac{1}{NT_2} \sum_i s_{it} I[t \geq 2016]-\frac{1}{NT_1} \sum_i \sum_t s_{it}I[t&lt; 2016]$$</span></p>
<p>If model (1) is true then <span class="math-container">$plim \ \ \hat \beta_1 = \beta_1 = \mathbb E[s_{it} \lvert t\geq 2016]- \mathbb E[s_{it} \lvert t&lt;2016]$</span> which is the expected level of R&amp;D expenses for any company for any year AFTER treatment.</p>
<p>You could plot the series <span class="math-container">$\{s_t\}_{t=1}^T$</span> and get - if model (1) is true - something like this</p>
<p><a href="https://i.sstatic.net/P27WF.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/P27WF.jpg" alt="model 1 - plot" /></a></p>
<p>However something else may be happening over time and while you do not have any control group of companies not affected by the policy implementation you can still potentially do better than estimate model (1).</p>
<p>Consider the following plot</p>
<p><a href="https://i.sstatic.net/BzIf0.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/BzIf0.jpg" alt="enter image description here" /></a></p>
<p>where in each time period going from <span class="math-container">$s_t$</span> to <span class="math-container">$s_{t+1}$</span> there is a small increase in R&amp;D let us call it <span class="math-container">$\lambda$</span>. Then in the year of treatment the increase is slightly smaller.</p>
<p>The blue and read line are found using the estimators above and it is clear that because they do not take the time trend into consideration they overestimate the effect. The data used for the plot was generated after the model</p>
<p><span class="math-container">$$s_{it} = \beta_0 + \lambda \cdot t + \beta_1 I[t\geq 2016] + e_{it},$$</span>
which includes the time trend.</p>
<p>Here in this model the effect is immediate. But the effect could also happen in the years after the treatment instead of only in the year of treatment hence you could have a model like</p>
<p><span class="math-container">$$s_{it} = \beta_0 + \lambda \cdot t + \beta_1 I[t\geq 2016] + \beta_2 I[
t\geq 2017]+ e_{it}.$$</span></p>
<p><strong>The take away is that you should do as the professor says and calculate the time averages <span class="math-container">$s_t$</span> and plot them and look at them. Paying particular attention to what goes on around in the year treatment and in the years close to but still after treatment (based on the principle that effect comes after cause and takes some time be fully realized)</strong></p>
<p>I R to simulate data and generate the plot by means of the following lines of code:</p>
<pre><code># simulate panel of RD
library(data.table)
set.seed(1)
N &lt;- 50 #Number of firms
T &lt;- 16     #Number of time periods
TT &lt;- 9 #policy implementation year (treatment year)

b_0 &lt;- 5
b_1 &lt;- 2

# Simulate model (1)
s &lt;- b_0 + b_1 * (1:T &gt;= TT) + rnorm(N*T)

# Make dataset 
id &lt;- rep(1:N,each=T)
time &lt;- rep(1:T,N)
I &lt;- as.numeric(time&gt;=TT)
mydata &lt;- data.table(id,time,s,I)

# Compute time averages
time_agg &lt;- mydata[,.(s_t=mean(s)),by=time]

#Plot time averages
plot(time_agg$s_t,ylim=c(0,10),pch=20,xlab=&quot;time&quot;,ylab=&quot;R&amp;D&quot;)
points(1:TT,rep(b_0,TT),type=&quot;l&quot;,lwd=1,col=&quot;blue&quot;)
points((TT):T,rep(b_0+b_1,T-TT+1),type=&quot;l&quot;,lwd=1,col=&quot;red&quot;)
savePlot(&quot;plot1.jpg&quot;,type=&quot;jpg&quot;)





# Simulate model (2)
s &lt;- b_0 + b_1 * (1:T &gt;= TT) + 0.5*1:T + rnorm(N*T)

# Make dataset 
mydata &lt;- data.table(id,time,s,I)

# Compute time averages
time_agg &lt;- mydata[,.(s_t=mean(s)),by=time]
model &lt;- lm(s~I,data=mydata)
hat_b_0 &lt;- coef(model)[1]
hat_b_1 &lt;- coef(model)[2]

#Plot time averages
plot(time_agg$s_t,ylim=c(5,16),pch=20,xlab=&quot;time&quot;,ylab=&quot;R&amp;D&quot;)
points(1:TT,rep(hat_b_0,TT),type=&quot;l&quot;,lwd=1,col=&quot;blue&quot;)
points((TT):T,rep(hat_b_0+hat_b_1,T-TT+1),type=&quot;l&quot;,lwd=1,col=&quot;red&quot;)
savePlot(&quot;plot2.jpg&quot;,type=&quot;jpg&quot;)
</code></pre>

CC BY-SA 4.0
2020-09-16T00:45:11.277Z
2020-09-16T00:45:11.277Z
Jesper Hybel
<p>I supose you are expected to compute some kind of regression showwing how (if) changes in R&amp;D expenses effect changes in sales in a certain temporal lapse.</p>

CC BY-SA 4.0
2020-09-14T12:17:06.363Z
2020-09-14T12:17:06.363Z
escaiguolquer
<p>The big problem is identification. You don't know if the change in R&amp;D was due to the policy change or to other changes that happened around the same time (e.g. discovery of <strong>INSERT_SOMETHING_THAT_WAS_DISCOVERED_IN_2016</strong>).</p>
<p>Suppose you have a counterfactual you think makes sense... let's say the policy was implemented to only companies whose names start with an &quot;A&quot;, then you could look at how these companies perform differently than before (say, R&amp;D spending in 2017 compared to 2015) <em>compared to</em> how other companies do on the same measure. If treatment was random, then you could attribute the difference of the difference to the policy. Read more <a href="https://en.wikipedia.org/wiki/Difference_in_differences" rel="nofollow noreferrer">here</a>.</p>
<p>You could also think about using the <em>growth</em> of R&amp;D before- and after-policy, i.e. growth of R&amp;D spending from 2016–17 compared to growth of R&amp;D from 2015–16, compared to other companies. Read more about this (sometimes called diff-in-diff-in-diff) <a href="https://stats.stackexchange.com/questions/99140/difference-in-difference-in-differences-estimator">here</a>. Without details given, it's hard to say which would be more suitable.</p>
<p>As I said, the challenge is identification. It has hardly happened that policy was applied randomly (unless the policymakers are really into knowing the effects of their policy). Suppose the policy was applied to one state but not the other, using R&amp;D for firms in another state as control might confound effects from <em>other changes</em> that also happen in the states around the same time as well.</p>

CC BY-SA 4.0
2020-09-15T09:59:21.383Z
2020-09-15T09:59:21.383Z
Art
<p>I am assuming your challenge is to establish a claim of causality between the policy and the R&amp;D/sales metrics. Maybe you can replicate some version of impact assessment studies. You will need data for before and after policy change. But this method will need a control group. Since you are trying to do an ex-post analysis, this might be difficult.</p>
<p>Alternatively, you can use scenario analysis. With this, you will be comparing two scenarios: with and without policy change. The former is your intervention scenario whereas the latter is your baseline scenario. For the baseline scenario, you will need to decide if there might be any other policy that will be implemented or continued.</p>
<p>You might also need to consider how different policies interact with each other. For example, some policies are complementary and enhance the outcomes for both. Some are antagonistic and cancel each other out. And some are redundant. Moreover, sometimes there is a time lag between policy implementation and its effects.</p>

CC BY-SA 4.0
2020-09-18T13:14:10.070Z
2020-09-18T13:14:10.070Z
Drashti Shah
<p>I have some data for medical R&amp;D and sales, my professor has asked me to check if a policy implemented in 2016 has made any effects on R&amp;D and R&amp;D/Sales data.
I tried to plot growth rates over years, but he says we don't need growth rates, and it has to be solved just by mean R&amp;D expenditures each year. (Mean over multiple companies). How do I go about significance testing of policy change?</p>

CC BY-SA 4.0
2020-09-14T10:50:55.407Z
There really is not enough information to answer, but you might consider: (a) what was the policy aiming to achieve? (b) what happened? (c) would it have happened anyway? Perhaps you are not expected to answer (a)
Henry
<econometrics><applied-econometrics><policy>
How do I test if a given policy was successful?
Hardik
# How do I test if a given policy was successful?

(Asked by: username_0 on 2020-09-14)

I have some data for medical R&D and sales, my professor has asked me to check if a policy implemented in 2016 has made any effects on R&D and R&D/Sales data.
I tried to plot growth rates over years, but he says we don't need growth rates, and it has to be solved just by mean R&D expenditures each year. (Mean over multiple companies). How do I go about significance testing of policy change?

**Tags:** econometrics, applied-econometrics, policy
> **Comments:**
> **username_1**: There really is not enough information to answer, but you might consider: (a) what was the policy aiming to achieve? (b) what happened? (c) would it have happened anyway? Perhaps you are not expected to answer (a)  

---

## 4 answers

### Answer by username_2 on 2020-09-16. Score: 2

Let us assume a balanced panel of companies $i=1,...,N$ observed in time periods $t=1,...,T$.
Let $s_{it}$ denote R&D expenses for company $i$ time $t$ and let
$$\bar s_t :=\frac{1}{N} \sum_i s_{it},$$
denote the time $t$ average.
Consider now a simple regression model
$$(1)\ \ s_{it} = \beta_0 + \beta_1I[t\geq 2016] + e_{it}$$
where $I[t\geq 2016]$ is indicator for the time being after policy implementation. I assume the policy is intended to stimulate R&D investments by companies and hence we assume $\beta_1>0$ if the policy has effect.
Using the normal equations for POOLED OLS estimator it is easy to show that
$$\hat \beta_0 = \frac{1}{NT_1} \sum_i \sum_t s_{it}I[t< 2016],$$
where $T_1$ is number of time periods before treatment. Hence $\hat \beta_1$ is roughly the average R&D expense level BEFORE treatment/policy implementation.
If model (1) is true then $plim \ \ \hat \beta_0 = \beta_0 = \mathbb E[s_{it} \lvert t<2016]$ which is the expected level of R&D expenses for any company for any year before treatment.
You can also show that
$$\hat \beta_1 = \frac{1}{NT_2} \sum_i s_{it} I[t \geq 2016]-\frac{1}{NT_1} \sum_i \sum_t s_{it}I[t< 2016]$$
If model (1) is true then $plim \ \ \hat \beta_1 = \beta_1 = \mathbb E[s_{it} \lvert t\geq 2016]- \mathbb E[s_{it} \lvert t<2016]$ which is the expected level of R&D expenses for any company for any year AFTER treatment.
You could plot the series $\{s_t\}_{t=1}^T$ and get - if model (1) is true - something like this

However something else may be happening over time and while you do not have any control group of companies not affected by the policy implementation you can still potentially do better than estimate model (1).
Consider the following plot

where in each time period going from $s_t$ to $s_{t+1}$ there is a small increase in R&D let us call it $\lambda$. Then in the year of treatment the increase is slightly smaller.
The blue and read line are found using the estimators above and it is clear that because they do not take the time trend into consideration they overestimate the effect. The data used for the plot was generated after the model
$$s_{it} = \beta_0 + \lambda \cdot t + \beta_1 I[t\geq 2016] + e_{it},$$
which includes the time trend.
Here in this model the effect is immediate. But the effect could also happen in the years after the treatment instead of only in the year of treatment hence you could have a model like
$$s_{it} = \beta_0 + \lambda \cdot t + \beta_1 I[t\geq 2016] + \beta_2 I[
t\geq 2017]+ e_{it}.$$
The take away is that you should do as the professor says and calculate the time averages $s_t$ and plot them and look at them. Paying particular attention to what goes on around in the year treatment and in the years close to but still after treatment (based on the principle that effect comes after cause and takes some time be fully realized)
I R to simulate data and generate the plot by means of the following lines of code:
# simulate panel of RD
library(data.table)
set.seed(1)
N <- 50 #Number of firms
T <- 16     #Number of time periods
TT <- 9 #policy implementation year (treatment year)

b_0 <- 5
b_1 <- 2

# Simulate model (1)
s <- b_0 + b_1 * (1:T >= TT) + rnorm(N*T)

# Make dataset 
id <- rep(1:N,each=T)
time <- rep(1:T,N)
I <- as.numeric(time>=TT)
mydata <- data.table(id,time,s,I)

# Compute time averages
time_agg <- mydata[,.(s_t=mean(s)),by=time]

#Plot time averages
plot(time_agg$s_t,ylim=c(0,10),pch=20,xlab="time",ylab="R&D")
points(1:TT,rep(b_0,TT),type="l",lwd=1,col="blue")
points((TT):T,rep(b_0+b_1,T-TT+1),type="l",lwd=1,col="red")
savePlot("plot1.jpg",type="jpg")





# Simulate model (2)
s <- b_0 + b_1 * (1:T >= TT) + 0.5*1:T + rnorm(N*T)

# Make dataset 
mydata <- data.table(id,time,s,I)

# Compute time averages
time_agg <- mydata[,.(s_t=mean(s)),by=time]
model <- lm(s~I,data=mydata)
hat_b_0 <- coef(model)[1]
hat_b_1 <- coef(model)[2]

#Plot time averages
plot(time_agg$s_t,ylim=c(5,16),pch=20,xlab="time",ylab="R&D")
points(1:TT,rep(hat_b_0,TT),type="l",lwd=1,col="blue")
points((TT):T,rep(hat_b_0+hat_b_1,T-TT+1),type="l",lwd=1,col="red")
savePlot("plot2.jpg",type="jpg")


---

### Answer by username_3 on 2020-09-14. Score: 0

I supose you are expected to compute some kind of regression showwing how (if) changes in R&D expenses effect changes in sales in a certain temporal lapse.


---

### Answer by username_4 on 2020-09-15. Score: 0

The big problem is identification. You don't know if the change in R&D was due to the policy change or to other changes that happened around the same time (e.g. discovery of **INSERT\_SOMETHING\_THAT\_WAS\_DISCOVERED\_IN\_2016**).

Suppose you have a counterfactual you think makes sense... let's say the policy was implemented to only companies whose names start with an "A", then you could look at how these companies perform differently than before (say, R&D spending in 2017 compared to 2015) *compared to* how other companies do on the same measure. If treatment was random, then you could attribute the difference of the difference to the policy. Read more [here](https://en.wikipedia.org/wiki/Difference_in_differences).

You could also think about using the *growth* of R&D before- and after-policy, i.e. growth of R&D spending from 2016–17 compared to growth of R&D from 2015–16, compared to other companies. Read more about this (sometimes called diff-in-diff-in-diff) [here](https://stats.stackexchange.com/questions/99140/difference-in-difference-in-differences-estimator). Without details given, it's hard to say which would be more suitable.

As I said, the challenge is identification. It has hardly happened that policy was applied randomly (unless the policymakers are really into knowing the effects of their policy). Suppose the policy was applied to one state but not the other, using R&D for firms in another state as control might confound effects from *other changes* that also happen in the states around the same time as well.


---

### Answer by username_5 on 2020-09-18. Score: 0

I am assuming your challenge is to establish a claim of causality between the policy and the R&D/sales metrics. Maybe you can replicate some version of impact assessment studies. You will need data for before and after policy change. But this method will need a control group. Since you are trying to do an ex-post analysis, this might be difficult.

Alternatively, you can use scenario analysis. With this, you will be comparing two scenarios: with and without policy change. The former is your intervention scenario whereas the latter is your baseline scenario. For the baseline scenario, you will need to decide if there might be any other policy that will be implemented or continued.

You might also need to consider how different policies interact with each other. For example, some policies are complementary and enhance the outcomes for both. Some are antagonistic and cancel each other out. And some are redundant. Moreover, sometimes there is a time lag between policy implementation and its effects.


---

