<p>The plm package is for panel data. Therefore there has to be variables in dataset with <strong>(id,time)</strong> to indicate the panel structure. If the argument <em>index</em> is not used when calling an estimation routine such as <em>pgmm()</em> then it is automatically assumed that the first two columns indicates panel structure hence are <strong>(id,time)</strong>.</p>
<p>The error occuring here is duplicates in the panel structure. These can occur if there are NA's in the variables indicating <strong>(id,time)</strong> or they can occur just because some observations are repeated. So do the following:</p>
<ol>
<li>Check for NA's in variables <em>id</em> and <em>time</em> and find some way to handle them by for example removing them (this is not necessarily the best from the point of view of estimation but it will get the code running).</li>
<li>Check for duplicates after having removed NA's and remove these (again removal is not necessarily the best way to handle duplicates from an estimation perspective)</li>
</ol>
<p>Once there are no NA's and no duplicates the error you are experiencing should dissappear.</p>
<p>To remove duplicates use for example the following code</p>
<pre><code>library(data.table)
dt &lt;- as.data.table(dt)
dt[,n:=1]
dt[,count:=cumsum(n),by=.(id,time)]
dt_no_dups &lt;- dt[count==1,]
</code></pre>
<p>However, if you are not using the index argument to indicate which variables are <em>id</em> and <em>time</em> then the <em>pgmm()</em> routine assumes your first two variables are giving the panel structure even if they are not and offcourse there could be duplicates in these even if you have removed them from the variables you know to indicating panel structure. So if you variables are column ordered [id,dependent,time,independent] then <em>pgmm()</em> will look for duplicates in <strong>(id,dependent)</strong> which may or may not throw an error.</p>
<p>Here is R code to simulate a Arellano-Bond model and estimate. The model simulated is
<span class="math-container">$$y_{it} = \rho y_{i,t-1} + x_{it}\beta + \mu_i + \epsilon_{it}.$$</span></p>
<p>If the variable <em>with_duplicates</em> is TRUE then duplicates are added to data before estimation in order to generate the error you are experiencing. The model simulated is</p>
<pre><code>library(data.table)
library(plm)
# simulate simple model
# y_it = \rho y_{i,t-1} + x_{it}^\top\beta + \epsilon_{it}

with_duplicates &lt;- TRUE # if TRUE duplicates are added to data throwing error
T &lt;- 10
N &lt;- 100 
dt &lt;- data.table(id=rep(1:N,each=T),time=rep(1:T,N)) 

# Choose y0 for all N countries
y0 &lt;- rnorm(N)
# Set model parameters
rho &lt;- 0.5
b &lt;- -0.5
mu &lt;- 2*runif(N)    

# simulate epsilon and x
epsilon &lt;- rnorm(N*T)
x &lt;- rnorm(N*T)
z &lt;- b*x + epsilon

index &lt;- 1
y &lt;- rep(0,N*T)
for (i in 1:N)
{
    for (t in 1:T)
    {

        if (t==1) 
            { 
                y[index] &lt;- rho*y0[i] + z[index] + mu[i]
            } else {
                y[index] &lt;- rho*y[index-1] + z[index] + mu[i]
            }
        index &lt;- index + 1
    }
}

dt<span class="math-container">$y &lt;- y
dt$</span>x &lt;- x
dt[,lag_y:=shift(y,1),by=id]

# Add a duplicate observation to generate error

if (with_duplicates) {dt &lt;- rbind(dt,dt[4,])}

model &lt;- pgmm(y ~ lag(y,1) + x| lag(y,2:4),effect=&quot;individual&quot;, model=&quot;twosteps&quot;,data=dt)
lm(y ~ lag_y + x,data=dt)
summary(model)
</code></pre>

CC BY-SA 4.0
2021-05-18T19:55:42.693Z
Thanks a lot! This really helped me. I will look into all of this and I understand what you are saying. Thanks once again.
Lovemydogsxx
CC BY-SA 4.0
2021-05-18T17:33:28.143Z
2021-05-18T17:59:20.033Z
2021-05-18T17:59:20.033Z
Jesper Hybel
<p>I want to use the pgmm function in R with a lag of 1, but I get the following error everytime:</p>
<pre><code>Error in pdim.default(index[[1L]], index[[2L]]) : 
  duplicate couples (id-time)
In addition: Warning message:
In pdata.frame(data, index) :
  duplicate couples (id-time) in resulting pdata.frame
 to find out which, use, e.g., table(index(your_pdataframe), useNA = &quot;ifany&quot;)
</code></pre>
<p>I understand from what I have read that this is because there are multiple year-id combinations, of which some of them are the same. However, I don't understand what the ID column would be in my case and, thus, I don't know how I should merge them (this is what I saw on the internet that would be the solution to the problem).</p>
<p>I run the following code:</p>
<pre><code>growthregressionpgmm &lt;- 
  pgmm(HDI ~ ND + GDPpcap + Fertility_rate + Life_expectancy + GDP_growth + 
         CO2_emissions | lag(ND, 1) + lag(GDPpcap,1) + lag(Fertility_rate, 1) + 
         lag(Life_expectancy, 1) + lag(GDP_growth, 1) + lag(CO2_emissions, 1), 
       data=fulldata, effect=&quot;twoways&quot;, model=&quot;twosteps&quot;)
</code></pre>
<p>Here is a part of my data:</p>
<pre><code>&gt; dput(fulldata)
</code></pre>
<pre><code>structure(list(Year = c(&quot;1990&quot;, &quot;1992&quot;, &quot;1993&quot;, &quot;1991&quot;, &quot;1991&quot;, 
&quot;1992&quot;, &quot;1993&quot;, &quot;1993&quot;, &quot;1993&quot;, &quot;1993&quot;, &quot;1990&quot;, &quot;1990&quot;, &quot;1990&quot;, 
&quot;1991&quot;, &quot;1992&quot;, &quot;1992&quot;, &quot;1993&quot;, &quot;1994&quot;, &quot;1996&quot;, &quot;1996&quot;, &quot;1996&quot;, 
&quot;1997&quot;, &quot;1997&quot;, &quot;1997&quot;, &quot;1997&quot;, &quot;1998&quot;, &quot;1997&quot;, &quot;1998&quot;, &quot;1998&quot;, 
&quot;1998&quot;, &quot;1998&quot;, &quot;1999&quot;, &quot;1999&quot;, &quot;1999&quot;, &quot;2000&quot;, &quot;2000&quot;, &quot;2000&quot;, 
&quot;1998&quot;, &quot;1998&quot;, &quot;1999&quot;, &quot;2001&quot;, &quot;2001&quot;, &quot;2001&quot;, &quot;2002&quot;, &quot;2002&quot;, 
&quot;2002&quot;, &quot;2002&quot;, &quot;2002&quot;, &quot;2002&quot;), `Disaster Type` = c(&quot;Storm&quot;, 
&quot;Storm&quot;, &quot;Storm&quot;, &quot;Storm&quot;, &quot;Landslide&quot;, &quot;Flood&quot;, &quot;Earthquake&quot;, 
&quot;Storm&quot;, &quot;Flood&quot;, &quot;Storm&quot;, &quot;Storm&quot;, &quot;Earthquake&quot;, &quot;Storm&quot;, &quot;Storm&quot;, 
&quot;Storm&quot;, &quot;Storm&quot;, &quot;Storm&quot;, &quot;Volcanic activity&quot;, &quot;Volcanic activity&quot;, 
&quot;Volcanic activity&quot;, &quot;Landslide&quot;, &quot;Drought&quot;, &quot;Storm&quot;, &quot;Storm&quot;, 
&quot;Wildfire&quot;, &quot;Earthquake&quot;, &quot;Storm&quot;, &quot;Storm&quot;, &quot;Storm&quot;, &quot;Drought&quot;, 
&quot;Drought&quot;, &quot;Storm&quot;, &quot;Drought&quot;, &quot;Flood&quot;, &quot;Earthquake&quot;, &quot;Epidemic&quot;, 
&quot;Epidemic&quot;, &quot;Drought&quot;, &quot;Storm&quot;, &quot;Earthquake&quot;, &quot;Earthquake&quot;, &quot;Epidemic&quot;, 
&quot;Storm&quot;, &quot;Earthquake&quot;, &quot;Storm&quot;, &quot;Storm&quot;, &quot;Storm&quot;, &quot;Earthquake&quot;, 
&quot;Volcanic activity&quot;), Country = c(&quot;Fiji&quot;, &quot;Fiji&quot;, &quot;Fiji&quot;, &quot;Marshall Islands (the)&quot;, 
&quot;Papua New Guinea&quot;, &quot;Papua New Guinea&quot;, &quot;Papua New Guinea&quot;, &quot;Papua New Guinea&quot;, 
&quot;Papua New Guinea&quot;, &quot;Solomon Islands&quot;, &quot;Tonga&quot;, &quot;Vanuatu&quot;, &quot;Samoa&quot;, 
&quot;Samoa&quot;, &quot;Vanuatu&quot;, &quot;Vanuatu&quot;, &quot;Vanuatu&quot;, &quot;Papua New Guinea&quot;, 
&quot;Papua New Guinea&quot;, &quot;Papua New Guinea&quot;, &quot;Papua New Guinea&quot;, &quot;Papua New Guinea&quot;, 
&quot;Fiji&quot;, &quot;Papua New Guinea&quot;, &quot;Papua New Guinea&quot;, &quot;Papua New Guinea&quot;, 
&quot;Tonga&quot;, &quot;Tonga&quot;, &quot;Vanuatu&quot;, &quot;Fiji&quot;, &quot;Micronesia (Federated States of)&quot;, 
&quot;Fiji&quot;, &quot;Kiribati&quot;, &quot;Papua New Guinea&quot;, &quot;Papua New Guinea&quot;, &quot;Micronesia (Federated States of)&quot;, 
&quot;Marshall Islands (the)&quot;, &quot;Solomon Islands&quot;, &quot;Tonga&quot;, &quot;Vanuatu&quot;, 
&quot;Papua New Guinea&quot;, &quot;Papua New Guinea&quot;, &quot;Tonga&quot;, &quot;Papua New Guinea&quot;, 
&quot;Micronesia (Federated States of)&quot;, &quot;Solomon Islands&quot;, &quot;Micronesia (Federated States of)&quot;, 
&quot;Papua New Guinea&quot;, &quot;Papua New Guinea&quot;), `Damage-to-GDP` = c(0.00468994375259065, 
0.000726874446693152, 0.0444821683115519, NA, NA, NA, 0.000527417655715239, 
0.000158225296714572, 0.000263708827857619, NA, 0.0102282455341558, 
NA, 0.51022968442747, 0.725915383444357, NA, NA, 0.014519768008217, 
0.0109523857215256, NA, NA, NA, NA, 0.0108773473588107, NA, NA, 
NA, NA, NA, NA, NA, NA, 0.00127934872374966, NA, 0.00438688566628981, 
NA, NA, NA, NA, NA, NA, NA, NA, 0.146837549271082, NA, NA, NA, 
0.00167377438200721, NA, NA), `Affected-people-to-total-population` = c(NA, 
NA, NA, 0.123956697793571, 0.00105807856741241, 0.0186095867906672, 
NA, NA, NA, 0.260674395588859, NA, 1.3645077879282e-05, NA, NA, 
6.44454469291745e-05, 0.00741122639685506, NA, NA, 5.64520135304186e-05, 
0.0002877170956267, 1.50538702747783e-06, 0.0917996981993122, 
NA, 0.00137699547298968, 0.001468795171189, 0.00176772137543665, 
0.0310497935188731, 0.00515293923654052, 0.01348413086349, 0.329254133876227, 
0.26615407363596, NA, 1.01238972183387, NA, 0.000855053692241551, 
0.0319454013891734, 0.00429531259235907, 0.000972630684450451, 
0.0316493527908319, 0.0777866659310954, 3.36422562806829e-05, 
0.0002334873010525, NA, 0.00073295258059158, 0.00163505559189012, 
0.0025491925260431, NA, 0.000163971494539503, 0.00213162942901354
), ND = c(0, 0, 1, 1, 0, 1, 0, 0, 0, 1, 0, 0, 1, 1, 0, 0, 0, 
0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 1, 1, 0, 1, 0, 0, 1, 0, 0, 
1, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0), HDI = c(0.662, 0.672, 0.675, 
NA, 0.389, 0.398, 0.411, 0.411, 0.411, NA, 0.654, NA, 0.633, 
0.634, NA, NA, NA, 0.417, 0.433, 0.433, 0.433, 0.435, 0.687, 
0.435, 0.435, 0.442, 0.675, 0.678, NA, 0.687, NA, 0.692, NA, 
0.445, 0.45, 0.546, NA, NA, 0.678, NA, 0.456, 0.456, 0.679, 0.462, 
0.559, 0.486, 0.559, 0.462, 0.462), CO2_emissions = c(1.11735406060889, 
1.0048343181516, 0.995724293773337, NA, 0.459388934233434, 0.453425890525591, 
0.443088179935911, 0.443088179935911, 0.443088179935911, 0.421240986851407, 
0.810011675730259, 0.450328505249944, 0.630676338888104, 0.648444788624182, 
0.401746471611781, 0.401746471611781, 0.390243139022436, 0.430716741605883, 
0.41194897189593, 0.41194897189593, 0.41194897189593, 0.47330106757541, 
0.999162394542445, 0.47330106757541, 0.47330106757541, 0.513086548656141, 
1.02473633550337, 0.906999752658917, 0.453257822200498, 0.962398629269006, 
1.0844299866923, 0.973254471333977, 0.353565058091886, 0.427890152149318, 
0.455899066725996, 1.16085361538891, NA, 0.366049558092927, 0.906999752658917, 
0.46529114831876, 0.537041714221921, 0.537041714221921, 0.893600170580889, 
0.571820580423017, 1.33619545921704, 0.362125429458561, 1.33619545921704, 
0.571820580423017, 0.571820580423017), GDPpcap = c(2926.57253822332, 
2956.74562527877, 2977.75102732399, NA, 1490.76034619757, 1658.37860177333, 
1915.54964077492, 1915.54964077492, 1915.54964077492, 1739.45172258361, 
2570.98733940999, 2644.90678139038, 2407.69721594666, 2335.19360351875, 
2643.63498560777, 2643.63498560777, 2586.82481239303, 1982.65811460788, 
1968.63071447196, 1968.63071447196, 1968.63071447196, 1845.78398296477, 
3131.2099301146, 1845.78398296477, 1845.78398296477, 1733.21642938823, 
3251.4184493856, 3317.1836235216, 2778.23593092383, 3142.49796380533, 
2528.7225952912, 3392.9763183624, 1648.96379800244, 1723.86945516225, 
1643.08412576665, 2707.56199905807, NA, 1834.61951190637, 3317.1836235216, 
2737.19559474216, 1606.1955026128, 1606.1955026128, 3547.32785335305, 
1571.03933225238, 2791.04965138618, 1256.29997638498, 2791.04965138618, 
1571.03933225238, 1571.03933225238), Fertility_rate = c(3.398, 
3.352, 3.33, NA, 4.756, 4.723, 4.7, 4.7, 4.7, 5.461, 4.644, 4.926, 
5.118, 5.034, 4.841, 4.841, 4.798, 4.683, 4.653, 4.653, 4.653, 
4.632, 3.209, 4.632, 4.632, 4.604, 4.34, 4.3, 4.573, 3.171, 4.471, 
3.132, 4.11, 4.569, 4.525, 4.3, NA, 4.872, 4.3, 4.531, 4.475, 
4.475, 4.236, 4.422, 4.105, 4.606, 4.105, 4.422, 4.422), Life_expectancy = c(65.379, 
65.278, 65.218, NA, 56.823, 57.152, 57.473, 57.473, 57.473, 64.961, 
68.935, 64.721, 66.281, 66.47, 65.349, 65.349, 65.633, 57.781, 
58.344, 58.344, 58.344, 58.594, 65.246, 58.594, 58.594, 58.828, 
69.471, 69.535, 66.899, 65.36, 64.298, 65.512, 62.829, 59.049, 
59.265, 64.55, NA, 66.665, 69.535, 67.134, 59.487, 59.487, 69.725, 
59.722, 64.888, 68.175, 64.888, 59.722, 59.722), CPI = c(49.9250037293591, 
59.4846188641146, 61.8987673569764, NA, NA, NA, NA, NA, NA, 45.2759095332228, 
32.005120180595, 55.8702936919443, 41.5819326210445, 22.8740281444522, 
38.2055963679636, 38.2055963679636, 38.5727156476084, NA, NA, 
NA, NA, NA, 67.6520051713937, NA, NA, NA, NA, NA, 42.9430895548233, 
69.5670319917053, NA, 71.8503332005384, 69.3852858719752, NA, 
NA, NA, NA, 56.7139321922187, NA, 44.8602213817088, 74.0722891566265, 
74.0722891566265, NA, 73.1084337349398, NA, 64.4171779141104, 
NA, 73.1084337349398, 73.1084337349398), GDP_growth = c(5.80000271926605, 
6.10000190987678, 2.13003242138274, NA, 9.54689770861241, 13.8490852689481, 
18.2022859527298, 18.2022859527298, 18.2022859527298, 3.99925595238095, 
-2.04409143450813, 11.6956997985937, -4.42145094868896, -2.30000940480883, 
2.5854137275348, 2.5854137275348, 0.735447995455772, 5.94210905967769, 
7.73369579796399, 7.73369579796399, 7.73369579796399, -3.90438965639359, 
-2.1999993686925, -3.90438965639359, -3.90438965639359, -3.76911321783457, 
1.22344961737785, 2.45875910300609, 1.17685436113621, 1.30000045153811, 
2.8505158599975, 8.79999871927239, -1.53846153846153, 1.85555399408817, 
-2.49484199260023, 4.83390378617111, NA, 1.29870129870129, 2.45875910300609, 
0.337293221894313, -0.121288605564772, -0.121288605564772, 3.74892569298142, 
-0.158900533082658, 0.546997672499344, -2.79654654654654, 0.546997672499344, 
-0.158900533082658, -0.158900533082658), Health_expenditure = c(NA, 
NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, 
NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, 
NA, 189516095.258765, 22580755.7012392, NA, NA, NA, NA, 215735067.486362, 
215735067.486362, 11653482.0216929, 253236754.502794, 23591006.3055499, 
58972354.1993895, 23591006.3055499, 253236754.502794, 253236754.502794
)), row.names = c(NA, -49L), class = c(&quot;tbl_df&quot;, &quot;tbl&quot;, &quot;data.frame&quot;
))
</code></pre>
<p>Can someone provide me with a code that merges id with year such that the regression works? It would really help me because I have been struggling for a month now already with this!...</p>

CC BY-SA 4.0
2021-05-18T15:04:57.113Z
I am willing to take a look at it provided that you provide  (1) What is the economic research question you are trying to answer (2) what is the statical model (write up a model equation) (3) what are the basic features of your data - what does independent and dependent variables measure. As long as this is not provided I personally consider this to be not a question about economics nor economic empirical methods and are as such in my view off topic (I am not a moderator).
Jesper Hybel
CC BY-SA 4.0
2021-05-18T15:06:02.377Z
See this discussion for when R code questions are on topic https://economics.meta.stackexchange.com/questions/1912/are-economics-related-code-requests-on-topic/2130?noredirect=1#comment4176_2130
Jesper Hybel
CC BY-SA 4.0
2021-05-18T15:47:57.640Z
Hey @JesperHybel. Thank you. For question (1), the research question is: "What is the impact of severe natural disasters on poverty in 11 Pacific Island countries?" Question (2): HDI= βND +αX_(it-1 )+ δ_1 FE_i + δ_2 FE_t + ε_it Where HDI = human development index, ND is the natural disaster dummy, X contains the lag of the control variables (GDPpcap + Fertility_rate + Life_expectancy + GDP_growth + 
         CO2_emissions), FE_i are country fixed effects and FE_t are time fixed effects.
Lovemydogsxx
CC BY-SA 4.0
2021-05-18T15:53:39.467Z
@JesperHybel I run the regression plm first and now I want to do a robustness check and for this I need to run this regression with pgmm. For (3), the Dependent variable is a proxy for poverty, namely the HDI. The independent variables are GDP per capita (constant US 2010 $), the fertility rate (births per woman), Life expectancy (at birth, in years), GDP growth (annual %) and CO2 emissions (in metric tons per capita). Hope this helps. Thank you in advance!
Lovemydogsxx
CC BY-SA 4.0
2021-05-18T16:10:35.967Z
I think you should add this to the question itself not in comments. Also I believe your data is then country panel. This means that your model index are given in variables ("Country","Year"). The error you get occurs due to duplicates in this index. So for some countries there are more than 1 observation in a given year. Also unless model index variables are the first two columns I believe you have to provide them as the argument "index" to the estimation call (this goes for all use of plm-package if I remember correctly)
Jesper Hybel
CC BY-SA 4.0
2021-05-18T16:16:30.357Z
Once you take care of this you will get other errors. This is first of all related to the fact that your data is very low quality with very many NA's on several variables. Removing alot of these variables leaves you with a much more basic model. Even with this done I have so scarce data left that I get the error: Fejl i solve.default(crossprod(WX, t(crossprod(WX, A2)))) :   system is computationally singular.
Jesper Hybel
CC BY-SA 4.0
2021-05-18T16:23:14.350Z
I have managed to estimate the following model: pgmm(formula = HDI ~ CO2_emissions + lag(HDI, 1) | lag(HDI, 2), 
    data = dt_new, effect = "individual", model = "twosteps"), but with a warning.
Jesper Hybel
CC BY-SA 4.0
2021-05-18T16:36:25.723Z
However, having removed duplicates and rows with NA's on the varibles entering the model (there are NA's on dependent variable) two countries: Micronesia and Samoa only have two obervations this means that 'lag(HDI,2)' which is the GMM instrument is impossible, which is probably why the warning occurs. Also not that in any case $y_{i,t-2}$ is weak instrument (see Anderson and Hsiao 1981). Furthermore, your country periods are not sequential in time one observation is for example 1990 and then next 1997 this could also influence estimation since lag 1 is then really a lag of seven years.
Jesper Hybel
<r>
Why does pgmm function gives error every time?
Lovemydogsxx
# Why does pgmm function gives error every time?

(Asked by: username_0 on 2021-05-18)

I want to use the pgmm function in R with a lag of 1, but I get the following error everytime:
Error in pdim.default(index[[1L]], index[[2L]]) : 
  duplicate couples (id-time)
In addition: Warning message:
In pdata.frame(data, index) :
  duplicate couples (id-time) in resulting pdata.frame
 to find out which, use, e.g., table(index(your_pdataframe), useNA = "ifany")

I understand from what I have read that this is because there are multiple year-id combinations, of which some of them are the same. However, I don't understand what the ID column would be in my case and, thus, I don't know how I should merge them (this is what I saw on the internet that would be the solution to the problem).
I run the following code:
growthregressionpgmm <- 
  pgmm(HDI ~ ND + GDPpcap + Fertility_rate + Life_expectancy + GDP_growth + 
         CO2_emissions | lag(ND, 1) + lag(GDPpcap,1) + lag(Fertility_rate, 1) + 
         lag(Life_expectancy, 1) + lag(GDP_growth, 1) + lag(CO2_emissions, 1), 
       data=fulldata, effect="twoways", model="twosteps")

Here is a part of my data:
> dput(fulldata)

structure(list(Year = c("1990", "1992", "1993", "1991", "1991", 
"1992", "1993", "1993", "1993", "1993", "1990", "1990", "1990", 
"1991", "1992", "1992", "1993", "1994", "1996", "1996", "1996", 
"1997", "1997", "1997", "1997", "1998", "1997", "1998", "1998", 
"1998", "1998", "1999", "1999", "1999", "2000", "2000", "2000", 
"1998", "1998", "1999", "2001", "2001", "2001", "2002", "2002", 
"2002", "2002", "2002", "2002"), `Disaster Type` = c("Storm", 
"Storm", "Storm", "Storm", "Landslide", "Flood", "Earthquake", 
"Storm", "Flood", "Storm", "Storm", "Earthquake", "Storm", "Storm", 
"Storm", "Storm", "Storm", "Volcanic activity", "Volcanic activity", 
"Volcanic activity", "Landslide", "Drought", "Storm", "Storm", 
"Wildfire", "Earthquake", "Storm", "Storm", "Storm", "Drought", 
"Drought", "Storm", "Drought", "Flood", "Earthquake", "Epidemic", 
"Epidemic", "Drought", "Storm", "Earthquake", "Earthquake", "Epidemic", 
"Storm", "Earthquake", "Storm", "Storm", "Storm", "Earthquake", 
"Volcanic activity"), Country = c("Fiji", "Fiji", "Fiji", "Marshall Islands (the)", 
"Papua New Guinea", "Papua New Guinea", "Papua New Guinea", "Papua New Guinea", 
"Papua New Guinea", "Solomon Islands", "Tonga", "Vanuatu", "Samoa", 
"Samoa", "Vanuatu", "Vanuatu", "Vanuatu", "Papua New Guinea", 
"Papua New Guinea", "Papua New Guinea", "Papua New Guinea", "Papua New Guinea", 
"Fiji", "Papua New Guinea", "Papua New Guinea", "Papua New Guinea", 
"Tonga", "Tonga", "Vanuatu", "Fiji", "Micronesia (Federated States of)", 
"Fiji", "Kiribati", "Papua New Guinea", "Papua New Guinea", "Micronesia (Federated States of)", 
"Marshall Islands (the)", "Solomon Islands", "Tonga", "Vanuatu", 
"Papua New Guinea", "Papua New Guinea", "Tonga", "Papua New Guinea", 
"Micronesia (Federated States of)", "Solomon Islands", "Micronesia (Federated States of)", 
"Papua New Guinea", "Papua New Guinea"), `Damage-to-GDP` = c(0.00468994375259065, 
0.000726874446693152, 0.0444821683115519, NA, NA, NA, 0.000527417655715239, 
0.000158225296714572, 0.000263708827857619, NA, 0.0102282455341558, 
NA, 0.51022968442747, 0.725915383444357, NA, NA, 0.014519768008217, 
0.0109523857215256, NA, NA, NA, NA, 0.0108773473588107, NA, NA, 
NA, NA, NA, NA, NA, NA, 0.00127934872374966, NA, 0.00438688566628981, 
NA, NA, NA, NA, NA, NA, NA, NA, 0.146837549271082, NA, NA, NA, 
0.00167377438200721, NA, NA), `Affected-people-to-total-population` = c(NA, 
NA, NA, 0.123956697793571, 0.00105807856741241, 0.0186095867906672, 
NA, NA, NA, 0.260674395588859, NA, 1.3645077879282e-05, NA, NA, 
6.44454469291745e-05, 0.00741122639685506, NA, NA, 5.64520135304186e-05, 
0.0002877170956267, 1.50538702747783e-06, 0.0917996981993122, 
NA, 0.00137699547298968, 0.001468795171189, 0.00176772137543665, 
0.0310497935188731, 0.00515293923654052, 0.01348413086349, 0.329254133876227, 
0.26615407363596, NA, 1.01238972183387, NA, 0.000855053692241551, 
0.0319454013891734, 0.00429531259235907, 0.000972630684450451, 
0.0316493527908319, 0.0777866659310954, 3.36422562806829e-05, 
0.0002334873010525, NA, 0.00073295258059158, 0.00163505559189012, 
0.0025491925260431, NA, 0.000163971494539503, 0.00213162942901354
), ND = c(0, 0, 1, 1, 0, 1, 0, 0, 0, 1, 0, 0, 1, 1, 0, 0, 0, 
0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 1, 1, 0, 1, 0, 0, 1, 0, 0, 
1, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0), HDI = c(0.662, 0.672, 0.675, 
NA, 0.389, 0.398, 0.411, 0.411, 0.411, NA, 0.654, NA, 0.633, 
0.634, NA, NA, NA, 0.417, 0.433, 0.433, 0.433, 0.435, 0.687, 
0.435, 0.435, 0.442, 0.675, 0.678, NA, 0.687, NA, 0.692, NA, 
0.445, 0.45, 0.546, NA, NA, 0.678, NA, 0.456, 0.456, 0.679, 0.462, 
0.559, 0.486, 0.559, 0.462, 0.462), CO2_emissions = c(1.11735406060889, 
1.0048343181516, 0.995724293773337, NA, 0.459388934233434, 0.453425890525591, 
0.443088179935911, 0.443088179935911, 0.443088179935911, 0.421240986851407, 
0.810011675730259, 0.450328505249944, 0.630676338888104, 0.648444788624182, 
0.401746471611781, 0.401746471611781, 0.390243139022436, 0.430716741605883, 
0.41194897189593, 0.41194897189593, 0.41194897189593, 0.47330106757541, 
0.999162394542445, 0.47330106757541, 0.47330106757541, 0.513086548656141, 
1.02473633550337, 0.906999752658917, 0.453257822200498, 0.962398629269006, 
1.0844299866923, 0.973254471333977, 0.353565058091886, 0.427890152149318, 
0.455899066725996, 1.16085361538891, NA, 0.366049558092927, 0.906999752658917, 
0.46529114831876, 0.537041714221921, 0.537041714221921, 0.893600170580889, 
0.571820580423017, 1.33619545921704, 0.362125429458561, 1.33619545921704, 
0.571820580423017, 0.571820580423017), GDPpcap = c(2926.57253822332, 
2956.74562527877, 2977.75102732399, NA, 1490.76034619757, 1658.37860177333, 
1915.54964077492, 1915.54964077492, 1915.54964077492, 1739.45172258361, 
2570.98733940999, 2644.90678139038, 2407.69721594666, 2335.19360351875, 
2643.63498560777, 2643.63498560777, 2586.82481239303, 1982.65811460788, 
1968.63071447196, 1968.63071447196, 1968.63071447196, 1845.78398296477, 
3131.2099301146, 1845.78398296477, 1845.78398296477, 1733.21642938823, 
3251.4184493856, 3317.1836235216, 2778.23593092383, 3142.49796380533, 
2528.7225952912, 3392.9763183624, 1648.96379800244, 1723.86945516225, 
1643.08412576665, 2707.56199905807, NA, 1834.61951190637, 3317.1836235216, 
2737.19559474216, 1606.1955026128, 1606.1955026128, 3547.32785335305, 
1571.03933225238, 2791.04965138618, 1256.29997638498, 2791.04965138618, 
1571.03933225238, 1571.03933225238), Fertility_rate = c(3.398, 
3.352, 3.33, NA, 4.756, 4.723, 4.7, 4.7, 4.7, 5.461, 4.644, 4.926, 
5.118, 5.034, 4.841, 4.841, 4.798, 4.683, 4.653, 4.653, 4.653, 
4.632, 3.209, 4.632, 4.632, 4.604, 4.34, 4.3, 4.573, 3.171, 4.471, 
3.132, 4.11, 4.569, 4.525, 4.3, NA, 4.872, 4.3, 4.531, 4.475, 
4.475, 4.236, 4.422, 4.105, 4.606, 4.105, 4.422, 4.422), Life_expectancy = c(65.379, 
65.278, 65.218, NA, 56.823, 57.152, 57.473, 57.473, 57.473, 64.961, 
68.935, 64.721, 66.281, 66.47, 65.349, 65.349, 65.633, 57.781, 
58.344, 58.344, 58.344, 58.594, 65.246, 58.594, 58.594, 58.828, 
69.471, 69.535, 66.899, 65.36, 64.298, 65.512, 62.829, 59.049, 
59.265, 64.55, NA, 66.665, 69.535, 67.134, 59.487, 59.487, 69.725, 
59.722, 64.888, 68.175, 64.888, 59.722, 59.722), CPI = c(49.9250037293591, 
59.4846188641146, 61.8987673569764, NA, NA, NA, NA, NA, NA, 45.2759095332228, 
32.005120180595, 55.8702936919443, 41.5819326210445, 22.8740281444522, 
38.2055963679636, 38.2055963679636, 38.5727156476084, NA, NA, 
NA, NA, NA, 67.6520051713937, NA, NA, NA, NA, NA, 42.9430895548233, 
69.5670319917053, NA, 71.8503332005384, 69.3852858719752, NA, 
NA, NA, NA, 56.7139321922187, NA, 44.8602213817088, 74.0722891566265, 
74.0722891566265, NA, 73.1084337349398, NA, 64.4171779141104, 
NA, 73.1084337349398, 73.1084337349398), GDP_growth = c(5.80000271926605, 
6.10000190987678, 2.13003242138274, NA, 9.54689770861241, 13.8490852689481, 
18.2022859527298, 18.2022859527298, 18.2022859527298, 3.99925595238095, 
-2.04409143450813, 11.6956997985937, -4.42145094868896, -2.30000940480883, 
2.5854137275348, 2.5854137275348, 0.735447995455772, 5.94210905967769, 
7.73369579796399, 7.73369579796399, 7.73369579796399, -3.90438965639359, 
-2.1999993686925, -3.90438965639359, -3.90438965639359, -3.76911321783457, 
1.22344961737785, 2.45875910300609, 1.17685436113621, 1.30000045153811, 
2.8505158599975, 8.79999871927239, -1.53846153846153, 1.85555399408817, 
-2.49484199260023, 4.83390378617111, NA, 1.29870129870129, 2.45875910300609, 
0.337293221894313, -0.121288605564772, -0.121288605564772, 3.74892569298142, 
-0.158900533082658, 0.546997672499344, -2.79654654654654, 0.546997672499344, 
-0.158900533082658, -0.158900533082658), Health_expenditure = c(NA, 
NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, 
NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, 
NA, 189516095.258765, 22580755.7012392, NA, NA, NA, NA, 215735067.486362, 
215735067.486362, 11653482.0216929, 253236754.502794, 23591006.3055499, 
58972354.1993895, 23591006.3055499, 253236754.502794, 253236754.502794
)), row.names = c(NA, -49L), class = c("tbl_df", "tbl", "data.frame"
))

Can someone provide me with a code that merges id with year such that the regression works? It would really help me because I have been struggling for a month now already with this!...

**Tags:** r
> **Comments:**
> **username_1**: I am willing to take a look at it provided that you provide  (1) What is the economic research question you are trying to answer (2) what is the statical model (write up a model equation) (3) what are the basic features of your data - what does independent and dependent variables measure. As long as this is not provided I personally consider this to be not a question about economics nor economic empirical methods and are as such in my view off topic (I am not a moderator).  
> **username_1**: See this discussion for when R code questions are on topic https://economics.meta.stackexchange.com/questions/1912/are-economics-related-code-requests-on-topic/2130?noredirect=1#comment4176_2130  
> **username_0**: Hey @JesperHybel. Thank you. For question (1), the research question is: "What is the impact of severe natural disasters on poverty in 11 Pacific Island countries?" Question (2): HDI= βND +αX_(it-1 )+ δ_1 FE_i + δ_2 FE_t + ε_it Where HDI = human development index, ND is the natural disaster dummy, X contains the lag of the control variables (GDPpcap + Fertility_rate + Life_expectancy + GDP_growth +           CO2_emissions), FE_i are country fixed effects and FE_t are time fixed effects.  
> **username_0**: @JesperHybel I run the regression plm first and now I want to do a robustness check and for this I need to run this regression with pgmm. For (3), the Dependent variable is a proxy for poverty, namely the HDI. The independent variables are GDP per capita (constant US 2010 $), the fertility rate (births per woman), Life expectancy (at birth, in years), GDP growth (annual %) and CO2 emissions (in metric tons per capita). Hope this helps. Thank you in advance!  
> **username_1**: I think you should add this to the question itself not in comments. Also I believe your data is then country panel. This means that your model index are given in variables ("Country","Year"). The error you get occurs due to duplicates in this index. So for some countries there are more than 1 observation in a given year. Also unless model index variables are the first two columns I believe you have to provide them as the argument "index" to the estimation call (this goes for all use of plm-package if I remember correctly)  
> **username_1**: Once you take care of this you will get other errors. This is first of all related to the fact that your data is very low quality with very many NA's on several variables. Removing alot of these variables leaves you with a much more basic model. Even with this done I have so scarce data left that I get the error: Fejl i solve.default(crossprod(WX, t(crossprod(WX, A2)))) :   system is computationally singular.  
> **username_1**: I have managed to estimate the following model: pgmm(formula = HDI ~ CO2_emissions + lag(HDI, 1) | lag(HDI, 2),      data = dt_new, effect = "individual", model = "twosteps"), but with a warning.  
> **username_1**: However, having removed duplicates and rows with NA's on the varibles entering the model (there are NA's on dependent variable) two countries: Micronesia and Samoa only have two obervations this means that 'lag(HDI,2)' which is the GMM instrument is impossible, which is probably why the warning occurs. Also not that in any case $y_{i,t-2}$ is weak instrument (see Anderson and Hsiao 1981). Furthermore, your country periods are not sequential in time one observation is for example 1990 and then next 1997 this could also influence estimation since lag 1 is then really a lag of seven years.  

---

## 1 answers

### Answer by username_1 on 2021-05-18. Score: 5 | Accepted answer

The plm package is for panel data. Therefore there has to be variables in dataset with (id,time) to indicate the panel structure. If the argument index is not used when calling an estimation routine such as pgmm() then it is automatically assumed that the first two columns indicates panel structure hence are (id,time).
The error occuring here is duplicates in the panel structure. These can occur if there are NA's in the variables indicating (id,time) or they can occur just because some observations are repeated. So do the following:

Check for NA's in variables id and time and find some way to handle them by for example removing them (this is not necessarily the best from the point of view of estimation but it will get the code running).
Check for duplicates after having removed NA's and remove these (again removal is not necessarily the best way to handle duplicates from an estimation perspective)

Once there are no NA's and no duplicates the error you are experiencing should dissappear.
To remove duplicates use for example the following code
library(data.table)
dt <- as.data.table(dt)
dt[,n:=1]
dt[,count:=cumsum(n),by=.(id,time)]
dt_no_dups <- dt[count==1,]

However, if you are not using the index argument to indicate which variables are id and time then the pgmm() routine assumes your first two variables are giving the panel structure even if they are not and offcourse there could be duplicates in these even if you have removed them from the variables you know to indicating panel structure. So if you variables are column ordered [id,dependent,time,independent] then pgmm() will look for duplicates in (id,dependent) which may or may not throw an error.
Here is R code to simulate a Arellano-Bond model and estimate. The model simulated is
$$y_{it} = \rho y_{i,t-1} + x_{it}\beta + \mu_i + \epsilon_{it}.$$
If the variable with_duplicates is TRUE then duplicates are added to data before estimation in order to generate the error you are experiencing. The model simulated is
library(data.table)
library(plm)
# simulate simple model
# y_it = \rho y_{i,t-1} + x_{it}^\top\beta + \epsilon_{it}

with_duplicates <- TRUE # if TRUE duplicates are added to data throwing error
T <- 10
N <- 100 
dt <- data.table(id=rep(1:N,each=T),time=rep(1:T,N)) 

# Choose y0 for all N countries
y0 <- rnorm(N)
# Set model parameters
rho <- 0.5
b <- -0.5
mu <- 2*runif(N)    

# simulate epsilon and x
epsilon <- rnorm(N*T)
x <- rnorm(N*T)
z <- b*x + epsilon

index <- 1
y <- rep(0,N*T)
for (i in 1:N)
{
    for (t in 1:T)
    {

        if (t==1) 
            { 
                y[index] <- rho*y0[i] + z[index] + mu[i]
            } else {
                y[index] <- rho*y[index-1] + z[index] + mu[i]
            }
        index <- index + 1
    }
}

dt$y <- y
dt$x <- x
dt[,lag_y:=shift(y,1),by=id]

# Add a duplicate observation to generate error

if (with_duplicates) {dt <- rbind(dt,dt[4,])}

model <- pgmm(y ~ lag(y,1) + x| lag(y,2:4),effect="individual", model="twosteps",data=dt)
lm(y ~ lag_y + x,data=dt)
summary(model)

> **Comments:**
> **username_0**: Thanks a lot! This really helped me. I will look into all of this and I understand what you are saying. Thanks once again.  

---

