<p>You get the error because your ID is string and the prodestLP function has a problem with that here - leading to incorrectly specifying that replacement matrix in this case.</p>
<p>You can solve this issue with:</p>
<pre><code> DataA<span class="math-container">$ID&lt;-as.numeric(as.factor( DataA$</span>ID))
</code></pre>
<p>once you declare the ID as numeric the code works:</p>
<pre><code>     mod2 = prodest::prodestLP(DataA<span class="math-container">$va,
+                            fX = DataA$</span>l,
+                            sX = DataA<span class="math-container">$k, 
+                            pX = DataA$</span>m,
+                            idvar = DataA<span class="math-container">$ID,
+                            timevar = DataA$</span>Year,
+                            opt='optim',
+                            exit = FALSE,
+                            tol = 1e-100)  
&gt;  
&gt;  omega = prodest::omega(mod2) 
</code></pre>
<p>with output:</p>
<pre><code>    summary(mod2)

-------------------------------------------------------------
-               Production Function Estimation              -
-------------------------------------------------------------
                   Method :    LP              
-------------------------------------------------------------
                               fX1       sX1 
Estimated Parameters      :   0.107     0.458 
                             (0.012)   (0.052)
-------------------------------------------------------------
N                         :  4397
-------------------------------------------------------------
Bootstrap repetitions     :  20
1st Stage Parameters      :  0.107     0.249 
Optimizer                 :  optim
-------------------------------------------------------------
Elapsed Time              :  0.02 mins

-------------------------------------------------------------


summary(omega)
       V1        
 Min.   :-3.452  
 1st Qu.: 1.428  
 Median : 1.842  
 Mean   : 1.778  
 3rd Qu.: 2.202  
 Max.   : 4.492  
</code></pre>
<hr />
<p>Update these are the packages I am running and code before the one shown above (also I am running R  version 4.0.2 (2020-06-22)):</p>
<pre><code>rm(list = ls())

Data &lt;- read.csv(&quot;~/R studio/excercises in randomness/tfp2/Gmail/LV.csv&quot;)

 View(Data)
 
 
 

 pckg&lt;-c(&quot;plm&quot;,&quot;readxl&quot;,&quot;dplyr&quot;,&quot;ggplot2&quot;, &quot;broom&quot;,&quot;prodest&quot;, &quot;estprod&quot;)
 #install.packages(c(&quot;plm&quot;,&quot;readxl&quot;,&quot;dplyr&quot;,&quot;ggplot2&quot;, &quot;broom&quot;,&quot;prodest&quot;, &quot;estprod&quot;))
 lapply(pckg, require, character.only = TRUE)
 
 head(Data)
 
 summary(Data)
 
 DataA &lt;- Data %&gt;% 
         filter(NACE == 'A') %&gt;% 
         filter(VA &gt; 0, L &gt; 0, K &gt; 0, M &gt; 0) %&gt;%
         mutate(l = log(L),
                va = log(VA),
                k = log(K),
                m = log(M))
</code></pre>

CC BY-SA 4.0
2020-10-02T17:44:18.510Z
1muflon1, thank you a lot for clarification, however, may I ask two more questions - 1. is it somehow possible to display not only floating and state variable but also the intermediate inputs? 2. Do you know how to interpret these t-statistics in brackets - I mean is it possible to display the p-value as well as like in other regressions formula with ***, **, *?
Wadim iLchuk
CC BY-SA 4.0
2020-10-02T18:31:33.723Z
@WadimiLchuk you are welcome, if you think my answer answered the Q consider accepting it. Regarding those further questions 1. you can recover all betas of the model using print(mod2@Model[["FSbetas"]]) but only intercept fX1 and sX1 are labeled. 2. Those values in parentheses are not t-statistics but bootstrapped standard errors you can calculate t-statistics by dividing coefficients by standard errors  ($\beta/SE$). The critical values for t-statistics with large number of observations are approximately 1.65, 1.96 and 2.58 for two sided test for 90,95 and 99 confidence respectively.
1muflon1
CC BY-SA 4.0
2020-10-02T18:44:47.473Z
You can use the built in qt function in R to get precise critical values. There is also build in function for p-value based on t-statistics (the pt built in function)
1muflon1
CC BY-SA 4.0
2020-10-02T20:31:35.447Z
I am so grateful to you, that you helped me to solve my issue. Are also are you a specialist in estprod package with simplier formula? I mean, why there are also only 2 coeffs printed, because as far as I researched to retrieve omega(TFP) from this formula you have to manually code it, but the issue there is the third coefficient that is before Materials variable is not printed out?
Wadim iLchuk
CC BY-SA 4.0
2020-10-02T21:24:27.083Z
@WadimiLchuk I would not call myself estprod package specialist (I am not sure if its even meaningful to say that one is a specialist in this or that package). I have some experience with estimating TFP in R because when I was undergraduate I was also working as a research assistant to a professor who worked in this area. Back then we actually built our own R script to estimate it- its really not that difficult its just like building a lego because you are not programming it from scratch but using pre-existing functions. Also this package seems to be quite new I am not sure it even existed
1muflon1
CC BY-SA 4.0
2020-10-02T21:26:58.990Z
when I was still undergrad. I am not sure why authors decided to exclude it. They probably did not considered it relevant. You can actually ask author of the package directly this is their mail: gabriele.rovigatti@gmail.com
1muflon1
CC BY-SA 4.0
2020-10-03T10:47:57.197Z
Yes, sure, I have already written to the author. But again I want to thank you for your help as well as explanations.
Wadim iLchuk
CC BY-SA 4.0
2020-10-03T10:54:13.050Z
May I answer one more question, while I am trying to make the summary(mod2) the only thing I receive is Length 1, Class prod, Mode S4. Would you be so kind as to elaborate on that
Wadim iLchuk
CC BY-SA 4.0
2020-10-03T10:55:16.230Z
@WadimiLchuk not sure why that is happening but then try to use print(mod2). I will have look at that  later tonight maybe I had some another package installed or I used different command and accidentally typed summary
1muflon1
CC BY-SA 4.0
2020-10-03T10:58:05.500Z
While I try to use simply mod2 or print(mod2), I receive only the first part without reporting N, Bootstrap repetitions, and other stuff - only the production function estimation, Method, and estimated parameters
Wadim iLchuk
CC BY-SA 4.0
2020-10-03T13:30:26.627Z
@WadimiLchuk I tested it and summary(mod2) gives me the output I showed you so I dont know where the problem is. I updated the answer with full r-script try to run it - maybe you have some package active - also make sure you have the same version. Otherwise post it as a separate problem on stack overflow
1muflon1
CC BY-SA 4.0
2020-10-02T12:44:55.760Z
2020-10-03T13:32:38.073Z
2020-10-03T13:32:38.073Z
1muflon1
<p>I am writing my Bachelor Thesis and I really need help with the TFP estimation. so far I have a dataset with log values of Value added (va), Labour (l), Capital (k), and Materials (m). The initial dataset is available here:
<a href="https://drive.google.com/file/d/1aedWYABus1fQjKWxkOmYOmxv-qSja7hF/view?usp=sharing" rel="nofollow noreferrer">https://drive.google.com/file/d/1aedWYABus1fQjKWxkOmYOmxv-qSja7hF/view?usp=sharing</a></p>
<p>So far my code is:</p>
<pre><code>remove(list=ls())

library(plm)
library(dplyr)
library(ggplot2)
library(prodest)
library(estprod)
library(broom)

setwd(&quot;C:/Users/vadya/Desktop/baka&quot;)


Data &lt;- read.csv(&quot;LV.csv&quot;, header=TRUE, sep=&quot;,&quot;)
str(Data)

summary(Data)

DataA &lt;- Data %&gt;% 
  filter(NACE == 'A') %&gt;% 
  filter(VA &gt; 0, L &gt; 0, K &gt; 0, M &gt; 0) %&gt;%
  mutate(l = log(L),
         va = log(VA),
         k = log(K),
         m = log(M))

wooldridge(data = DataA, va ~ l | k | m, id = &quot;ID&quot;, time = &quot;Year&quot;, bootstrap = TRUE, gross = FALSE)


levinsohn_petrin(data = DataA, va ~ l | k | m, id = &quot;ID&quot;, time = &quot;Year&quot;, bootstrap = TRUE, gross = FALSE)


olley_pakes(data = DataA, va ~ l | k | m, id = &quot;ID&quot;, time = &quot;Year&quot;, bootstrap = TRUE, gross = FALSE)

mod1 = estprod::levinsohn_petrin(data = DataA,
                                 formula = va ~ l | k | m, 
                                 id = &quot;ID&quot;, 
                                 time = &quot;Year&quot;, 
                                 reps = 20,
                                 gross = FALSE) 
mod1
  
mod2 = prodest::prodestLP(DataA<span class="math-container">$va,
                      fX = DataA$</span>l,
                          sX = DataA<span class="math-container">$k, 
                      pX = DataA$</span>m,
                          idvar = DataA<span class="math-container">$ID,
                      timevar = DataA$</span>Year,
                          opt='optim',
                          exit = FALSE,
                          tol = 1e-100)  

omega = prodest::omega(mod2) 
</code></pre>
<p>So far, the problem is the following - with estprod package I get only coefficients estimated for l and k, without m
<a href="https://i.sstatic.net/FLRHC.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/FLRHC.png" alt="estprod result" /></a></p>
<p>With prodest package, the problem is the following:
<a href="https://i.sstatic.net/FeSGI.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/FeSGI.png" alt="prodest result" /></a></p>
<p>A saw the same discussion in another theme on StackExchange where a guy was asking the same question, but he had provided some part of his data, and everything worked. But in my case the problems are different.</p>
<p>Does anyone faced the same problem, and is it possible to solve these issues, since I only have begun studying TFP in R, so I would really appreciate your help and any impact provided. Thanks in advance!!!</p>

CC BY-SA 4.0
2020-10-02T00:29:18.433Z
Not an answer, but somewhere in your code, you are trying to join two data frames of different sizes. You find that, you find the problem
Jamzy
CC BY-SA 4.0
2020-10-02T08:45:05.803Z
Yes, I do understand that the error in prodest package says that, but the dataset is the size that it is, so I basically use the same dataset (with the same variables) as the author of prodest function did
Wadim iLchuk
<production-function><productivity><r>
TFP estimation in R by using prodest and estprod packages
Wadim iLchuk
# TFP estimation in R by using prodest and estprod packages

(Asked by: username_0 on 2020-10-01)

I am writing my Bachelor Thesis and I really need help with the TFP estimation. so far I have a dataset with log values of Value added (va), Labour (l), Capital (k), and Materials (m). The initial dataset is available here:
https://drive.google.com/file/d/1aedWYABus1fQjKWxkOmYOmxv-qSja7hF/view?usp=sharing
So far my code is:
remove(list=ls())

library(plm)
library(dplyr)
library(ggplot2)
library(prodest)
library(estprod)
library(broom)

setwd("C:/Users/vadya/Desktop/baka")


Data <- read.csv("LV.csv", header=TRUE, sep=",")
str(Data)

summary(Data)

DataA <- Data %>% 
  filter(NACE == 'A') %>% 
  filter(VA > 0, L > 0, K > 0, M > 0) %>%
  mutate(l = log(L),
         va = log(VA),
         k = log(K),
         m = log(M))

wooldridge(data = DataA, va ~ l | k | m, id = "ID", time = "Year", bootstrap = TRUE, gross = FALSE)


levinsohn_petrin(data = DataA, va ~ l | k | m, id = "ID", time = "Year", bootstrap = TRUE, gross = FALSE)


olley_pakes(data = DataA, va ~ l | k | m, id = "ID", time = "Year", bootstrap = TRUE, gross = FALSE)

mod1 = estprod::levinsohn_petrin(data = DataA,
                                 formula = va ~ l | k | m, 
                                 id = "ID", 
                                 time = "Year", 
                                 reps = 20,
                                 gross = FALSE) 
mod1
  
mod2 = prodest::prodestLP(DataA$va,
                      fX = DataA$l,
                          sX = DataA$k, 
                      pX = DataA$m,
                          idvar = DataA$ID,
                      timevar = DataA$Year,
                          opt='optim',
                          exit = FALSE,
                          tol = 1e-100)  

omega = prodest::omega(mod2) 

So far, the problem is the following - with estprod package I get only coefficients estimated for l and k, without m

With prodest package, the problem is the following:

A saw the same discussion in another theme on StackExchange where a guy was asking the same question, but he had provided some part of his data, and everything worked. But in my case the problems are different.
Does anyone faced the same problem, and is it possible to solve these issues, since I only have begun studying TFP in R, so I would really appreciate your help and any impact provided. Thanks in advance!!!

**Tags:** production-function, productivity, r
> **Comments:**
> **username_1**: Not an answer, but somewhere in your code, you are trying to join two data frames of different sizes. You find that, you find the problem  
> **username_0**: Yes, I do understand that the error in prodest package says that, but the dataset is the size that it is, so I basically use the same dataset (with the same variables) as the author of prodest function did  

---

## 1 answers

### Answer by username_2 on 2020-10-02. Score: 1 | Accepted answer

You get the error because your ID is string and the prodestLP function has a problem with that here - leading to incorrectly specifying that replacement matrix in this case.
You can solve this issue with:
 DataA$ID<-as.numeric(as.factor( DataA$ID))

once you declare the ID as numeric the code works:
     mod2 = prodest::prodestLP(DataA$va,
+                            fX = DataA$l,
+                            sX = DataA$k, 
+                            pX = DataA$m,
+                            idvar = DataA$ID,
+                            timevar = DataA$Year,
+                            opt='optim',
+                            exit = FALSE,
+                            tol = 1e-100)  
>  
>  omega = prodest::omega(mod2) 

with output:
    summary(mod2)

-------------------------------------------------------------
-               Production Function Estimation              -
-------------------------------------------------------------
                   Method :    LP              
-------------------------------------------------------------
                               fX1       sX1 
Estimated Parameters      :   0.107     0.458 
                             (0.012)   (0.052)
-------------------------------------------------------------
N                         :  4397
-------------------------------------------------------------
Bootstrap repetitions     :  20
1st Stage Parameters      :  0.107     0.249 
Optimizer                 :  optim
-------------------------------------------------------------
Elapsed Time              :  0.02 mins

-------------------------------------------------------------


summary(omega)
       V1        
 Min.   :-3.452  
 1st Qu.: 1.428  
 Median : 1.842  
 Mean   : 1.778  
 3rd Qu.: 2.202  
 Max.   : 4.492  


Update these are the packages I am running and code before the one shown above (also I am running R  version 4.0.2 (2020-06-22)):
rm(list = ls())

Data <- read.csv("~/R studio/excercises in randomness/tfp2/Gmail/LV.csv")

 View(Data)
 
 
 

 pckg<-c("plm","readxl","dplyr","ggplot2", "broom","prodest", "estprod")
 #install.packages(c("plm","readxl","dplyr","ggplot2", "broom","prodest", "estprod"))
 lapply(pckg, require, character.only = TRUE)
 
 head(Data)
 
 summary(Data)
 
 DataA <- Data %>% 
         filter(NACE == 'A') %>% 
         filter(VA > 0, L > 0, K > 0, M > 0) %>%
         mutate(l = log(L),
                va = log(VA),
                k = log(K),
                m = log(M))

> **Comments:**
> **username_0**: 1muflon1, thank you a lot for clarification, however, may I ask two more questions - 1. is it somehow possible to display not only floating and state variable but also the intermediate inputs? 2. Do you know how to interpret these t-statistics in brackets - I mean is it possible to display the p-value as well as like in other regressions formula with ***, **, *?  
> **username_2**: @WadimiLchuk you are welcome, if you think my answer answered the Q consider accepting it. Regarding those further questions 1. you can recover all betas of the model using print(mod2@Model[["FSbetas"]]) but only intercept fX1 and sX1 are labeled. 2. Those values in parentheses are not t-statistics but bootstrapped standard errors you can calculate t-statistics by dividing coefficients by standard errors  ($\beta/SE$). The critical values for t-statistics with large number of observations are approximately 1.65, 1.96 and 2.58 for two sided test for 90,95 and 99 confidence respectively.  
> **username_2**: You can use the built in qt function in R to get precise critical values. There is also build in function for p-value based on t-statistics (the pt built in function)  
> **username_0**: I am so grateful to you, that you helped me to solve my issue. Are also are you a specialist in estprod package with simplier formula? I mean, why there are also only 2 coeffs printed, because as far as I researched to retrieve omega(TFP) from this formula you have to manually code it, but the issue there is the third coefficient that is before Materials variable is not printed out?  
> **username_2**: @WadimiLchuk I would not call myself estprod package specialist (I am not sure if its even meaningful to say that one is a specialist in this or that package). I have some experience with estimating TFP in R because when I was undergraduate I was also working as a research assistant to a professor who worked in this area. Back then we actually built our own R script to estimate it- its really not that difficult its just like building a lego because you are not programming it from scratch but using pre-existing functions. Also this package seems to be quite new I am not sure it even existed  
> **username_2**: when I was still undergrad. I am not sure why authors decided to exclude it. They probably did not considered it relevant. You can actually ask author of the package directly this is their mail: gabriele.rovigatti@gmail.com  
> **username_0**: Yes, sure, I have already written to the author. But again I want to thank you for your help as well as explanations.  
> **username_0**: May I answer one more question, while I am trying to make the summary(mod2) the only thing I receive is Length 1, Class prod, Mode S4. Would you be so kind as to elaborate on that  
> **username_2**: @WadimiLchuk not sure why that is happening but then try to use print(mod2). I will have look at that  later tonight maybe I had some another package installed or I used different command and accidentally typed summary  
> **username_0**: While I try to use simply mod2 or print(mod2), I receive only the first part without reporting N, Bootstrap repetitions, and other stuff - only the production function estimation, Method, and estimated parameters  
> **username_2**: @WadimiLchuk I tested it and summary(mod2) gives me the output I showed you so I dont know where the problem is. I updated the answer with full r-script try to run it - maybe you have some package active - also make sure you have the same version. Otherwise post it as a separate problem on stack overflow  

---

