<p>I'm currently tasked in my job to design a Prospect Theory (here CPT-)index as mentioned in Barberis, Mukherjee and Wang (2016): &quot;Prospect Theory and Stock Returns: An Empirical Test&quot;, see <a href="https://nicholasbarberis.github.io/bmw17b.pdf" rel="nofollow noreferrer">https://nicholasbarberis.github.io/bmw17b.pdf</a></p>
<p>I wrote the CPT function in C# and got a bunch of stocks incl. their historical prices, which i fed into my analytical engine. While running the program, I noted that the ranking of high/low CPT (cumulative prospect theory) stocks varies wildly with the degree of risk sensitivity, loss aversion and the decision weighting parameter.</p>
<p>BMW (2016) used standard parameter values in their paper but didn't elaborate much on the sensitivity of their CPT-index wrt. the different feasible parameter settings.</p>
<p>My idea is now to circumvent the issue of finding the &quot;right/plausible&quot; CPT parameters and to &quot;calibrate&quot; the parameters used in my CPT-index using in-sample data wrt. the best fit as in BMW (2016) via gridsearch: I want to maximize the significance of the (negative) coefficient in my ts-regression. However, I doubt whether this is the right way to do things as it smells a bit like overfitting and p-hacking to me (although i don't try to increase the predicitive power of the CPT-index..).</p>
<p>All those econometricians among you: Is this a permissible way to get my prospect theory parameters and what might be the issues with this?</p>
<p>I appreciate your help and suggestions, I'm grateful for any comments
Thomas</p>
<p>PS: This is not a homework or thesis task but something i need for my job..
PPS: Corrected the paper reference (wrong autor, It's Baolian Wang, not Wang Baolian as you refer to the familiy name first in China (my chinese wife standing next to me insists i clearify it here:-D )</p>

CC BY-SA 4.0
2022-08-31T09:35:00.403Z
The title question in its generality is interesting by itself!
Richard Hardy
CC BY-SA 4.0
2022-08-31T09:37:13.283Z
@RichardHardy Yes, agreed. In my case unfortunately, it's not only academic interest that motivates the question.. :-/
T123
CC BY-SA 4.0
2022-08-31T11:23:02.023Z
I did not mean to imply academic interest alone. Applications are relevant and important by themselves.
Richard Hardy
CC BY-SA 4.0
2022-08-31T11:39:59.267Z
The proposed approach smells a lot like p-hacking to me. Why would the metric to optimize be the direction of the coefficient, let alone the significance? The result of the t-test is anyway invalid because it is conditional on your optimization procedure. It would be more intuitive to maximise some indicators of overall model fit like R2 or RSME.
Papayapap
CC BY-SA 4.0
2022-08-31T11:44:35.553Z
@Papayapap Yes, thank you for your idea. Interestingly, the coefficient is significant for  nearly the whole range of plausible CPT-parameter values, that's why i didn't consider that a problem first, however, i don't have a good feeling on this as i wrote above. R2 sounds good to me! Seems obvious...
T123
CC BY-SA 4.0
2022-08-31T11:47:19.373Z
If the results don't change much, then I don't see much of a story here, ie no need to device a fancy algorithm. This is closer to standard robustness checks where we try different possibilities and hope that the results do not depend critically on arbitrary decisions taken.
Papayapap
CC BY-SA 4.0
2022-08-31T18:05:15.123Z
Would anyone mind to write an answer so I can mark it as solved? 
T123
<econometrics><utility><applied-econometrics><measuring-utility><calibration>
p-hacking vs. parameter calibration?
T123
# p-hacking vs. parameter calibration?

(Asked by: username_0 on 2022-08-31)

I'm currently tasked in my job to design a Prospect Theory (here CPT-)index as mentioned in Barberis, Mukherjee and Wang (2016): "Prospect Theory and Stock Returns: An Empirical Test", see <https://nicholasbarberis.github.io/bmw17b.pdf>

I wrote the CPT function in C# and got a bunch of stocks incl. their historical prices, which i fed into my analytical engine. While running the program, I noted that the ranking of high/low CPT (cumulative prospect theory) stocks varies wildly with the degree of risk sensitivity, loss aversion and the decision weighting parameter.

BMW (2016) used standard parameter values in their paper but didn't elaborate much on the sensitivity of their CPT-index wrt. the different feasible parameter settings.

My idea is now to circumvent the issue of finding the "right/plausible" CPT parameters and to "calibrate" the parameters used in my CPT-index using in-sample data wrt. the best fit as in BMW (2016) via gridsearch: I want to maximize the significance of the (negative) coefficient in my ts-regression. However, I doubt whether this is the right way to do things as it smells a bit like overfitting and p-hacking to me (although i don't try to increase the predicitive power of the CPT-index..).

All those econometricians among you: Is this a permissible way to get my prospect theory parameters and what might be the issues with this?

I appreciate your help and suggestions, I'm grateful for any comments
Thomas

PS: This is not a homework or thesis task but something i need for my job..
PPS: Corrected the paper reference (wrong autor, It's Baolian Wang, not Wang Baolian as you refer to the familiy name first in China (my chinese wife standing next to me insists i clearify it here:-D )

**Tags:** econometrics, utility, applied-econometrics, measuring-utility, calibration
> **Comments:**
> **username_1**: The title question in its generality is interesting by itself!  
> **username_0**: @RichardHardy Yes, agreed. In my case unfortunately, it's not only academic interest that motivates the question.. :-/  
> **username_1**: I did not mean to imply academic interest alone. Applications are relevant and important by themselves.  
> **username_2**: The proposed approach smells a lot like p-hacking to me. Why would the metric to optimize be the direction of the coefficient, let alone the significance? The result of the t-test is anyway invalid because it is conditional on your optimization procedure. It would be more intuitive to maximise some indicators of overall model fit like R2 or RSME.  
> **username_0**: @Papayapap Yes, thank you for your idea. Interestingly, the coefficient is significant for  nearly the whole range of plausible CPT-parameter values, that's why i didn't consider that a problem first, however, i don't have a good feeling on this as i wrote above. R2 sounds good to me! Seems obvious...  
> **username_2**: If the results don't change much, then I don't see much of a story here, ie no need to device a fancy algorithm. This is closer to standard robustness checks where we try different possibilities and hope that the results do not depend critically on arbitrary decisions taken.  
> **username_0**: Would anyone mind to write an answer so I can mark it as solved?   

---

## 0 answers

