Impact of redefining statistical significance on P-hacking and false positive rates.

Music City Center 

In recent years, concern has grown about the inappropriate application and interpretation of
P values, especially the use of P<0.05 to denote "statistical significance" and the practice of
P-hacking to produce results below this threshold and selectively reporting these in publications.
Such behavior is said to be a major contributor to the large number of false and nonreproducible
discoveries found in academic journals. In response, it has been proposed that
the threshold for statistical significance be changed from 0.05 to 0.005. The aim of the current
study was to use an evolutionary agent-based model comprised of researchers who
test hypotheses and strive to increase their publication rates in order to explore the impact
of a 0.005 P value threshold on P-hacking and published false positive rates. The results supported the view that a more stringent
P value threshold can serve to reduce the rate of published false positive results.
Researchers still engaged in P-hacking with the new threshold, but the effort they expended
increased substantially and their overall productivity was reduced, resulting in a decline in
the published false positive rate.

significance threshold

p-hacking

agent-based most

simulation

effect size 

Health Policy Statistics Section