Post-hoc p-values

2319 

Contributed Abstract 

Paper 

Nick Koning (1)

(1) Erasmus University, Rotterdam, the Netherlands

Nick Koning  
Erasmus University

Nick Koning  
N/A

A pervasive methodological error is the post-hoc interpretation of p-values. A p-value p is the smallest significance level \alpha at which we would have rejected the null had we chosen level p. It is not the smallest significance level at which we reject the null. We introduce post-hoc p-values, that do admit such a post-hoc interpretation. We show that p is a post-hoc p-value if and only if 1/p is an e-value, a recently introduced statistical object. The product of independent post-hoc p-values is a post-hoc p-value, making them easy to combine. Moreover, any post-hoc p-value can be trivially improved if we permit external randomization, but only (essentially) non-randomized post-hoc p-values can be arbitrarily merged through multiplication. In addition, we discuss what constitutes a 'good' post-hoc p-value. Finally, we argue that post-hoc p-values eliminate the need of a pre-specified significance level, such as alpha = .05 or alpha = .005 Benjamin et al. (2018). We believe this may take away incentives for p-hacking and contribute to solving the file-drawer problem, as both these issues arise from using a pre-specified significance level.

p-value|e-value|post-hoc inference|anytime valid inference|p-hacking|Scientific discourse

IMS

Statistical Theory