Post-hoc Hypothesis Testing

Oregon Convention Center 

An unfortunate feature of traditional hypothesis testing is the necessity to pre-specify a significance level α to bound the 'size' of the test: its probability to falsely reject the hypothesis. Indeed, a data-dependent selection of α would generally distort the size, possibly making it larger than the selected level α. We develop post-hoc hypothesis testing, which guarantees that there is no such size distortion in expectation, even if the level α is arbitrarily selected based on the data. Unlike regular p-values, resulting 'post-hoc p-values' allow us to 'reject at level p' and still provide this guarantee. Moreover, they can easily be combined since the product of independent post-hoc p-values is a post-hoc p-value. Interestingly, we find that p is a post-hoc p-value if and only if 1/p is an e-value, a recently introduced measure of evidence. Post-hoc hypothesis testing eliminates the need for standardized levels such as α = 0.05. We believe this may take away incentives for p-hacking and contribute to solving the file-drawer problem, as both these issues arise from using a pre-specified significance level.

p-value

e-value

post-hoc inference

anytime valid inference

p-hacking

Scientific discourse 

IMS