Breaking Welch’s F Test

Music City Center 

The ANOVA F test famously fails to control the Type I error rate when the variances of the populations differ. Welch's F test is commonly recommended as a robust alternative. However, we find that there is at least one situation where Welch's F test should not be recommended. Specifically, we find through simulation that if the common sample size is fixed at some small number and the number of samples becomes large, then the Type I error rate for Welch's F test tends towards 100% even if all assumptions for the ANOVA F test are met. We explore whether this finding can be confirmed theoretically. We also explore how this finding fits with existing recommendations, whether alternative tests share this deficiency of Welch's F test, and whether this behavior of Welch's F test would have been surprising to Welch and others working on alternatives to the ANOVA F test.

One-way ANOVA;

Simulation

Type I error rate 

Section on Nonparametric Statistics