A unifying framework for selective inference with applications in rank verification

Music City Center 

Upon observing n-dimensional multivariate Gaussian data, when can we infer that the largest K observations came from the largest K means? When K=1 and the covariance is isotropic, Gutmann and Maymin (1987) argue that this inference is justified when the two-sided difference-of-means test comparing the largest and second largest observation rejects. After developing a unifying framework for selective inference centered on p-values, we provide a generalization of their procedure that applies for both any K and any covariance structure. We show that our procedure draws the desired inference whenever the two-sided difference-of-means test comparing the pair of observations inside and outside the top K with the smallest standardized difference rejects, and sometimes even when this test fails to reject. Using this insight, we argue that our procedure renders existing simultaneous inference approaches inadmissible when n>2. When the observations are independent (with possibly unequal variances) our procedure corresponds exactly to running the two-sided difference-of-means test comparing the pair of observations inside and outside the top K with the smallest standardized difference.

Selective inference

Winner's curse

Rank Verification

Publication bias

Data carving

Conditional inference 

IMS