LLMs Empower Structure-Adaptive Multiple Testing

Music City Center 

We introduce a general framework for augmenting structure-adaptive multiple testing with auxiliary information extracted from large language models (LLMs). Modern multiple testing procedures such as CAMT, SABHA, and OrderShapeEM gain substantial power by leveraging covariates or prior rankings, yet such structural information is often unavailable or domain-specific. We propose to apply LLMs to generate soft priors, such as feature relevance scores or hypothesis orderings, through prompt-based querying of natural language descriptions. These LLM-derived outputs can be directly integrated into multiple testing algorithms without altering their core logic. We demonstrate the effectiveness of this approach through simulations and real-world applications pertinent to identifying proteomics for cardiovascular disease risk, showing that knowledge from LLMs consistently enhances the power while maintaining valid FDR control. Our results highlight LLMs as flexible and powerful tools for automating auxiliary information in high-dimensional statistical inference.

Benjamini-Hochberg procedure, Cross-fitting, False discovery rate, Leave-one-out analysis, Multiple testing