Spatial GEE for identifying differentially expressed genes in spatial transcriptomics

Music City Center 

Spatial transcriptomics (ST) provides unprecedented insights into gene expression patterns while retaining spatial context, making it valuable for understanding complex tissue architectures like cancers. Seurat, the most popular ST analysis tool, uses the Wilcoxon rank-sum test by default for differential expression (DE) analysis. However, as a nonparametric method that disregards spatial correlations, the Wilcoxon test can lead to inflated false positive rates and misleading findings, highlighting the need for a more robust statistical approach.

We propose a Generalized Score Test (GST) in the Generalized Estimating Equations (GEE) framework as a robust solution for DE analysis in ST. By appropriately accounting for spatial correlations, extensive simulations showed that the GEE GST demonstrated superior Type I error control and comparable power relative to the Wilcoxon test and the GEE robust Wald test. Applications to ST datasets from breast and prostate cancer revealed that the GST-identified DE genes were predominantly enriched in pathways directly implicated in cancer progression, while the Wilcoxon test produced substantial false positives.

Differential expression

GEE

Generalized score test

Spatial transcriptomics

Wilcoxon rank-sum test

Type I error 

Section on Statistics in Genomics and Genetics