Enhancing the Validity of Online A/B Tests with Divergent Units

Music City Center 

In online A/B experiments, aligning the diversion unit with the analysis unit is crucial for unbiased and interpretable results. However, practical constraints frequently force a divergence-for example, when business metrics are granular, but operational realities or the nature of the experiments themselves, necessitate diversion at a higher level. This misalignment introduces hierarchical correlations and jeopardizes the statistical validity of experimental outcomes. This research presents a suite of innovative solutions, widely adopted and proven effective within Google Cloud, to address these complexities. Through rigorous simulations and real-world case studies, we demonstrate how these approaches reduce bias, improve statistical power, and deliver actionable insights from A/B experiments with divergent units. Our findings offer practical guidance for experimenters facing these challenges, ensuring business-critical decisions are based on statistically sound evidence.

Online experiments

A/B experiment

Google

Experiment design

Bias reduction

Hierarchical data 

Section on Statistical Learning and Data Science