Running Effective Product Randomized Experiments for Shopping Ads

Music City Center 

A/B testing is a cornerstone of online experimentation, but it presents unique challenges when randomizing users is not feasible, such as when testing changes to e-commerce product feeds in Shopping Ads. While product-level randomization offers a solution, it often leads to high variance and skewed performance data, and therefore unreliable experiment results. This talk explores practical techniques to improve the reliability and sensitivity of such experiments.

We will dive into variance reduction methods like CUPED and crossover designs, demonstrating how they control for pre-experiment performance differences and leverage within-item comparisons. We will also explore how trimming outliers enhances the robustness of results, while acknowledging the inherent trade-offs.

A key theme will be the importance of pre-experiment power analysis for determining minimum detectable effects and ultimately ensuring your experiment is sufficiently powered. We will then illustrate the significant reductions in required sample sizes these techniques deliver for real-world advertiser data.

Finally, we'll introduce FeedX, an open-source implementation of these methods available on GitHub, enabling you to easily apply these best practices to your own experiments.

Online experiments

Variance reduction methods