Modern Statistical Decision Theory

This paper proposes methods for compound selection decisions in a Gaussian sequence model. Our objective is welfare, defined as the expected utility of a data-dependent decision rule. Inspired by Stein's unbiased risk estimate (SURE), we introduce ASSURE, a family of estimators for welfare. ASSURE enables selection of rules from a pre-specified class by optimizing estimated welfare, thereby borrowing strength across noisy payoff estimates. A leading variant, ASSURE*, is nearly unbiased and achieves near-parametric rates, yielding rules with favorable regret properties conditional on unknown parameters. When the pre-specified class is derived from random-effects models for decision payoffs, these regret guarantees provide robustness to potential prior misspecification, improving the empirical Bayes approach. We apply ASSURE to selecting Census tracts for economic mobility, identifying discriminating firms, and evaluating p-value decision rules in A/B testing. 

Many applications involve estimating the mean of multiple binomial outcomes as a common problem -- assessing intergenerational mobility of census tracts, estimating prevalence of infectious diseases across countries, and measuring click-through rates for different demographic groups. The most standard approach is to report the plain average of each outcome. Despite simplicity, the estimates are noisy when the sample sizes or mean parameters are small. In contrast, the Empirical Bayes (EB) methods are able to boost the average accuracy by borrowing information across tasks. Nevertheless, the EB methods require a Bayesian model where the parameters are sampled from a prior distribution which, unlike the commonly-studied Gaussian case, is unidentified due to discreteness of binomial measurements. Even if the prior distribution is known, the computation is difficult when the sample sizes are heterogeneous as there is no simple joint conjugate prior for the sample size and mean parameter.

In this paper, we consider the compound decision framework which treats the sample size and mean parameters as fixed quantities. We develop an approximate Stein's Unbiased Risk Estimator (SURE) for the average mean squared error given any class of estimators. For a class of machine learning-assisted linear shrinkage estimators, we establish asymptotic optimality, regret bounds, and valid inference. Unlike existing work, we work with the binomials directly without resorting to Gaussian approximations. This allows us to work with small sample sizes and/or mean parameters in both one-sample and two-sample settings. We demonstrate our approach using three datasets on firm discrimination, education outcomes, and innovation rates.
 

Policy decisions often depend on evidence generated elsewhere. We take a Bayesian decision-theoretic approach to choosing where to experiment to optimize external validity. We frame external validity through a policy lens, developing a prior specification for the joint distribution of site-level treatment effects using a microeconometric structural model and allowing for other sources of heterogeneity. With data from South Asia, we show that, relative to basing policies on experiments in optimal sites, large efficiency losses result from instead using evidence from randomly-selected sites or, conversely, from sites with the largest expected treatment effects. 

This paper provides identification results to characterize a fairness-accuracy (FA) frontier, and statistical inference tools to test hypotheses and build a confidence set for the FA-frontier, when outcomes are observed only for selected individuals. When the selection process is unrestricted but loss is measured in specific ways, we provide a characterization of the sharp identification region of the FA-frontier. Under an assumption of unconfoundedness conditional on observables (and unrestricted loss functions), we obtain point identification and propose a debiased machine learning estimator, derive its asymptotic distribution, and show how this can be used to carry out inference for the FA-frontier. In work in progress, we extend the partial identification results to a broader class of loss functions.