Innovative Statistical Methods for Robust Analysis from Small Samples

A central goal in designing clinical trials is to find the test that maximizes power (or equivalently minimizes required sample size) for finding a false null hypothesis while controlling the type I error rate. When there is more than one test, such as in clinical trials with multiple endpoints, the issues of optimal design and optimal procedures become more complex. We address the question of how such optimal tests should be defined and how they can be found. We present a complete solution for deriving optimal procedures for two hypotheses, which have desired monotonicity properties, and are computationally simple. For some of the optimization formulations this yields optimal procedures that are identical to existing procedures, such as Hommel's procedure or the procedure of Bittman et al. (2009), while for other cases it yields completely novel and more powerful procedures than existing ones. For settings with more than two hypotheses, we introduce a bottom-up approach that builds on the closure principle and our optimal two-hypothesis solution to construct new, powerful procedures for familywise error rate control. We demonstrate the advantages of our methods through extensive simulations and real-data applications, highlighting their improved power over existing approaches.

Joint work with Rajesh Karmakar, Abba Krieger, and Saharon Rosset 

In this talk, we consider power-maximizing response-adaptive designs (RADs). In addition to Neyman allocation and variations thereof, we consider RADs found using a recently introduced constrained Markov decision process (CMDP) approach. Such CMDP procedures target maximum Bayesian average power while constraining the type I error rate. We demonstrate that, while the Wald test shows higher power under Neyman allocation than under equal allocation, the type I error rate inflation can be substantially higher. We investigate whether, for two-arm binary outcome trials, combinations of RADs and statistical tests can be found that show a higher power than equal allocation while controlling type I errors. For the RADs, we investigate modifications to control allocations to the inferior arm. For testing, we employ a recently introduced approach to efficiently construct exact tests for RADs. Our results show that CMDP procedures that constrain both type I error and the expected allocations to the inferior arm show the best performance, can reach power gains over equal allocation, control type I errors, and tend to allocate most participants to the superior treatment in expectation. 

Traditional p-value methods for binomial tests can be either too liberal or overly conservative, especially in small samples. We investigate an alternative approach, first proposed in 1969, that formulates hypothesis testing as a direct decision-making problem. This method represents the test as a binary decision for each outcome pair and uses integer programming to find a decision boundary that optimizes power subject to type I error constraints. Our analysis provides new insights into this approach's properties and advantages. When optimized for average power over all possible parameter configurations under the alternative, the method exhibits remarkable robustness - performing optimally or near-optimally across specific alternatives while maintaining exactness. It can then be further customized for particular prior beliefs. We establish theoretical guarantees for controlling type I error despite discretization of the null space, and empirically quantify the power advantages compared to sophisticated p-value calculations. The findings highlight both the method's practical utility and how techniques from combinatorial optimization can enhance statistical methodology.
 

We introduce Causal Responders Detection (CARD), a novel framework for responder analysis that identifies individuals who exhibit a substantial response to treatment, while controlling the false discovery rate (FDR) among treated subjects. CARD builds on recent advances in conformal prediction for novelty detection, providing finite-sample guarantees for responder identification. Additionally, we show how CARD can be extended with propensity score adjustment to account for non-randomized treatment assignment, enhancing its applicability in observational settings. 

In this talk, I revisit optimal response-adaptive designs through the lens of type-I error rate control, uncovering when and how these designs can inflate type-I error — an issue largely overlooked in earlier work. While several methods in the literature attempt to mitigate this inflation, we show they lack robustness, especially in finite samples.
To address this, I propose two new optimal allocation proportions that integrate a more reliable score test (instead of the Wald test) and finite-sample estimators. One allocation proportion optimizes statistical power; the other minimizes the number of treatment failures, both under a fixed variance constraint.

Simulations based on early-phase and confirmatory trials illustrate the practical benefits of these designs: improved patient outcomes and controlled type-I error. While our focus is on binary outcomes, the methodology naturally extends to other settings, including multi-arm trials and alternative performance metrics.

The talk will provide both theoretical insights and practical guidance for designing robust adaptive trials.