Wednesday, Aug 5: 10:30 AM - 12:20 PM
6261
Contributed Papers
Thomas M. Menino Convention & Exhibition Center
Room: CC-252A
Main Sponsor
Section on Statistical Learning and Data Science
Presentations
It is quite popular nowadays for researchers and data analysts holding different datasets to seek assistance from each other to enhance their modelling performance. We consider a scenario where different learners hold datasets with potentially distinct variables, and their observations can be aligned by a nonprivate identifier. Their collaboration faces the following difficulties: first, learners may need to keep data values or even variable names undisclosed due to, e.g. commercial interest or privacy regulations; second, there are restrictions on the number of transmission rounds between them due to e.g. communication costs. To address these challenges, we develop a two-stage assisted learning architecture for an agent, Alice, to seek assistance from another agent, Bob. In the first stage, we propose a privacy-aware hypothesis testing-based screening method for Alice to decide on the usefulness of the data from Bob, in a way that only requires Bob to transmit sketchy data. Once Alice recognizes Bob's usefulness, Alice and Bob move to the second stage, where they jointly apply a synergistic iterative model training procedure. With limited transmissions of summary statistics, we sh
Keywords
additive effects
assisted learning
decentralized learning
generalized linear model
hypothesis testing
Cross-validation (CV) is a popular statistical learning technique used to select and evaluate prediction models. This work specifically concerns network-assisted prediction, where latent node positions are extracted from an observed network and further used to predict a node-level response, possibly combined with traditional node-level covariates. The difficulty for model evaluation in this very standard framework arises from the non-exchangeability of the network-connected observations. If one fits all the embeddings using the entire network and then evaluates the regression model via standard CV, training data will see test node connections ahead of time and thus have an "unfairly'' more accurate estimator of the training design. We first demonstrate that this leakage of test node information causes standard cross-validation to underestimate the true out-of-sample risk. We then propose Leave-One-Node-Out Cross-Validation (LONOCV), which corrects for this by refitting the embeddings with one node removed before evaluating the regression model on that node. Additionally, we derive an efficient rank-one update approximation that efficiently recomputes the leave-one-node-out embeddings from the full-network embeddings. Experiments on synthetic and real datasets confirm that LONOCV substantially improves both model selection and evaluation relative to standard cross-validation, at negligible additional computational cost.
Keywords
Cross-validation
Network-assisted prediction
Network analysis
Model selection
In many forms of algorithm-assisted decision-making, an algorithmic decision support tool provides a recommendation, but the human ultimately makes the decision. Determining whether algorithm assistance actually improves human decision-making is critical, and randomized experiments are one way to collect robust evidence. Historically, however, experimental designs and analyses ignore how decision-making behavior adapts with repeated algorithm exposure. In this work, we demonstrate how using a per-decision randomized design and estimating an average effect across all decisions results in misleading effect estimates under three forms of adaptation: gradual overreliance on the algorithm, gradual ignoring of the algorithm, and gradual learning of where the algorithm makes mistakes. We then propose using a staggered rollout design to target alternative effect estimands: the global effect, the habituation effect, and the immediate effect. Finally, drawing on the concept of local average treatment effects, we show that we can identify habituated and immediate effects specifically for decisions where the decision-maker is persuaded by the algorithm to change their decision.
Keywords
experimental design
time-varying treatment effects
human-algorithm collaboration
Sample size can often be limited in various AI/ML modeling contexts where the objective is to develop a binary classification model and where discrimination performance is summarized using area under the ROC curve (AUC). Under these conditions, nested cross-validation (NCV) may be used to examine performance summarized across outer folds versus a standard single train-test split. When there are multiple competing models, there may be additional interest in comparing the respective performances. A naive testing strategy would be to apply standard paired DeLong testing using stacked results across the NCV outer folds. However, this strategy results in overly conservative variance estimates and inflates Type I error; in contrast, bootstrapping the full training process for valid variance estimates may not be computationally feasible for more demanding algorithms. Here, we conduct simulations to examine multiple competing strategies for paired AUC comparisons under NCV, including a novel approach based on influence functions. By considering different outer fold counts, ML algorithms, and target AUC values, we illustrate Type I error and power under comprehensive testing conditions.
Keywords
Machine learning
AUC
Cross-validation
Hypothesis testing
Group testing is a cost-effective strategy for screening large populations for infectious diseases. Instead of testing individually, biospecimens (e.g., blood, urine, swabs) are pooled and tested together to reduce overall testing costs. In infectious disease screening programs using group testing, it is often desirable to relate individual-level covariates (e.g., age, gender, symptoms) to the underlying infection status. However, this task is challenging because individual infection statuses are unobserved; i.e., they are masked by imperfect testing and potentially the pooling protocol. While regression models that address these issues have been developed, existing approaches generally lack the ability to automatically detect and account for nonlinear associations and higher-order interactions between the covariates and the infection status. To address these limitations, we propose a neural network framework for group testing data that automatically detects and accounts for nonlinear relationships and high-order interactions. We illustrate the practical utility of our proposed approach by using it to analyze Chlamydia group testing data collected by the Iowa Public Health Lab.
Keywords
group testing
neural network
imbalanced data
diagnostic testing
EM algorithm
Network datasets appear across a wide range of scientific fields, including biology, physics, and the social sciences. To enable data-driven discoveries from these networks, statistical inference techniques like estimation and hypothesis testing are crucial. However, the size of modern networks often exceeds the storage and computational capacities of existing methods, making timely, statistically rigorous inference difficult. In this work, we introduce a subsampling-based approach aimed at reducing the computational burden associated with estimation and two-sample hypothesis testing. Our strategy involves selecting a small random subset of nodes from the network, conducting inference on the resulting subgraph, and then using interpolation based on the observed connections between the subsample and the rest of the nodes to estimate the entire graph. We develop the methodology under the generalized random dot product graph framework, which affords broad applicability and permits rigorous analysis. Within this setting, we establish consistency guarantees and corroborate the practical effectiveness of the approach through comprehensive simulation studies.
Keywords
Estimation
Hypothesis testing
Network data
Generalized random dot product graph model
Subsampling
Predictive modeling
Classical hypothesis testing frameworks break down in contemporary settings in which null hypotheses are increasingly abstract, the same data are used to both generate and test hypotheses, and minimal assumptions about the underlying data are made. In this work, we propose a new framework for conducting valid hypothesis tests in broad contexts. We propose to add and subtract external noise generated from a symmetric shift-family to our data, X, to partition it into two pieces, X1 and X2. We provide a generic strategy for orthogonalizing X2 against X1 under the null hypothesis H0, then show that testing whether the orthogonalization was successful provides a valid test of H0 under mild assumptions. Remarkably, this framework extends naturally to the post-selection inference setting: we simply select a hypothesis on X1, then perform orthogonalization under the selected null. As our approach neither requires pre-specification of the selection mechanism, nor is restricted to a small class of data-generating distributions, it dramatically expands the settings for which valid post-selection inference can be conducted. We showcase the flexibility of our proposal in several case studies involving challenging pre-specified null hypotheses and post-selection inference scenarios.
Keywords
hypothesis testing
post-selection inference
orthogonalization
randomization
unsupervised learning