50: An Extensive and Reproducible Comparison of Computer Model Emulators

Music City Center 

Computer model emulation --approximating expensive simulations with surrogate models-- has become an essential tool in uncertainty quantification and scientific computing. Various methods, including Gaussian processes, basis function expansions, and deep learning, have been developed to improve prediction accuracy and computational efficiency. However, their relative performance varies across different problem settings, making systematic evaluation crucial. In this work, we present an extensive and reproducible comparison of 11 emulation methods across 40 simulated and 25 real-world datasets. To facilitate standardized benchmarking, we introduce duqling, an R package designed for organizing and evaluating emulator performance on common test functions and real-world applications. This study provides practical insights into emulator effectiveness and offers a robust framework for future method development and comparison.

emulation

surrogate model

gaussian process

BART

neural networks

uncertainty quantification 

Section on Statistics in Defense and National Security