Presenting the Analytic Process: Beyond Reproducibility

Oregon Convention Center 

A significant trend in data analysis over the past 20 years has been the efforts at promoting computational transparency and reproducibility. These efforts have had many benefits, including the wide dissemination of code and datasets that can be used for both verification and extension with new analyses. However, a question remains as to whether computational reproducibility is a useful indicator of the trustworthiness of a data analysis. While reproducible analyses can be checked more easily for problems or errors, a heavy burden is placed on others to similarly reproduce the time and resources to execute the analytic code. We argue that reproducibility, while useful as a minimum standard for trustworthiness, is not sufficient and that other formats for presenting and distributing data analyses should be considered. We borrow ideas from systems engineering and demonstrate some of these techniques through case studies.