Normatively Backwards Rubric Scoring: Evidence from NIH Peer Review

Oregon Convention Center 

Rubrics are thought to improve quality and decrease social bias in scientific peer review. However, rubrics cannot serve these functions if reviewers sequence their judgments in a normatively backwards order. If reviewers determine the overall merit of a submission before scoring for specific criteria, criteria scores serve as post hoc rationalizations that can, intentionally or unintentionally, mask intellectual and social biases. Despite the importance of proper sequencing in rubric review and the wide adoption of rubrics in high-stakes peer review contexts, there is little to no research on the order with which reviewers score rubric elements in practice. Using a large dataset of preliminary scores for R01 proposals submitted to the National Institutes of Health (NIH) in fiscal years 2014-2016, we employ causal discovery methodology to investigate the causal direction with which assigned reviewers tended to score criteria (Significance, Investigator(s), Innovation, Approach, and Environment) and Overall Impact before panel discussion. We find that Overall Impact tends to be evaluated before Approach – which focuses on scientific strategy, methodology, analyses, and feasibility. We also find that Investigator and Environment tend to be evaluated first, before evaluations of scientific criteria relevant to the content of the proposed research. This evidence stresses the importance of structuring and sequencing rubric review processes to minimize the potential for normatively backwards assessment.
(This is joint work with Carole J. Lee, Fan Xia, Kwun C. G. Chan, Sheridan Grant, and Thomas S. Richardson)