Bayesian reliability assessment for ordinal scoring system

Music City Center 

Assessing the reliability of ordinal scoring systems is a common challenge in research, yet existing tools are limited. To address this gap, we propose a latent variable model that extends the intra-class correlation coefficient (ICC) to accommodate ordinal scores, providing greater flexibility for diverse study designs. The model incorporates mixed effects to account for random variability among raters and subjects. It is particularly suited for unbalanced designs, where the number of evaluations varies across raters and subjects. We develop a full Bayesian framework for inference, enabling estimation of unknown parameters and evaluation of the reliability of ordinal scoring systems. Our results indicate that the proposed approach performs comparably to the cumulative link mixed model when sample sizes are large and significantly outperforms it in small-sample settings. In addition, our method is robust to unbalanced studies, where the number of observations per rater or subject varies. A key advantage of our method is the ability to directly obtain credible intervals for variance parameters and the ICC, which are challenging to estimate using other existing methods.

Ordinal data

intra-class correlation

mixed effects

kappa statistic

Bayesian inference 

International Society for Bayesian Analysis (ISBA)