Multi-reader multi-case AUC analysis methodology for Artificial Intelligence Applications

Music City Center 

As artificial intelligence (AI) applications become more frequently employed, it is important to be able to statistically evaluate the performance of such systems when used by themselves compared to the performance of human readers using the AI system as an aid, as well as with the performance of unaided human readers.

In this talk I discuss how the Obuchowski-Rockette (OR) method, which treats both cases and human readers as random samples, can be easily adapted for comparing the usefulness of the following three modalities: (1) AI standalone; (2) AI-unaided human readers; and (3) AI-aided human readers. The adaption results from using a a "workaround" that involves a straightforward rearrangement of the data. A real-data example is presented to illustrate this method. A simulation study shows acceptable performance for this approach.

Artificial intelligence



Obuchowski-Rockette


Diagnostic studies


Area under the ROC curve (AUC)


ROC


Multi-reader multi-case study design 

Section on Medical Devices and Diagnostics