Print  Close

Impact of Performance Metrics in AI Model Evaluation

Abstract Number:

1936 

Submission Type:

Contributed Abstract 

Contributed Abstract Type:

Poster 

Participants:

Youssouf Salami (1), Victor Lo (1)

Institutions:

(1) Fidelity Investments, N/A

Co-Author:

Victor Lo  
Fidelity Investments

First Author:

Youssouf Salami  
Fidelity Investments

Presenting Author:

Youssouf Salami  
Fidelity Investments

Abstract Text:

The selection of a performance metric for the purpose of model evaluation is not as trivial as it may appear. On one hand, the model commissioners' expectations of the model's contribution to achieving their business objective (s) often lack empirical support. On the other, the model developers can easily be confused by the multitude of quantitative metrics recommended by the statistical literature. Hence the need for a methodology to guide the effective selection of statistical performance metric during model evaluation. In Salami et al (2024), we considered a fraud detection use case and we showed that F-beta (F_β, β>1) is more appropriate than F_1 or the Area Under the Precision Recall Curve (AUPRC) metric in measuring the model's contribution to the business objective. In this paper, we examine two facets of the F_β, namely the weighted F_β and the non-weighted F_β, and discuss how the selection of one in lieu of the other can lead to erroneous decisions with adverse impacts. As the use of AI algorithms becomes more prevalent in decision making, our paper brings a new perspective to the selection of statistical performance metrics for the purpose of evaluating AI models.

Keywords:

artificial intelligence|machine learning| performance evaluation|performance metrics|model testing|model monitoring, performance thresholds

Sponsors:

Business and Economic Statistics Section

Tracks:

Miscellaneous

Can this be considered for alternate subtype?

Yes

Are you interested in volunteering to serve as a session chair?

No

I have read and understand that JSM participants must abide by the Participant Guidelines.

Yes

I understand that JSM participants must register and pay the appropriate registration fee by June 3, 2025. The registration fee is non-refundable.

I understand