Efficacious End User Measures—Part 1: Relative Class Size and End User Problem Domains

Figure 11

All of the normalized summary statistics exhibit rCS invariance. All of the lines vary from the horizontal, indicating that rCS invariance weakens when rCS>2⁶. This is a well-known absolute sample size issue. In our tests, the problem became statistically significant when |𝐴|<400.

(a) Peak normalized accuracy rate (90% CI). It strongly resembles the Figures 11(b), 11(c), and 11(d)

(b) Peak normalized F-score (90% CI). when 𝛽=1, it strongly resembles Figures 11(a), 11(c) and 11(d)

(c) Peak normalized MCC (90% CI): it strongly resembles Figures 11(a), 11(b) and 11(d)

(d) Peak Youden index (90% CI). This measure turns out to be related to the normalized accuracy rate

(e) Peak AUC (90% CI). It appears somewhat less sensitive to absolute sample size

(f) Best DOR (90% CI). As rCS invariance weakens, the DOR value drops