Partial Area Under the ROC Curve explained

The Partial Area Under the ROC Curve (pAUC) is a metric for the performance of binary classifier.

It is computed based on the receiver operating characteristic (ROC) curve that illustrates the diagnostic ability of a given binary classifier system as its discrimination threshold is varied. The ROC curve is created by plotting the true positive rate (TPR) against the false positive rate (FPR) at various threshold settings.The area under the ROC curve (AUC)^{[1] [2]} is often used to summarize in a single number the diagnostic ability of the classifier. The AUC is simply defined as the area of the ROC space that lies below the ROC curve.

However, in the ROC space there are regions where the values of FPR or TPR are unacceptable or not viable in practice. For instance, the region where FPR is greater than 0.8 involves that more than 80% of negative subjects are incorrectly classified as positives: this is unacceptable in many real cases. As a consequence, the AUC computed in the entire ROC space (i.e., with both FPR and TPR ranging from 0 to 1) can provide misleading indications.

To overcome this limitation of AUC, it was proposed to compute the area under the ROC curve in the area of the ROC space that corresponds to interesting (i.e., practically viable or acceptable) values of FPR and TPR.

Basic concept

In the ROC space, where x=FPR (false positive rate) and y=ROC(x)=TPR (true positive rate), it is

AUC = \int ^{1} _{x =0} ROC(x) dx

The AUC is widely used, especially for comparing the performances of two (or more) binary classifiers: the classifier that achieves the highest AUC is deemed better. However, when comparing two classifiers

C _{a} and C _{b} , three situations are possible:

1. the ROC curve of

C _{a} is never above the ROC curve of C _{b}

1. the ROC curve of

C _{a} is never below the ROC curve of C _{b}

1. the classifiers’ ROC curves cross each other.

There is general consensus that in case 1 classifier

C _{b} is preferable and in case 2) classifier C _{a} is preferable. Instead, in case 3) there are regions of the ROC space where C _{a} is preferable and other regions where C _{b} is preferable. This observation led to evaluating the accuracy of classifications by computing performance metrics that consider only a specific region of interest (RoI) in the ROC space, rather than the whole space. These performance metrics are commonly known as “partial AUC” (pAUC): the pAUC is the area of the selected region of the ROC space that lies under the ROC curve.

Partial AUC obtained by constraining FPR

The idea of the partial AUC was originally proposed ^[3] with the goal of restricting the evaluation of given ROC curves in the range of false positive rates that are considered interesting for diagnostic purposes. Thus, the partial AUC was computed as the area under the ROC curve in the vertical band of the ROC space where FPR is in the range [<math>FPR_{low}</math>, <math>FPR_{high}</math>].

The pAUC computed by constraining FPR helps compare two partial areas. Nonetheless, it has a few limitations:

• the RoI must be a vertical band of the ROC space;
• no criteria are given for identifying the RoI: it is expected that some expert is able to identify

FPR _{low} and FPR _{high} ;

• when comparing two classifiers via the associated ROC curves, a relatively small change in selecting the RoI may lead to different conclusions: in the example above, considering the band where

0 . 1 \leq FPR \leq 0 . 3 leads to conclude that C _{b} is better, while considering the band where 0 . 2 \leq FPR \leq 0 . 4 leads to conclude that C _{a} is better.

Partial AUC obtained by constraining TPR

Another type of partial AUC is obtained by constraining the true positive rate, rather than the false positive rate. That is, the partial AUC is the area under the ROC curve and above the horizontal line

TPR = TPR _{0} .^[4] In other words, the pAUC is computed in the portion of the ROC space where the true positive rate is greater than a given threshold TPR _{0} (no upper limit is used, since it would not make sense to limit the number of true positives).

This proposal too has a few limitations:

• by limiting the true positive rate, a limit on the false positive rate is also implicitly set;
• no criteria are given for identifying the RoI: it is expected that experts can identify the minimum acceptable true positive rate;
• when comparing two classifiers via the associated ROC curves, a relatively small change in selecting the RoI may lead to different conclusions: this happens when

TPR _{0} is close to the point where the given ROC curves cross each other.

Partial AUC obtained by constraining both FPR and TPR

A “two-way” pAUC was defined by constraining both the true positive and false negative rates.^[5] A minimum value

TPR _{0} is specified for TPR and a maximum value FPR _{0} is set for FPR, thus the RoI is the upper-left rectangle with vertices in points (FPR _{0} , TPR _{0} ), (FPR _{0} , 1), (0, 1) and (0, TPR _{0} ). The two-way pAUC is the area under the ROC curve that belongs to such rectangle.

The two-way pAUC is clearly more flexible than the pAUC defined by constraining only FPR or TPR. Actually, the latter two types of pAUC can be seen as special cases of the two-way pAUC.As with the pAUC described above, when comparing two classifiers via the associated ROC curves, a relatively small change in selecting the RoI may lead to different conclusions. This is a particularly delicate issue, since no criteria are given for identifying the RoI (as with the other mentioned pAUC, it is expected that experts can identify

TPR _{0} and FPR _{0} ).

Partial AUC obtained by applying objective constraints to the region of interest

A few objective and sound criteria for defining the RoI were defined.^{[6] [7]} Specifically, the computation of pAUC can be restricted to the region where

• the considered classifiers are better (according to some performance metric of choice) than the random classification;
• the considered classifiers achieve at least a minimum value of some performance metrics of choice;
• the cost due to misclassifications by the considered classifiers is acceptable.

Defining the RoI based on the performance of the random classification

A possible way of defining the region where pAUC is computed consists of excluding the regions of the ROC space that represent performances worse than the performance achieved by the random classification.

Random classification evaluates a given item positive with probability

\rho and negative with probability (1-\rho ). In a dataset of n items, of which AP are actually positive, the best guess is obtained by setting \rho = (AP)/(n) (\rho is also known as the “prevalence” of the positives in the dataset).It was shown that random classification with \rho = (AP)/(n) achieves TPR = \rho , precision = \rho , and FPR = \rho , on average. Therefore, if the performance metrics of choice are TPR, FPR and precision, the RoI should be limited to the portion of the ROC space where TPR>\rho , FPR<\rho , and precision>\rho . It was shown that this region is the rectangle having vertices in (0,0), (0,1), (\rho , 1), and (\rho , \rho ).

This technique solves the problems of constraining TPR and FPR when two-ways pAUC has to be computed:

FPR _{0=TPR} _{0=\rho} .

The Ratio of Relevant Areas (RRA) indicator

Computing the pAUC requires that a RoI is first defined. For instance, when requiring better accuracy than mean random classification, the RoI is the rectangle having vertices in (0,0), (0,1), (

\rho , 1), and (\rho , \rho ). This implies that the size of the RoI varies depending on \rho . Besides, the perfect ROC, i.e., the one that goes through point (0,1), has pAUC=\rho (1-\rho ).

To get a pAUC-based indicator that accounts for

\rho and ranges in [0,1], RRA was proposed:RRA = {pAUC \over area of the RoI}

RRA=1 indicates perfect accuracy, while RRA=0 indicates that the area under the ROC curve belonging to the RoI is null; thus accuracy is no better than random classification's.

Defining the RoI based on some performance metric threshold

Several performance metrics are available for binary classifiers. One of the most popular is the Phi coefficient^[8] (also known as the Matthews Correlation Coefficient^[9]). Phi measures how better (or worse) is a classification, with respect to the random classification, which is characterized by Phi = 0. According to the reference values suggested by Cohen, one can take Phi = 0.35 as a minimum acceptable level of Phi for a classification. In the ROC space, Phi equal to a non null constant corresponds to the arc of an ellipse, while Phi = 0 corresponds to the diagonal, i.e., to the points where FPR=TPR. So, considering the portion of the ROC where Phi>0.35 corresponds to defining the RoI as the portion of the ROC space above the ellipse. The pAUC is the area above the ellipse and under the ROC curve.

Defining the RoI based on the cost of misclassifications

Most binary classifiers yield misclassifications, which result in some cost.

The cost C of misclassifications is defined as

C = c _{FN} FN + c _{FP} FP , where c _{FN} is the unitary cost of a false negative, c _{FP} is the unitary cost of a false positive, and FN and FP are, respectively, the number of false negatives and false positives.

The normalized cost NC^[10] is defined as

NC = (C)/(n(c _{FN} + c _{FP} )) .

By setting

λ= (c _{FN})/(c _{FP} + c _{FN}) , we get NC = λ \rho ( 1- TPR) + ( 1-λ )( 1- \rho) FPR

The average NC obtained via random classification is

NC _{rnd} = (AP ⋅ AN)/(n ^{2})

To evaluate a classifier excluding the performances whose cost is greater than

NC _{rnd} , it is possible to define the RoI where the normalized cost is lower than the NC _{rnd} : such region is above the line(AP ⋅ AN)/(n ^{2}) =λ \rho( 1- TPR)( 1-λ )( 1- \rho)FPR It is also possible to define the RoI where NC is less than a fraction \mu of NC _{rnd} . In such case, the lower boundary of the RoI is the lineTPR = (1-λ)/(λ) (1- \rho)/(\rho) (FPR - \mu \rho) +1- \mu( 1- \rho)

Different values of

λ define the RoI in the same way as some of the best known performance metrics:λ=0 equates to using FPR for delimiting the RoIλ=1- \rho equates to using precision for delimiting the RoIλ=1- (\rho)/(2) equates to using the F-1 score ^[11] for delimiting the RoIλ=1 equates to using TPR for delimiting the RoI

Therefore, choosing a performance metric equates to choosing a specific value of the relative cost of false positives with respect to false negatives. In the ROC space, the slope of the line that represents constant normalized cost (hence, constant total cost) depends on

λ , or, equivalently, on the performance metrics being used.

It is common practice^{[12] [13]} to select as the best classification the point of the ROC curve with the highest value of Youden’s J =TPR−FPR.^[14] When considering the cost associated with the misclassifications, this practice corresponds to making a hypothesis on the relative cost of false positives and false negatives, which is rarely correct.

How to compute pAUC and RRA

Software libraries to compute pAUC and RRA are available for Python and R.^[15]

Notes and References

1. Van der Schouw. Y.T.. Verbeek. A.. Ruijs. J.H.. 1992. ROC Curves For the Initial Assessment of New Diagnostic Tests. Family Practice. 9. 4. 506–511. 10.1093/fampra/9.4.506. 1490547 . 0263-2136.
2. Bradley. Andrew P.. 1997. The use of the area under the ROC curve in the evaluation of machine learning algorithms. Pattern Recognition. en. 30. 7. 1145–1159. 10.1016/S0031-3203(96)00142-2. 1997PatRe..30.1145B . 13806304 . 0031-3203.
3. McClish. D. K.. 1989. Analyzing a portion of the ROC curve. Medical Decision Making. 9. 3. 190–195. 10.1177/0272989X8900900307. 0272-989X. 2668680. 24442201 .
4. Jiang. Y. Metz. C E. Nishikawa. R M. 1996. A receiver operating characteristic partial area index for highly sensitive diagnostic tests.. Radiology. 201. 3. 745–750. 10.1148/radiology.201.3.8939225. 8939225. 0033-8419.
5. Yang. Hanfang. Lu. Kun. Lyu. Xiang. Hu. Feifang. 2017-07-14. Two-way partial AUC and its properties. Statistical Methods in Medical Research. 28. 1. 184–195. 10.1177/0962280217718866. 28707503 . 1508.00298 . 10666428 . 0962-2802.
6. Morasca. Sandro. Lavazza. Luigi. 2020-08-19. On the assessment of software defect prediction models via ROC curves. Empirical Software Engineering. 25. 5. 3977–4019. 10.1007/s10664-020-09861-4. 221167342 . 1382-3256. free.
7. Lavazza. Luigi. Morasca. Sandro. 2021-12-20. Considerations on the region of interest in the ROC space. Statistical Methods in Medical Research. 31 . 3 . 419–437. 10.1177/09622802211060515 . 34928729 . 245354164 . 0962-2802.
8. Book: Cohen, Jacob. Statistical power analysis for the behavioral sciences, 2nd edition. Lawrence Earlbaum Associates. 1988. 9780203771587.
9. Matthews. B.W.. 1975. Comparison of the predicted and observed secondary structure of T4 phage lysozyme. Biochimica et Biophysica Acta (BBA) - Protein Structure. 405. 2. 442–451. 10.1016/0005-2795(75)90109-9. 1180967 . 0005-2795.
10. Book: Cahill. Jaspar. Hogan. James M.. Thomas. Richard. 2013 22nd Australian Software Engineering Conference . Predicting Fault-Prone Software Modules with Rank Sum Classification . 2013. http://dx.doi.org/10.1109/aswec.2013.33. 211–219 . IEEE. 10.1109/aswec.2013.33. 978-0-7695-4995-8 . 302819 .
11. Book: van Rijsbergen, C.J.. Information Retrieval. Butterworth. 1979.
12. Perkins. N.J.. Schisterman. E.F.. 2006. The inconsistency of "optimal" cutpoints obtained using two criteria based on the receiver operating characteristic curve. American Journal of Epidemiology. 163. 7 . 670–675. 10.1093/aje/kwj063 . 16410346 . 1444894 . free.
13. Akobeng. A.K.. 2007. Understanding diagnostic tests: Receiver operating characteristic curves. Acta Paediatrica. 95. 5 . 644–647. 10.1111/j.1651-2227.2006.00178.x . 17376185 . 25035474 .
14. Youden. W. J.. 1950. Index for rating diagnostic tests. Cancer. 3. 1. 32–35. 10.1002/1097-0142(1950)3:1<32::aid-cncr2820030106>3.0.co;2-3. 15405679 . 43532502 . 0008-543X. free.
15. http://www.dista.uninsubria.it/supplemental_material/iRRA/iRRA_package_python.zip Python Library Download