In statistics, the phi coefficient (or mean square contingency coefficient and denoted by φ or rφ) is a measure of association for two binary variables.
In machine learning, it is known as the Matthews correlation coefficient (MCC) and used as a measure of the quality of binary (two-class) classifications, introduced by biochemist Brian W. Matthews in 1975.[1]
Introduced by Karl Pearson,[2] and also known as the Yule phi coefficient from its introduction by Udny Yule in 1912[3] this measure is similar to the Pearson correlation coefficient in its interpretation.
A Pearson correlation coefficient estimated for two binary variables will return the phi coefficient.[4]
Two binary variables are considered positively associated if most of the data falls along the diagonal cells. In contrast, two binary variables are considered negatively associated if most of the data falls off the diagonal.
If we have a 2×2 table for two random variables x and y
| y = 1 | y = 0 | total | ||
| x = 1 | n11 | n10 | n1\bullet | |
| x = 0 | n01 | n00 | n0\bullet | |
| total | n\bullet1 | n\bullet0 | n |
\phi=
| n11n00-n10n01 | |
| \sqrt{n1\bulletn0\bulletn\bullet0n\bullet1 |
Phi is related to the point-biserial correlation coefficient and Cohen's d and estimates the extent of the relationship between two variables (2×2).[5]
The phi coefficient can also be expressed using only
n
n11
n1\bullet
n\bullet1
\phi=
| nn11-n1\bulletn\bullet1 | |
| \sqrt{n1\bulletn\bullet1(n-n1\bullet)(n-n\bullet1) |
Although computationally the Pearson correlation coefficient reduces to the phi coefficient in the 2×2 case, they are not in general the same. The Pearson correlation coefficient ranges from −1 to +1, where ±1 indicates perfect agreement or disagreement, and 0 indicates no relationship. The phi coefficient has a maximum value that is determined by the distribution of the two variables if one or both variables can take on more than two values. See Davenport and El-Sanhury (1991) [6] for a thorough discussion.
The MCC is defined identically to phi coefficient, introduced by Karl Pearson,[2] [7] also known as the Yule phi coefficient from its introduction by Udny Yule in 1912.[3] Despite these antecedents which predate Matthews's use by several decades, the term MCC is widely used in the field of bioinformatics and machine learning.
The coefficient takes into account true and false positives and negatives and is generally regarded as a balanced measure which can be used even if the classes are of very different sizes.[8] The MCC is in essence a correlation coefficient between the observed and predicted binary classifications; it returns a value between −1 and +1. A coefficient of +1 represents a perfect prediction, 0 no better than random prediction and −1 indicates total disagreement between prediction and observation. However, if MCC equals neither −1, 0, or +1, it is not a reliable indicator of how similar a predictor is to random guessing because MCC is dependent on the dataset.[9] MCC is closely related to the chi-square statistic for a 2×2 contingency table
|MCC|=\sqrt{
| \chi2 | |
| n |
where n is the total number of observations.
While there is no perfect way of describing the confusion matrix of true and false positives and negatives by a single number, the Matthews correlation coefficient is generally regarded as being one of the best such measures.[10] Other measures, such as the proportion of correct predictions (also termed accuracy), are not useful when the two classes are of very different sizes. For example, assigning every object to the larger set achieves a high proportion of correct predictions, but is not generally a useful classification.
The MCC can be calculated directly from the confusion matrix using the formula:
MCC=
| TP x TN-FP x FN | |
| \sqrt{(TP+FP)(TP+FN)(TN+FP)(TN+FN) |
}
In this equation, TP is the number of true positives, TN the number of true negatives, FP the number of false positives and FN the number of false negatives. If exactly one of the four sums in the denominator is zero, the denominator can be arbitrarily set to one; this results in a Matthews correlation coefficient of zero, which can be shown to be the correct limiting value. In case two or more sums are zero (e.g. both labels and model predictions are all positive or negative), the limit does not exist.
The MCC can be calculated with the formula:
MCC= \sqrt{PPV x TPR x TNR x NPV
The original formula as given by Matthews was:
\begin{align} N&=TN+TP+FN+FP\\ S&=
| TP+FN | |
| N |
\\ P&=
| TP+FP | |
| N |
\\ MCC&=
| TP/N-S x P | |
| \sqrt{PS(1-S)(1-P) |
} \end{align}
This is equal to the formula given above. As a correlation coefficient, the Matthews correlation coefficient is the geometric mean of the regression coefficients of the problem and its dual. The component regression coefficients of the Matthews correlation coefficient are Markedness (Δp) and Youden's J statistic (Informedness or Δp').[11] Markedness and Informedness correspond to different directions of information flow and generalize Youden's J statistic, the
\delta
Some scientists claim the Matthews correlation coefficient to be the most informative single score to establish the quality of a binary classifier prediction in a confusion matrix context.[12] [13]
Given a sample of 12 pictures, 8 of cats and 4 of dogs, where cats belong to class 1 and dogs belong to class 0,
actual = [1,1,1,1,1,1,1,1,0,0,0,0],
assume that a classifier that distinguishes between cats and dogs is trained, and we take the 12 pictures and run them through the classifier, and the classifier makes 9 accurate predictions and misses 3: 2 cats wrongly predicted as dogs (first 2 predictions) and 1 dog wrongly predicted as a cat (last prediction).
prediction = [0,0,'''1''','''1''','''1''','''1''','''1''','''1''','''0''','''0''','''0''',1]
With these two labelled sets (actual and predictions) we can create a confusion matrix that will summarize the results of testing the classifier:
| Cat | Dog | ||
|---|---|---|---|
| Cat | 6 | 2 | |
| Dog | 1 | 3 |
In this confusion matrix, of the 8 cat pictures, the system judged that 2 were dogs, and of the 4 dog pictures, it predicted that 1 was a cat. All correct predictions are located in the diagonal of the table (highlighted in bold), so it is easy to visually inspect the table for prediction errors, as they will be represented by values outside the diagonal.
In abstract terms, the confusion matrix is as follows:
| P | N | ||
|---|---|---|---|
| P | TP | FN | |
| N | FP | TN |
Plugging the numbers from the formula:
MCC=
| 6 x 3-1 x 2 | |
| \sqrt{(6+1) x (6+2) x (3+1) x (3+2) |
See main article: Confusion matrix.
Let us define an experiment from P positive instances and N negative instances for some condition. The four outcomes can be formulated in a 2×2 contingency table or confusion matrix, as follows:
The Matthews correlation coefficient has been generalized to the multiclass case. The generalization called the
RK
K x K
C
MCC=
| \sumk\suml\summCkkClm-CklCmk | |
| \sqrt{\sumk\left(\sumlCkl\right)\left(\sumk'\suml'Ck'l'\right) |
\sqrt{\sumk\left(\sumlClk\right)\left(\sumk'\suml'Cl'k'\right)} }
When there are more than two labels the MCC will no longer range between −1 and +1. Instead the minimum value will be between −1 and 0 depending on the true distribution. The maximum value is always +1.
This formula can be more easily understood by defining intermediate variables:[16]
tk=\sumiCik
pk=\sumiCki
c=\sumkCkk
s=\sumi\sumjCij
MCC=
| cs-\vec{t | |
| ⋅ |
\vec{p}}{ \sqrt{s2-\vec{p} ⋅ \vec{p}} \sqrt{s2-\vec{t} ⋅ \vec{t}} }
| Cat | Dog | Sum | ||
|---|---|---|---|---|
| Cat | 6 | 2 | ||
| Dog | 1 | 3 | ||
| Sum |
Using above formula to compute MCC measure for the dog and cat example discussed above, where the confusion matrix is treated as a 2 × Multiclass example:
| MCC= | (6+3) x {\color{green |
| 12} - {\color{blue}5} x {\color{brown}4} - {\color{purple}7} x {\color{maroon}8}}{\sqrt{{\color{green}12} |
2-{\color{blue}5}2-{\color{purple}7}2}\sqrt{{\color{green}12}2-{\color{brown}4}2-{\color{maroon}8}
| ||||
An alternative generalization of the Matthews Correlation Coefficient to more than two classes was given by Powers by the definition of Correlation as the geometric mean of Informedness and Markedness.
Several generalizations of the Matthews Correlation Coefficient to more than two classes along with new Multivariate Correlation Metrics for multinary classification have been presented by P Stoica and P Babu [17] .
As explained by Davide Chicco in his paper "Ten quick tips for machine learning in computational biology" (BioData Mining, 2017) and "The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation" [18] (BMC Genomics, 2020), the Matthews correlation coefficient is more informative than F1 score and accuracy in evaluating binary classification problems, because it takes into account the balance ratios of the four confusion matrix categories (true positives, true negatives, false positives, false negatives).[18]
The former article explains, for Tip 8:
Chicco's passage might be read as endorsing the MCC score in cases with imbalanced data sets. This, however, is contested; in particular, Zhu (2020) offers a strong rebuttal.[19]
Note that the F1 score depends on which class is defined as the positive class. In the first example above, the F1 score is high because the majority class is defined as the positive class. Inverting the positive and negative classes results in the following confusion matrix:
TP = 0, FP = 0; TN = 5, FN = 95
This gives an F1 score = 0%.
The MCC doesn't depend on which class is the positive one, which has the advantage over the F1 score to avoid incorrectly defining the positive class.