In statistics, Spearman's rank correlation coefficient or Spearman's ρ, named after Charles Spearman[1] and often denoted by the Greek letter \rho
rs
The Spearman correlation between two variables is equal to the Pearson correlation between the rank values of those two variables; while Pearson's correlation assesses linear relationships, Spearman's correlation assesses monotonic relationships (whether linear or not). If there are no repeated data values, a perfect Spearman correlation of +1 or −1 occurs when each of the variables is a perfect monotone function of the other.
Intuitively, the Spearman correlation between two variables will be high when observations have a similar (or identical for a correlation of 1) rank (i.e. relative position label of the observations within the variable: 1st, 2nd, 3rd, etc.) between the two variables, and low when observations have a dissimilar (or fully opposed for a correlation of −1) rank between the two variables.
Spearman's coefficient is appropriate for both continuous and discrete ordinal variables.[2] [3] Both Spearman's
\rho
\tau
The coefficient can be used to determine how well data fits a model,[4] or to determine the similarity of text documents.[5]
The Spearman correlation coefficient is defined as the Pearson correlation coefficient between the rank variables.[6]
For a sample of size n, the n raw scores
Xi,Yi
\operatorname{R}({Xi}),\operatorname{R}({Yi})
rs
rs= \rho\operatorname{R(X),\operatorname{R}(Y)}=
\operatorname{cov | |
(\operatorname{R}(X), |
\operatorname{R}(Y))} {\sigma\operatorname{R(X)}\sigma\operatorname{R(Y)}},
where
\rho
\operatorname{cov}(\operatorname{R}(X),\operatorname{R}(Y))
\sigma\operatorname{R(X)}
\sigma\operatorname{R(Y)}
Only if all n ranks are distinct integers, it can be computed using the popular formula
rs=1-
| |||||||||
n(n2-1) |
,
where
di=\operatorname{R}(Xi)-\operatorname{R}(Yi)
n is the number of observations.
Consider a bivariate sample
(xi,yi),i=1...,n
(R(Xi),R(Yi))=(Ri,Si)
x,y
rs=
| ||||||||||||
\overline{S} |
}{ \sigmaR\sigmaS},
where, as usual,
\overline{R}=
n | ||||
| ||||
i=1 |
Ri
\overline{S}=
n | ||||
| ||||
i=1 |
Si
2 | |
\sigma | |
R |
=
n | ||||
| ||||
i=1 |
(Ri-\overline{R})2
2 | |
\sigma | |
S |
=
n | ||||
| ||||
i=1 |
(Si-\overline{S})2
We shall show that
rs
di:=Ri-Si
Under this assumption, we have that
R,S
U
\{1,2,\ldots,n\}
\overline{R}=\overline{S}=E[U]
2 | |
\sigma | |
R |
2 | |
=\sigma | |
S |
=Var(U)=E[U2]-E[U]2
E[U]=
n | ||||
| ||||
i=1 |
i=
|
E[U2]=
n | ||||
| ||||
i=1 |
i2=
|
Var(U)=
|
-
2 | ||||
\left(
|
=
|
Observe now that
\begin{align}
1 | |
n |
n | |
\sum | |
i=1 |
RiSi-\overline{R}\overline{S} &=
1 | |
n |
n | |
\sum | |
i=1 |
1 | |
2 |
2 | |
(R | |
i |
+
2 | |
S | |
i |
-
2 | |
d | |
i |
) -\overline{R}2\\ &=
1 | |
2 |
1 | |
n |
n | |
\sum | |
i=1 |
2 | |
R | |
i |
+
1 | |
2 |
1 | |
n |
n | |
\sum | |
i=1 |
2 | |
S | |
i |
-
1 | |
2n |
n | |
\sum | |
i=1 |
2 | |
d | |
i |
-\overline{R}2\\ &=(
1 | |
n |
n | |
\sum | |
i=1 |
2 | |
R | |
i |
-\overline{R}2) -
1 | |
2n |
n | |
\sum | |
i=1 |
2 | |
d | |
i |
\\ &=
2 | |
\sigma | |
R |
-
1 | |
2n |
n | |
\sum | |
i=1 |
2 | |
d | |
i |
\\ &=\sigmaR\sigmaS-
1 | |
2n |
n | |
\sum | |
i=1 |
2 | |
d | |
i |
\\ \end{align}
Putting this all together thus yields
rs=
| ||||||||||||||||||
\sigmaR\sigmaS |
=1-
| |||||||||||||||
|
=1-
| |||||||||||||||
n(n2-1) |
.
Identical values are usually[7] each assigned fractional ranks equal to the average of their positions in the ascending order of the values, which is equivalent to averaging over all possible permutations.
If ties are present in the data set, the simplified formula above yields incorrect results: Only if in both variables all ranks are distinct, then
\sigma\operatorname{R(X)}\sigma\operatorname{R(Y)}=\operatorname{Var}{(\operatorname{R}(X))}=\operatorname{Var}{(\operatorname{R}(Y))}=(n2-1)/12
The simplified method should also not be used in cases where the data set is truncated; that is, when the Spearman's correlation coefficient is desired for the top X records (whether by pre-change rank or post-change rank, or both), the user should use the Pearson correlation coefficient formula given above.[8]
See main article: Correlation and dependence.
There are several other numerical measures that quantify the extent of statistical dependence between pairs of observations. The most common of these is the Pearson product-moment correlation coefficient, which is a similar correlation method to Spearman's rank, that measures the “linear” relationships between the raw numbers rather than between their ranks.
An alternative name for the Spearman rank correlation is the “grade correlation”;[9] in this, the “rank” of an observation is replaced by the “grade”. In continuous distributions, the grade of an observation is, by convention, always one half less than the rank, and hence the grade and rank correlations are the same in this case. More generally, the “grade” of an observation is proportional to an estimate of the fraction of a population less than a given value, with the half-observation adjustment at observed values. Thus this corresponds to one possible treatment of tied ranks. While unusual, the term “grade correlation” is still in use.[10]
The sign of the Spearman correlation indicates the direction of association between X (the independent variable) and Y (the dependent variable). If Y tends to increase when X increases, the Spearman correlation coefficient is positive. If Y tends to decrease when X increases, the Spearman correlation coefficient is negative. A Spearman correlation of zero indicates that there is no tendency for Y to either increase or decrease when X increases. The Spearman correlation increases in magnitude as X and Y become closer to being perfectly monotonic functions of each other. When X and Y are perfectly monotonically related, the Spearman correlation coefficient becomes 1. A perfectly monotonic increasing relationship implies that for any two pairs of data values and, that and always have the same sign. A perfectly monotonic decreasing relationship implies that these differences always have opposite signs.
The Spearman correlation coefficient is often described as being "nonparametric". This can have two meanings. First, a perfect Spearman correlation results when X and Y are related by any monotonic function. Contrast this with the Pearson correlation, which only gives a perfect value when X and Y are related by a linear function. The other sense in which the Spearman correlation is nonparametric is that its exact sampling distribution can be obtained without requiring knowledge (i.e., knowing the parameters) of the joint probability distribution of X and Y.
In this example, the arbitrary raw data in the table below is used to calculate the correlation between the IQ of a person with the number of hours spent in front of TV per week [fictitious values used].
IQ, Xi | Hours of TV per week, Yi | |
---|---|---|
106 | 7 | |
100 | 27 | |
86 | 2 | |
101 | 50 | |
99 | 28 | |
103 | 29 | |
97 | 20 | |
113 | 12 | |
112 | 6 | |
110 | 17 |
Firstly, evaluate
2 | |
d | |
i |
Xi
xi
xi
Yi
yi
di
xi
yi
2 | |
d | |
i |
di
IQ, Xi | Hours of TV per week, Yi | rank xi | rank yi | di |
| ||||||
---|---|---|---|---|---|---|---|---|---|---|---|
86 | 2 | 1 | 1 | 0 | 0 | ||||||
97 | 20 | 2 | 6 | −4 | 16 | ||||||
99 | 28 | 3 | 8 | −5 | 25 | ||||||
100 | 27 | 4 | 7 | −3 | 9 | ||||||
101 | 50 | 5 | 10 | −5 | 25 | ||||||
103 | 29 | 6 | 9 | −3 | 9 | ||||||
106 | 7 | 7 | 3 | 4 | 16 | ||||||
110 | 17 | 8 | 5 | 3 | 9 | ||||||
112 | 6 | 9 | 2 | 7 | 49 | ||||||
113 | 12 | 10 | 4 | 6 | 36 |
With
2 | |
d | |
i |
\sum
2 | |
d | |
i |
=194
\rho=1-
| |||||||||
n(n2-1) |
to give
\rho=1-
6 x 194 | |
10(102-1) |
,
which evaluates to with a p-value = 0.627188 (using the t-distribution).
That the value is close to zero shows that the correlation between IQ and hours spent watching TV is very low, although the negative value suggests that the longer the time spent watching television the lower the IQ. In the case of ties in the original values, this formula should not be used; instead, the Pearson correlation coefficient should be calculated on the ranks (where ties are given ranks, as described above).
Confidence intervals for Spearman's ρ can be easily obtained using the Jackknife Euclidean likelihood approach in de Carvalho and Marques (2012).[11] The confidence interval with level
\alpha
\left\{\theta:
| ||||||||||
2}{\sum |
n | |
i=1 |
(Zi-\theta)2}\leq
2 | |
\chi | |
1,\alpha |
\right\},
where
2 | |
\chi | |
1,\alpha |
\alpha
Zi
One approach to test whether an observed value of ρ is significantly different from zero (r will always maintain) is to calculate the probability that it would be greater than or equal to the observed r, given the null hypothesis, by using a permutation test. An advantage of this approach is that it automatically takes into account the number of tied data values in the sample and the way they are treated in computing the rank correlation.
Another approach parallels the use of the Fisher transformation in the case of the Pearson product-moment correlation coefficient. That is, confidence intervals and hypothesis tests relating to the population value ρ can be carried out using the Fisher transformation:
F(r)=
1 | ln | |
2 |
1+r | |
1-r |
=\operatorname{arctanh}r.
If F(r) is the Fisher transformation of r, the sample Spearman rank correlation coefficient, and n is the sample size, then
z=\sqrt{
n-3 | |
1.06 |
is a z-score for r, which approximately follows a standard normal distribution under the null hypothesis of statistical independence .[12] [13]
One can also test for significance using
t=r\sqrt{
n-2 | |
1-r2 |
which is distributed approximately as Student's t-distribution with degrees of freedom under the null hypothesis.[14] A justification for this result relies on a permutation argument.[15]
A generalization of the Spearman coefficient is useful in the situation where there are three or more conditions, a number of subjects are all observed in each of them, and it is predicted that the observations will have a particular order. For example, a number of subjects might each be given three trials at the same task, and it is predicted that performance will improve from trial to trial. A test of the significance of the trend between conditions in this situation was developed by E. B. Page[16] and is usually referred to as Page's trend test for ordered alternatives.
Classic correspondence analysis is a statistical method that gives a score to every value of two nominal variables. In this way the Pearson correlation coefficient between them is maximized.
There exists an equivalent of this method, called grade correspondence analysis, which maximizes Spearman's ρ or Kendall's τ.[17]
There are two existing approaches to approximating the Spearman's rank correlation coefficient from streaming data.[18] [19] The first approachinvolves coarsening the joint distribution of
(X,Y)
X,Y
m1,m2
X
Y
-infty
infty
(m1+1) x (m2+1)
M
M[i,j]
(i,j)
M[i,j]
M
The second approach to approximating the Spearman's rank correlation coefficient from streaming data involves the use of Hermite series based estimators. These estimators, based on Hermite polynomials,allow sequential estimation of the probability density function and cumulative distribution function in univariate and bivariate cases. Bivariate Hermite series densityestimators and univariate Hermite series based cumulative distribution function estimators are plugged into a large sample version of theSpearman's rank correlation coefficient estimator, to give a sequential Spearman's correlation estimator. This estimator is phrased interms of linear algebra operations for computational efficiency (equation (8) and algorithm 1 and 2). These algorithms are only applicable to continuous random variable data, but havecertain advantages over the count matrix approach in this setting. The first advantage is improved accuracy when applied to large numbers of observations. The second advantage is that the Spearman's rank correlation coefficient can becomputed on non-stationary streams without relying on a moving window. Instead, the Hermite series based estimator uses an exponential weighting scheme to track time-varying Spearman's rank correlation from streaming data,which has constant memory requirements with respect to "effective" moving window size. A software implementation of these Hermite series based algorithms exists and is discussed in Software implementations.
cor.test(x, y, method = "spearman")
in its "stats" package (also cor(x, y, method = "spearman")
will work. The package spearmanCI computes confidence intervals. The package hermiter[20] computes fast batch estimates of the Spearman correlation along with sequential estimates (i.e. estimates that are updated in an online/incremental manner as new observations are incorporated). spearman <i>varlist</i>
calculates all pairwise correlation coefficients for all variables in varlist.[r,p] = corr(x,y,'Type','Spearman')
where r
is the Spearman's rank correlation coefficient, p
is the p-value, and x
and y
are vectors.[21]scipy.stats
module, as well as with the DataFrame.corr(method='spearman')
method from the pandas library, and the corr(x, y, method='spearman')
function from the statistical package pingouin.