Extensions of Fisher's method explained

In statistics, extensions of Fisher's method are a group of approaches that allow approximately valid statistical inferences to be made when the assumptions required for the direct application of Fisher's method are not valid. Fisher's method is a way of combining the information in the p-values from different statistical tests so as to form a single overall test: this method requires that the individual test statistics (or, more immediately, their resulting p-values) should be statistically independent.

Dependent statistics

A principal limitation of Fisher's method is its exclusive design to combine independent p-values, which renders it an unreliable technique to combine dependent p-values. To overcome this limitation, a number of methods were developed to extend its utility.

Known covariance

Brown's method

Fisher's method showed that the log-sum of k independent p-values follow a χ²-distribution with 2k degrees of freedom:^{[1] [2]}

X = -2 \sum ^{k} _{i =1} log _{e(p} _{i)} \sim \chi ^{2(2k)} .

In the case that these p-values are not independent, Brown proposed the idea of approximating X using a scaled χ²-distribution, cχ²(k’), with k’ degrees of freedom.

The mean and variance of this scaled χ² variable are:

\operatorname{E}[c\chi ^{2(k')]} = ck' , \operatorname{Var}[c\chi ^{2(k')]} = 2 c ^{2k'} .

where

c = \operatorname{Var}(X)/( 2 \operatorname{E}[X]) and k' =2 (\operatorname{E}[X]) ^{2/\operatorname{Var}(X)} . This approximation is shown to be accurate up to two moments.

Unknown covariance

Harmonic mean p-value

See main article: harmonic mean p-value. The harmonic mean p-value offers an alternative to Fisher's method for combining p-values when the dependency structure is unknown but the tests cannot be assumed to be independent.^{[3] [4]}

This method requires the test statistics' covariance structure to be known up to a scalar multiplicative constant.

Cauchy combination test

This is conceptually similar to Fisher's method: it computes a sum of transformed p-values. Unlike Fisher's method, which uses a log transformation to obtain a test statistic which has a chi-squared distribution under the null, the Cauchy combination test uses a tan transformation to obtain a test statistic whose tail is asymptotic to that of a Cauchy distribution under the null. The test statistic is:

X = \sum ^{k} _{i =1} \omega _{i} \tan[( 0 . 5- p _{i)\pi]} ,

where

\omega _{i} are non-negative weights, subject to \sum ^{k} _{i =1} \omega _{i} = 1 . Under the null, p _{i} are uniformly distributed, therefore \tan[( 0 . 5- p _{i)\pi]} are Cauchy distributed. Under some mild assumptions, but allowing for arbitrary dependency between the p _{i} , the tail of the distribution of X is asymptotic to that of a Cauchy distribution. More precisely, letting W denote a standard Cauchy random variable:\lim _{t} (P[X > t])/(P[W > t]) = 1 .

This leads to a combined hypothesis test, in which X is compared to the quantiles of the Cauchy distribution.^[5]

Notes and References

1. Brown . M. . A method for combining non-independent, one-sided tests of significance . Biometrics . 31 . 987–992 . 1975 . 4 . 10.2307/2529826 . 2529826 .
2. Kost . J. . McDermott . M. . Combining dependent P-values . Statistics & Probability Letters . 60 . 183–190 . 2002 . 2 . 10.1016/S0167-7152(02)00310-3 .
3. Good, I J. 1958. Significance tests in parallel and in series. Journal of the American Statistical Association. 53. 284. 799–813. 10.1080/01621459.1958.10501480. 2281953.
4. Wilson, D J. 2019. The harmonic mean p-value for combining dependent tests. Proceedings of the National Academy of Sciences USA. 116. 4. 1195–1200. 10.1073/pnas.1814092116. 30610179. 6347718. free.
5. Liu Y, Xie J. 2020. Cauchy combination test: a powerful test with analytic p-value calculation under arbitrary dependency structures. Journal of the American Statistical Association. 115. 529. 393–402. 10.1080/01621459.2018.1554485. 7531765.