In particle physics, CLs[1] represents a statistical method for setting upper limits (also called exclusion limits) on model parameters, a particular form of interval estimation used for parameters that can take only non-negative values. Although CLs are said to refer to Confidence Levels, "The method's name is ... misleading, as the CLs exclusion region is not a confidence interval."[2] It was first introduced by physicists working at the LEP experiment at CERN and has since been used by many high energy physics experiments. It is a frequentist method in the sense that the properties of the limit are defined by means of error probabilities, however it differs from standard confidence intervals in that the stated confidence level of the interval is not equal to its coverage probability. The reason for this deviation is that standard upper limits based on a most powerful test necessarily produce empty intervals with some fixed probability when the parameter value is zero, and this property is considered undesirable by most physicists and statisticians.[3]
Upper limits derived with the CLs method always contain the zero value of the parameter and hence the coverage probability at this point is always 100%. The definition of CLs does not follow from any precise theoretical framework of statistical inference and is therefore described sometimes as ad hoc. It has however close resemblance to concepts of statistical evidence[4] proposed by the statistician Allan Birnbaum.
\theta\in[0,infty)
1-\alpha'
\thetaup(X)
P(\thetaup(X)\geq\theta|\theta)
1-\alpha'
An equivalent definition can be made by considering a hypothesis test of the null hypothesis
H0:\theta=\theta0
H1:\theta=0
\theta0
\alpha
\theta0
\thetaup(X)<\theta0
1-\beta
H0
\alpha/(1-\beta)
\alpha'
\theta0
\alpha'
\theta0
\theta=0
q\theta(X)
\theta
| |||||||||||||
|
=\alpha'.
where
* | |
q | |
\theta |
Upper limits based on the CLs method were used in numerous publications of experimental results obtained at particle accelerator experiments such as LEP, the Tevatron and the LHC, most notable in the searches for new particles.
The original motivation for CLs was based on a conditional probability calculation suggested by physicist G. Zech[5] for an event counting experiment. Suppose an experiment consists of measuring
n
s
b
n\simPoiss(s+b)
b
s
s
n*
s
P(n\leqn*|s+b)\leq\alpha
1-\alpha
b=3
n*=0
s+b\geq3
s\geq0
s
s
n\leqn*
nb\leqn*
nb
nb
nb
nb
s
P(n\leqn*|nb\leqn*,s+b)=
P(n\leqn*,nb\leqn*|s+b) | |
P(nb\leqn*|s+b) |
=
P(n\leqn*|s+b) | |
P(n\leqn*|b) |
.
which correspond to the above definition of CLs. The first equality just uses the definition of Conditional probability, and the second equality comes from the fact that if
n\leqn* ⇒ nb\leqn*
Zech's conditional argument can be formally extended to the general case. Suppose that
q(X)
p\theta=P(q(X)>q*|\theta)
where
q*
p\theta
\theta
\theta
q*
p\theta\leqP(q(X)>q*|0)\equiv
* | |
p | |
0 |
from which, similarly to conditioning on
nb
P(q(X)\geqq*|p\theta\leq
* | |
p | |
0 |
,\theta)=
P(q(X)\geqq*|\theta) | ||||||||||||
|
=
P(q(X)\geqq*|\theta) | ||||||
|
=
P(q(X)\geqq*|\theta) | |
P(q(X)>q*|0) |
.
The arguments given above can be viewed as following the spirit of the conditionality principle of statistical inference, although they express a more generalized notion of conditionality which do not require the existence of an ancillary statistic. The conditionality principle however, already in its original more restricted version, formally implies the likelihood principle, a result famously shown by Birnbaum.[6] CLs does not obey the likelihood principle, and thus such considerations may only be used to suggest plausibility, but not theoretical completeness from the foundational point of view. (The same however can be said on any frequentist method if the conditionality principle is regarded as necessary).
Birnbaum himself suggested in his 1962 paper that the CLs ratio
\alpha/(1-\beta)
\alpha
\alpha
(1-\beta)
1-\alpha
(\beta)
H1,(H2)
H1
\alpha/(1-\beta)
\alpha/(1-\beta)
A more direct approach leading to a similar conclusion can be found in Birnbaum's formulation of the Confidence principle, which, unlike the more common version, refers to error probabilities of both kinds. This is stated as follows:[7]
"A concept of statistical evidence is not plausible unless it finds 'strong evidence foras againstH2
' with small probabilityH1
when(\alpha)
is true, and with much larger probabilityH1
when(1-\beta)
is true."H2
Such definition of confidence can naturally seem to be satisfied by the definition of CLs. It remains true thatboth this and the more common (as associated with the Neyman-Pearson theory) versions of the confidence principle are incompatible with the likelihood principle, and therefore no frequentist method can be regarded as a truly complete solution to the problems raised by considering conditional properties of confidence intervals.
If certain regularity conditions are met, then a general likelihood function will become a Gaussian function in the large sample limit. In such case the CLs upper limit at confidence level
1-\alpha'
\thetaup=\hat\theta+\sigma\Phi-1(1-\alpha'\Phi(\hat\theta/\sigma)),
where
\Phi
\hat\theta
\theta
\sigma
\theta
Comment
. Statistical Science . 17 . 2 . 161–163 . 2002 . 3182818 . 10.1214/ss/1030550859. free .