Empirical process explained

In probability theory, an empirical process is a stochastic process that characterizes the deviation of the empirical distribution function from its expectation.In mean field theory, limit theorems (as the number of objects becomes large) are considered and generalise the central limit theorem for empirical measures. Applications of the theory of empirical processes arise in non-parametric statistics.^[1]

Definition

For X₁, X₂, ... X_n independent and identically-distributed random variables in R with common cumulative distribution function F(x), the empirical distribution function is defined by

F _{n(x) = (1)/(n)} \sum ^{n} _{i =1} I _{(-infty,x]} (X _{i),} where I_C is the indicator function of the set C.

For every (fixed) x, F_n(x) is a sequence of random variables which converge to F(x) almost surely by the strong law of large numbers. That is, F_n converges to F pointwise. Glivenko and Cantelli strengthened this result by proving uniform convergence of F_n to F by the Glivenko–Cantelli theorem.^[2]

A centered and scaled version of the empirical measure is the signed measure

G _{n(A)=\sqrt{n}(P} _{n(A)-P(A))} It induces a map on measurable functions f given byf\mapsto G _{n} f = \sqrt{n}(P _{n - P)f = \sqrt{n}\left( (1)/(n)} \sum ^{n} _{i =1} f(X _{i)-Ef\right)}

By the central limit theorem,

G _{n(A)} converges in distribution to a normal random variable N(0, P(A)(1 − P(A))) for fixed measurable set A. Similarly, for a fixed function f, G _{nf} converges in distribution to a normal random variable N( 0 , E (f - E f) ^{2)} , provided that E f and E f ^{2} exist.

Definition

l (G _{n(c)r)} _{c\inl{C} } is called an empirical process indexed by l{C} , a collection of measurable subsets of S.l (G _{nfr)} _{f\inl{F} } is called an empirical process indexed by l{F} , a collection of measurable functions from S to R .

A significant result in the area of empirical processes is Donsker's theorem. It has led to a study of Donsker classes: sets of functions with the useful property that empirical processes indexed by these classes converge weakly to a certain Gaussian process. While it can be shown that Donsker classes are Glivenko–Cantelli classes, the converse is not true in general.

Example

As an example, consider empirical distribution functions. For real-valued iid random variables X₁, X₂, ..., X_n they are given by

F _{n(x)=P} _{n((-infty,x])=P} _{nI} _{(-infty,x]} .

In this case, empirical processes are indexed by a class

l{C}=\{(-infty,x]:x\in R \}. It has been shown that l{C} is a Donsker class, in particular,\sqrt{n}(F _{n(x)-F(x))} converges weakly in \ell ^{infty(R)} to a Brownian bridge B(F(x)) .

See also

• Khmaladze transformation
• Weak convergence of measures
• Glivenko–Cantelli theorem

Further reading

• Book: Billingsley, P. . Probability and Measure . John Wiley and Sons . New York . Third . 1995 . 0471007102 .
• Donsker . M. D. . Justification and Extension of Doob's Heuristic Approach to the Kolmogorov- Smirnov Theorems . 10.1214/aoms/1177729445 . The Annals of Mathematical Statistics . 23 . 2 . 277–281 . 1952 . free .
• Dudley . R. M. . Central Limit Theorems for Empirical Measures . 10.1214/aop/1176995384 . The Annals of Probability . 6 . 6 . 899–929 . 1978 . free .
• Book: Dudley, R. M. . Uniform Central Limit Theorems . Cambridge Studies in Advanced Mathematics . 63 . Cambridge University Press . Cambridge, UK . 1999 .
• Book: Kosorok . M. R. . Introduction to Empirical Processes and Semiparametric Inference . 10.1007/978-0-387-74978-5 . Springer Series in Statistics . 2008 . 978-0-387-74977-8 .
• Book: Galen Shorack . Jon Wellner . Shorack . G. R. . Wellner . J. A. . 10.1137/1.9780898719017 . Empirical Processes with Applications to Statistics . 2009 . 978-0-89871-684-9 .
• Book: Aad W. . van der Vaart . Aad van der Vaart . Jon A. . Wellner . Weak Convergence and Empirical Processes: With Applications to Statistics . 2nd . Springer . 2000 . 978-0-387-94640-5 .
• Dzhaparidze . K. O. . Nikulin . M. S. . 10.1007/BF01239992 . Probability distributions of the Kolmogorov and omega-square statistics for continuous distributions with shift and scale parameters . Journal of Soviet Mathematics . 20 . 3 . 2147 . 1982 . 123206522 .

External links

• Empirical Processes: Theory and Applications, by David Pollard, a textbook available online.
• Introduction to Empirical Processes and Semiparametric Inference, by Michael Kosorok, another textbook available online.

Notes and References

1. Mojirsheibani . M. . Nonparametric curve estimation with missing data: A general empirical process approach . 10.1016/j.jspi.2006.02.016 . Journal of Statistical Planning and Inference . 137 . 9 . 2733–2758 . 2007 .
2. Wolfowitz . J. . 10.1214/aoms/1177728852 . Generalization of the Theorem of Glivenko-Cantelli . The Annals of Mathematical Statistics . 25 . 131–138 . 1954 . free .