Kernel-independent component analysis explained

In statistics, kernel-independent component analysis (kernel ICA) is an efficient algorithm for independent component analysis which estimates source components by optimizing a generalized variance contrast function, which is based on representations in a reproducing kernel Hilbert space.^{[1] [2]} Those contrast functions use the notion of mutual information as a measure of statistical independence.

Main idea

Kernel ICA is based on the idea that correlations between two random variables can be represented in a reproducing kernel Hilbert space (RKHS), denoted by

l{F} , associated with a feature map L _{x:} l{F} \mapsto R defined for a fixed x \in R . The l{F} -correlation between two random variables X and Y is defined as\rho _{l{F} }(X,Y) = \max_ \operatorname(\langle L_X,f \rangle, \langle L_Y,g \rangle)

where the functions

f,g: R \to R range over l{F} and\operatorname{corr}( \langle L _{X,f} \rangle, \langle L _{Y,g} \rangle) : = (\operatorname{cov)/((f(X),) g(Y)) }{\operatorname{var}(f(X)) ^{1/2} \operatorname{var}(g(Y)) ^{1/2} }

for fixed

f,g \in l{F} . Note that the reproducing property implies that f(x) = \langle L _{x,} f \rangle for fixed x \in R and f \in l{F} .^[3] It follows then that the l{F} -correlation between two independent random variables is zero.

This notion of

l{F} -correlations is used for defining contrast functions that are optimized in the Kernel ICA algorithm. Specifically, if X : = (x _{ij} ) \in R ^{n} is a prewhitened data matrix, that is, the sample mean of each column is zero and the sample covariance of the rows is the m x m dimensional identity matrix, Kernel ICA estimates a m x m dimensional orthogonal matrix A so as to minimize finite-sample l{F} -correlations between the columns of S : = X A ^{\prime} .

Notes and References

1. Bach . Francis R. . Jordan . Michael I. . 10.1162/153244303768966085 . Kernel independent component analysis . The Journal of Machine Learning Research . 3 . 1–48 . 2003 .
2. Book: Bach . Francis R. . Jordan . Michael I. . 2003 IEEE International Conference on Acoustics, Speech, and Signal Processing, 2003. Proceedings. (ICASSP '03) . Kernel independent component analysis . 10.1109/icassp.2003.1202783 . 4 . IV-876-9 . 2003 . 978-0-7803-7663-2 . 7691428 .
3. Book: Saitoh, Saburou . Theory of Reproducing Kernels and Its Applications . Longman . 1988. 978-0582035645.