ALOPEX explained

ALOPEX (an abbreviation of "algorithms of pattern extraction") is a correlation based machine learning algorithm first proposed by Tzanakou and Harth in 1974.

Principle

In machine learning, the goal is to train a system to minimize a cost function or (referring to ALOPEX) a response function. Many training algorithms, such as backpropagation, have an inherent susceptibility to getting "stuck" in local minima or maxima of the response function. ALOPEX uses a cross-correlation of differences and a stochastic process to overcome this in an attempt to reach the absolute minimum (or maximum) of the response function.

Method

ALOPEX, in its simplest form is defined by an updating equation:

\DeltaWij(n)=\gamma\DeltaWij(n-1)\DeltaR(n)+ri(n)

where:

n\geq0

is the iteration or time-step.

\DeltaWij(n)

is the difference between the current and previous value of system variable

Wij

at iteration

n

.

\DeltaR(n)

is the difference between the current and previous value of the response function

R,

at iteration

n

.

\gamma

is the learning rate parameter

(\gamma<0

minimizes

R,

and

\gamma>0

maximizes

R)

ri(n)\simN(0,\sigma2)

Discussion

Essentially, ALOPEX changes each system variable

Wij(n)

based on a product of: the previous change in the variable

\Delta

Wij(n-1)

, the resulting change in the cost function

\Delta

R(n)

, and the learning rate parameter

\gamma

. Further, to find the absolute minimum (or maximum), the stochastic process

rij(n)

(Gaussian or other) is added to stochastically "push" the algorithm out of any local minima.

References