Hyper basis function network explained

In machine learning, a Hyper basis function network, or HyperBF network, is a generalization of radial basis function (RBF) networks concept, where the Mahalanobis-like distance is used instead of Euclidean distance measure. Hyper basis function networks were first introduced by Poggio and Girosi in the 1990 paper “Networks for Approximation and Learning”.[1] [2]

Network Architecture

The typical HyperBF network structure consists of a real input vector

x\inRn

, a hidden layer of activation functions and a linear output layer. The output of the network is a scalar function of the input vector,

\phi:Rn\toR

, is given by
N
\phi(x)=\sum
j=1

aj\rhoj(||x-\muj||)

where

N

is a number of neurons in the hidden layer,

\muj

and

aj

are the center and weight of neuron

j

. The activation function

\rhoj(||x-\muj||)

at the HyperBF network takes the following form

\rhoj(||x-\mu

T
(x-\muRj(x-\muj)
j)
j||)=e

where

Rj

is a positive definite

d x d

matrix. Depending on the application, the following types of matrices

Rj

are usually considered[3]
R
j=1
2\sigma2

Id x

, where

\sigma>0

. This case corresponds to the regular RBF network.
R
j=1
2
2\sigma
j

Id x

, where

\sigmaj>0

. In this case, the basis functions are radially symmetric, but are scaled with different width.
R,...,
j=diag\left(1
2
2\sigma
j1
1
2
2\sigma
jz

\right)Id x

, where

\sigmaji>0

. Every neuron has an elliptic shape with a varying size.

Training

Training HyperBF networks involves estimation of weights

aj

, shape and centers of neurons

Rj

and

\muj

. Poggio and Girosi (1990) describe the training method with moving centers and adaptable neuron shapes. The outline of the method is provided below.

Consider the quadratic loss of the network

N
H[\phi
i=1
*
(y
i-\phi
2
(x
i))
. The following conditions must be satisfied at the optimum:
\partialH(\phi*)
\partialaj

=0

,
\partialH(\phi*)
\partial\muj

=0

,
\partialH(\phi*)
\partialW

=0

where

TW
R
j=W
. Then in the gradient descent method the values of

aj,\muj,W

that minimize

H[\phi*]

can be found as a stable fixed point of the following dynamic system:
aj=-\omega
\partialH(\phi*)
\partialaj
,
\muj=-\omega
\partialH(\phi*)
\partial\muj

,
W=-\omega
\partialH(\phi*)
\partialW

where

\omega

determines the rate of convergence.

Overall, training HyperBF networks can be computationally challenging. Moreover, the high degree of freedom of HyperBF leads to overfitting and poor generalization. However, HyperBF networks have an important advantage that a small number of neurons is enough for learning complex functions.[2]

Notes and References

  1. T. Poggio and F. Girosi (1990). "Networks for Approximation and Learning". Proc. IEEE Vol. 78, No. 9:1481-1497.
  2. R.N. Mahdi, E.C. Rouchka (2011). "Reduced HyperBF Networks: Regularization by Explicit Complexity Reduction and Scaled Rprop-Based Training". IEEE Transactions of Neural Networks 2:673–686.
  3. F. Schwenker, H.A. Kestler and G. Palm (2001). "Three Learning Phases for Radial-Basis-Function Network" Neural Netw. 14:439-458.