Transcript Radial Basis Networks: - University of Wisconsin–Madison

Radial Basis Networks:
An Implementation
of
Adaptive Centers
Nivas Durairaj
ECE539 Final Project
Brief Description of RBF Networks
• Consists of 3 layers (input, hidden, output)
• Input layer made up of nodes that connect
network to environment
• At input of each neuron (hidden layer), distance
between neuron center & input vector is
calculated
• Apply RBF (Gaussian bell function) to form
output of the neurons.
• Output layer is linear and supplies response of
network to activation function.
Project Overview
Purpose:
Develop a Radial Basis Network with a
supervised selection of centers
A RBF network
with multiple
outputs
Question: Are there any disadvantages or advantages
between a fixed center RBF network and an adaptive
RBF network?
Adaptation Formulas
RBF with supervised selection of centers
require the following formulas:
1. Linear Weights (output layer)
E (n)
wi (n  1)  wi (n)  1
wi (n)
2. Positions of centers (hidden layer)
t i (n  1)  t i (n)   2
E (n)
t i (n)
3. Spreads of centers (hidden layer)
i1 (n  1)  i1 (n) 3
E (n)
 i1 (n)
W: 1x1
T: 1xm vector
 i1 : mxm matrix
M is the feature dimension
Programming
• Used Matlab to implement RBF Network with
Adaptive Centers
• Sample code for calculation of linear weights
given below:
wi (n  1)  wi (n)  1
E (n)
wi (n)
%Calculation of linear weights
weightdiff=0;
for j=1:n
g=exp(-0.5((x(j,:)-t(i,:)))*covinv(:,:,i)*((x(j,:)-t(i,:))'));
weightdiff = weightdiff + e(j)*g;
end
w(i)=w(i) - (eta1*weightdiff);
Testing & Comparison
• Tested Adaptive Center RBF against Fixed
Center RBF.
• Used data for three functions, namely sinusoidal,
piecewise-linear, and polynomial functions.
• Made use of the cost function given below
analyze differences between two networks
Cost Function
1N 2
E  ej
2 j 1
e j d j  F * ( x j )
where
M
 d j   wi G( x j  t i
i 1
Ci
)
Sinusoidal Function Testing
For fewer radial basis functions, adaptive center RBF
network seems to perform a bit better. However, after
number of RBFs increase, results in cost function are
negligible.
Sinosoid Function Data
RBF with Adaptive Centers
1
test samples
approximated curve
train samples
radial basis
0.6
0.5
Cost Function Output
0.5
0
-0.5
0.4
Fixed Center RBF Network
0.3
Adaptive Center RBF Network
0.2
0.1
-1
0
2
-1.5
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
3
4
5
6
No. of Radial Basis Functions
7
Piecewise Linear Function Testing
Adaptive center RBF network performed better till the
number of radial basis functions reached 6. I found that at
higher numbers of radial basis functions (9 and above), both
RBF networks were providing similar approximations of
piecewise-linear function.
Piecewise-Linear Function Data Chart
RBF with Adaptive Centers
1.5
0.0045
1
0.004
0.5
Cost Function Output
0.0035
0
-0.5
-1
-1.5
test samples
approximated curve
train samples
radial basis
0.003
0.0025
Fixed Center RBF Network
Adaptive Center RBF Network
0.002
0.0015
0.001
0.0005
0
-2
2
-2.5
-0.5
-0.4
-0.3
-0.2
-0.1
0
0.1
0.2
0.3
0.4
0.5
3
4
5
6
7
8
9
No. Of Radial Basis Functions
10
Polynomial Function Testing
The adaptive center RBF network was clearly the
winner in the approximation of the polynomial function.
Differences in cost function for higher numbers of RBFs
were too small for Excel to plot.
Polynomial Function Data Chart
RBF with Adaptive Centers
0.1
8.00E-04
0.08
7.00E-04
Cost Function Outputs
0.06
0.04
0.02
0
test samples
approximated curve
train samples
radial basis
-0.02
-0.04
5.00E-04
Fixed Center RBF Network
4.00E-04
Adaptive Center RBF Network
3.00E-04
2.00E-04
1.00E-04
0.00E+00
-0.06
-0.08
-0.5
6.00E-04
2
3
4
5
No. of Radial Basis Functions
-0.4
-0.3
-0.2
-0.1
0
0.1
0.2
0.3
0.4
0.5
6
Conclusion
• Results show RBF network with adaptive
centers performs slightly better than fixed-center
RBF.
• Advantage of Adaptive RBF: Performs better
with fewer RBFs
• Disadvantage of Adaptive RBF: Takes longer to
run.
• Unless situation is known, one cannot say with
certainty that one model is better than other.