Transcript Supervised learning in neural networks: Perceptrons and

INFO331
Machine learning. Neural
networks. Supervised learning in
neural networks.MLP and BP
(Text book: section 2.11, pp.146-155; section
3.7.3., pp.218-221); section 4.2, pp.267282;catch-up reading: pp.251-266)
© N. Kasabov Foundations of Neural Networks, Fuzzy Systems, and Knowledge Engineering, MIT Press, 1996
Machine learning
Issues in machine learning
 Learning from static versus learning from
dynamic data
 Incremental learning
 On-line learning, adaptive learning
 Life-long learning
 Cognitive learning processes in humans

© N. Kasabov Foundations of Neural Networks, Fuzzy Systems, and Knowledge Engineering, MIT Press, 1996
Inductive learning
learning from examples
 Inductive decision trees and the ID3
algorithm
 Information gain evaluation

© N. Kasabov Foundations of Neural Networks, Fuzzy Systems, and Knowledge Engineering, MIT Press, 1996
Other methods of machine
learning
Learning by doing
 Learning from advice
 Learning by analogy
 Case-based learning and reasoning
 Template-based learning (Kasabov and
Clarke) - Iris example

© N. Kasabov Foundations of Neural Networks, Fuzzy Systems, and Knowledge Engineering, MIT Press, 1996
Learning fuzzy rules from data
Cluster-based methods
 Fuzzy template -based method
(Kasabov, 96), pp.218-219
 Wang’s method (pp.220-221)
 Advantages and disadvantages

© N. Kasabov Foundations of Neural Networks, Fuzzy Systems, and Knowledge Engineering, MIT Press, 1996
Supervised learning in neural
networks
Supervised learning in neural networks
 Perceptrons
 Multilayer perceptrons (MLP) and the
backpropagation algorithm
 MLP as universal approximators
 Problems and features of the MPL

© N. Kasabov Foundations of Neural Networks, Fuzzy Systems, and Knowledge Engineering, MIT Press, 1996
Supervised learning in neural
networks



The learning principle is to provide the input
values and the desired output values for each
of the training examples.
The neural network changes its connection
weights during training.
Calculate the error:
• training error - how well a NN has learned the data
• test error - how well a trained NN generalises over
new input data.
© N. Kasabov Foundations of Neural Networks, Fuzzy Systems, and Knowledge Engineering, MIT Press, 1996
Perceptrons

fig.4.8
y
w0
x0
w1
x1
w2
x2
© N. Kasabov Foundations of Neural Networks, Fuzzy Systems, and Knowledge Engineering, MIT Press, 1996
Perceptrons

fig.4.9
P1.
Set a (n+1)-input, m-output perceptron. Randomize all network weights wij,
i=0,1,2,..n, j=1,2,...,m, to small numbers.
P2.
Apply an input feature vector x and calculate the net input signal uj to each output
perceptron neuron j using the standard formula:
uj =  (xi . wij), for i = 0,1,2,...,n, for j = 1,2,...,m, where x0=1 is the bias.
P3.
Apply a hard-limited threshold activation function to the net input signals as
follows:
oj = 1 if uj > threshold, oj = 0 otherwise,
(Applying linear thresholding function is also possible).
P4.
Compute the error for each neuron by subtracting the actual output from the
target output: Errj = yj - oj
P5.
Modify each weight wij by calculating its next value wij(t+1) from the previous
one wij(t) and from the evaluated error Errj:
wij(t+1) = wij(t) + xi. Errj ,
where:  is a learning coefficient - a number between 0 and 1;
P6.
Repeat steps P2 through P5 until the error vector Err is sufficiently low, i.e. the
perceptron goes into a convergence.
© N. Kasabov Foundations of Neural Networks, Fuzzy Systems, and Knowledge Engineering, MIT Press, 1996
Perceptrons
x2

fig.4.10
P4
P2
A
P3
P1
B
x1
© N. Kasabov Foundations of Neural Networks, Fuzzy Systems, and Knowledge Engineering, MIT Press, 1996
MLP and the backpropagation
algorithm

y
fig.4.11
Output level
Intermediate
level (Hidden)
x1
x2
Input level
© N. Kasabov Foundations of Neural Networks, Fuzzy Systems, and Knowledge Engineering, MIT Press, 1996
MLP and the backpropagation
algorithm

fig.4.12
Signal
Error
X
hi
xk
wij
| yj - oj |
oj
wki
Input Layer
yj
(desired
output)
Hidden Layer
Output Layer
© N. Kasabov Foundations of Neural Networks, Fuzzy Systems, and Knowledge Engineering, MIT Press, 1996
MLP and the backpropagation
algorithm

fig.4.13
Forward pass:
BF1.
Apply an input vector x and its corresponding output vector y (the desired output).
BF2.
Propagate forward the input signals through all the neurons in all the layers and
calculate the output signals.
BF3.
Calculate the Errj for every output neuron j as for example:
Errj = yj - oj, where yj is the jth element of the desired output vector y.
Backward pass:
BB1.
Adjust the weights between the intermediate neurons i and output neurons j
according to the calculated error:
wij(t+1) = lrate. oj(1 - oj). Errj. oi + momentum. wij (t)
BB2.
Calculate the error Erri for neurons i in the intermediate layer:
Erri =  Errj. wij
BB3.
Propagate the error back to the neurons k of lower level:
wki(t +1) =lrate.oi(1 - oi). Erri.xk + momentum. wki(t)
© N. Kasabov Foundations of Neural Networks, Fuzzy Systems, and Knowledge Engineering, MIT Press, 1996
MLPs as statistical tools
A MLP with one hidden layer can
approximate any continuous function to
any desired accuracy (Hornik et al,
1989)
 MLP are multivariate non-linear
regression models
 MLP can learn conditional probabilities

© N. Kasabov Foundations of Neural Networks, Fuzzy Systems, and Knowledge Engineering, MIT Press, 1996
Problems and features of the
MPL
How to chose the number of the hidden
nodes
 Catastrophic forgetting
 Introducing hints in neural networks
 Overfitting (overlearning)

© N. Kasabov Foundations of Neural Networks, Fuzzy Systems, and Knowledge Engineering, MIT Press, 1996
Problems and features of the
MPL
Catastrophic forgetting
Error
 fig. 4.14
for set A

for set B
1000
2000
3000
Iterations
© N. Kasabov Foundations of Neural Networks, Fuzzy Systems, and Knowledge Engineering, MIT Press, 1996
Problems and features of the
MPL
Introducing hints
 fig.4.15

Character
Age
Income
Loan approval
Loan
Previous
credit
© N. Kasabov Foundations of Neural Networks, Fuzzy Systems, and Knowledge Engineering, MIT Press, 1996
Problems and features of the
MPL
Overfitting
 fig. 4.16

Error
Etest
Etrain
Number of iterations
© N. Kasabov Foundations of Neural Networks, Fuzzy Systems, and Knowledge Engineering, MIT Press, 1996