Transcript Introduction to Neural Networks

Introduction to
Neural Networks
Freek Stulp
Overview
Biological Background
Artificial Neuron
Classes of Neural Networks
1.
2.
3.
Perceptrons
Multi-Layered Feed-Forward Networks
Recurrent Networks
Conclusion
2
Biological Background
Neuron consists of:




Cell body
Dendrites
Axon
Synapses
Neural activation :


Throught dendrites/axon
Synapses have different
strengths
3
Artificial Neuron
Input links
(dendrites)
aj
Unit
(cell body)
Output links
(axon)
Wji
ini = ai =
SajWji g(ini)
ai
4
Class I: Perceptron
Ij
Wj
-1 W0
W1
a1
W2
a2
O
in =
a=
SajWj g(in)
a = g(-W0 + W1a1 + W2a2)
a
{
0, in<0
g(in) = 1, in>0
5
Learning in Perceptrons
Perceptrons can learn mappings
from inputs I to outputs O by
changing weights W
Training set D:
 Inputs: I0, I1 ... In
 Targets: T0, T1 ...Tn
Example: boolean OR
D: d
I
T
0
00
0
1
01
1
2
10
1
3
11
1
Output O of network is
not necessary equal to T!
6
Learning in Perceptrons
Error often defined as:
E(W) = 1/2SdD(td-od)2
Go towards the minimum error!
Update rules:



wi = wi + Dwi
Dwi = -hdE/dwi
dE/dwi = d/dwi 1/2SdD(td-od)2
= SdD(td-od)iid
i
This is called gradient descent
7
Class II: Multi-layer
Feed-forward Networks
Multiple layers:

hidden layer(s)
Input
Hidden Output
Feed-forward:

Output links only connected
to input links in the next
layer
Complex non-linear
functions can be
represented
8
Learning in MLFF Networks
For output layer, weight updating similar to
perceptrons.
Problem: What are the errors in the hidden layer?
Backpropagation Algorithm
 For each hidden layer (from output to input):
 For each unit in the layer determine how much it contributed to
the errors in the previous layer.
 Adapt the weight according to this contribution
This is also gradient descent
9
Class III: Recurrent Networks
No restrictions on
connections
Input
Hidden Output
Behaviour more
difficult to predict/
understand
10
Conclusion
Inspiration from biology, though artificial brains
are still very far away.
Perceptrons too simple for most problems.
MLFF Networks good as function approximators.

Many of your articles use these networks!
Recurrent networks complex but useful too.
11
Literature
Artificial Intelligence: A Modern Approach

Stuart Russel and Peter Norvig
Machine Learning

Tom M. Mitchell
12