Transcript Introduction to AI

Neural Networks
Multilayer Perceptron (MLP)
Oscar Herrera Alcántara
[email protected]
Outline

Neuron


Artificial neural networks
Activation functions

Perceptrons

Multilayer perceptrons

Backpropagation
Generalization

Introduction to Artificial Intelligence - APSU
Neuron

A neuron is a cell in the brain



collection, processing, and dissemination of electrical signals
1011 neurons of > 20 types, 1014 synapses, 1ms-10ms cycle time
brain’s information processing relies on networks of such neurons
Introduction to Artificial Intelligence - APSU
Biological Motivation

dendrites: nerve fibres carrying electrical signals to the
cell

cell body: computes a non-linear function of its inputs

axon: single long fiber that carries the electrical signal
from the cell body to other neurons

synapse: the point of contact between the axon of one
cell and the dendrite of another, regulating a chemical
connection whose strength affects the input to the cell.
Introduction to Artificial Intelligence - APSU
Artificial neural networks


A mathematical model of the neuron is McCulloch-Pitts unit
Neural networks consists of nodes (units) connected by directed links
1
x1
b :Bias
wi1
x2
m
y  j (v)   ( wi , j  b)
j 1
Neuron i
S
v
j
y
x3
xm

Wim
Synaptic Induced local field Activation
Inputs
Weights Activation potential
function Output
A bias weight Wi,0 connected to a fixed input xi,0 = +1
Introduction to Artificial Intelligence - APSU
Activation functions
j (v) 
1 if   0
j (v ) 
0 if   0
1
1  e  av
a) Step function or Threshold function
b) Sigmoid function
c)
Hyperbolic tangent function
j (v)  a tanh(b j (n))
a, b  0
Introduction to Artificial Intelligence - APSU
Perceptron learning

Learn by adjusting weights to reduce error on training set

Error correction learning rule
e( n )  d ( n )  y ( n )
1 2
E e
2
w ji (n  1)  w ji (n)  [d (n)  y (n)]xi (n)
η  learning rate parameter
Gradient  g(n)  -x(n)e(n)

Perform optimization search by gradient descent
Introduction to Artificial Intelligence - APSU
Implementing logic functions

McCulloch-Pitts unit can implement any Boolean function
X1
X2
X1
y
X1
y
B
v  w0  1  w1  X1  w2  X 2
 1 if v  0
y
0 otherwise
Introduction to Artificial Intelligence - APSU
y
Expressiveness of perceptrons

A perceptron


can represent AND, OR, NOT
can represent a linear separator (function) in input space:
Introduction to Artificial Intelligence - APSU
Multilayer Perceptron (MLP): Architecture
Bias
Input Hidden Layers
Layer
j
j
x1
Inputs
x2
1
j
j
j
y1
Outputs
1
j
x3
Output
Layer
1
wij
j
j
j
wjk
j
wkl
Introduction to Artificial Intelligence - APSU
y2
Solve XOR problem using MLPs



A two-layer network with two nodes in the hidden layer
The hidden layer maps the points from non linear separable space
to linear separable space.
The output layer finds a decision line
-1
w01
A
w11

g1
y1
w13
w21

w12
B
w22

g2
y2
w23
g3
w03
-1
w02
-1
j (v)
Introduction to Artificial Intelligence - APSU
y
Back-propagation Algorithm
1. Initialization. Weights are initialized with random values whose mean
is zero
2. Presentations of training examples
3. Forward computation
4.-Backward computation
for the neuron j of the hidden layer l
 j l (n)  j ' (v j l (n)) k l 1 (n)wkj l 1 (n)
k
for the neuron j of the output layer L
 j (n)  j ' (v j (n))e j
l
l
l 1
w ji (n  1)  w ji  a[w ji (n  1)]   j (n) yi (n)
l
l
l
l
5.- Iteration. Repeat step 2 to 4 until E< desired error
a the momentum parameter is ajusted
 the learning-rate parameter is ajusted
Introduction to Artificial Intelligence - APSU
L
MLP Training
i
Left
j
k
Forward Pass
• Fix wji(n)
• Compute yj(n)
x
Right
y
Backward Pass
• Calculate j(n)
• Update weights wji(n+1)
Left
i
j
Introduction to Artificial Intelligence - APSU
k
Right
Generalization

Total Data are divided in two parts:






Data Training (80%)
MLP is trained with Data Training
Data Test (20%)
MLP is tested with Data Test
Generalization
MLP is used with inputs which have never been
presented in order to predict the outputs
Introduction to Artificial Intelligence - APSU