Transcript Lecture 10 - Artificial Neuron

ARTIFICIAL NEURON
Artificial Neuron Model
ARTIFICIAL NEURON
Artificial Neuron Model
Implementation of AND
function
Let W1 = W2 = 1
X1
0
0
1
1
X2 X1W1 + X2W2
0
0
1
1
0
1
1
2
Y
0
0
0
1
If we make  = 2 (or any value >1 but <=2), we will get
correct results with a unit step activation function
ARTIFICIAL NEURON
Artificial Neuron Model
If we place the 4 points in a
two coordinate system (X1
and X2), we have drawn a line
from (2, 0) to (0, 2) in the
resulting plane
Any new data falling on the
left side of the line will give an
output of zero and the data on
the right side of the line will be
classified as one
ARTIFICIAL NEURON
Artificial Neuron Model
Implementation of OR
function
Let W1 = W2 = 1
X1
0
0
1
1
X2 X1W1 + X2W2
0
0
1
1
0
1
1
2
Y
0
1
1
1
If we make  = 1 (or any value >0 but <=1), we will get
correct results with a unit step activation function
ARTIFICIAL NEURON
Artificial Neuron Model
If we place the 4 points in a
two coordinate system (X1
and X2), we have drawn a line
from (1, 0) to (0, 1) in the
resulting plane
Any new data falling on the
left side of the line will give an
output of zero and the data on
the right side of the line will be
classified as one
ARTIFICIAL NEURON
Artificial Neuron Model
If we want to utilize a unit step function centered at zero for
both AND and OR neurons, we can incorporate another
input X0 constantly set at –1
The weight W0 corresponding to this input would be the ,
calculated previously
ARTIFICIAL NEURON
Setting of weights (Training)
X1
1.0
9.4
2.5
8.0
0.5
7.9
7.0
2.8
1.2
7.8
X2
1.0
6.4
2.1
7.7
2.2
8.4
7.0
0.8
3.0
6.1
Y
1
-1
1
-1
1
-1
-1
1
1
-1
ARTIFICIAL NEURON
Training of weights
Supervised training: the classes of training samples are
known
Random initialization of weights
(threshold or bias is considered as a weight and its input is
fixed at 1)
ARTIFICIAL NEURON
Training of weights: Algorithm
For each input, calculate the output with the current weights
The error will be equal to
ydesired – f(xiwi)
How to reduce this error?
Input is fixed
We have to change weights
Which weight to change?
Since we don’t know which weight is contributing
how much to the error, hence we change all weights
ARTIFICIAL NEURON
Training of weights: Algorithm
How much will be the change in a weight?
The change in each connection weight has to be
proportional to the error and the magnitude of its
input
(we assume that each weight is contributing to the
error a value proportional to its input)
Change each weight by  wi = (y – f(xiwi))xi
The learning rate  is fixed at a small value, so that we may
not only learn the current data sample, but also the retain
the previous learning
ARTIFICIAL NEURON
Training of weights: Algorithm
Iterate repeatedly over the whole data set.
Stop when the combined error of all the inputs is below a
certain threshold
ARTIFICIAL NEURON
Training of weights
For our example let [0.75, 0.5, –0.6] be the initial weights (set
randomly)
Let the learning rate be 0.2
The first data sample is classified correctly
The second data sample gives an error of –2 and the weight
vector should be updated
Wnew = Wold + 0.2 * Error * X
=
-3.01
-2.06
-1.00
ARTIFICIAL NEURON
Training of weights
After 500 iterations the weights converge to [-1.3, -1.1, 10.9]
i.e. output = 1.3x1 –1.1x2 + 10.9 = 0
ARTIFICIAL NEURON
Training of weights
This training rule is called
Error-Correction Learning Rule
It is usually used for step functions
Weights are only adjusted when the neurons give an error at
the output
 wi = (y – f(xiwi))xi
ARTIFICIAL NEURON
Linearly Separable Problems
If the data can be correctly divided into two categories by a
line or hyper-plane
ARTIFICIAL NEURON
Non Linear Problems
If the data cannot be correctly separated by a single line or
plane, then it is not linearly separable. A single layer of
neurons cannot classify the input patterns that are not
linearly separable; e.g. exclusive OR problem
ARTIFICIAL NEURON
Non Linear Problems
For such problems we need two or more layers of neurons
-1
0.5
1
-1
x1
x2
-1
-1
1
-0.5
1
-1
-1
ARTIFICIAL NEURON
Non Linear Problems
The first neuron’s output is
Point (0, 0) => Sum = -0.5 and Y1 = 0
Point (0, 1) => Sum = -1.5 and Y1 = 0
Point (1, 0) => Sum = 0.5 and Y1 = 1
Point (1, 1) => Sum = -0.5 and Y1 = 0
It places a line
ARTIFICIAL NEURON
Non Linear Problems
The second neuron’s output is
Point (0, 0) => Sum = 0.5 and Y2 = 1
Point (0, 1) => Sum = -0.5 and Y2 = 0
Point (1, 0) => Sum = 1.5 and Y2 = 1
Point (1, 1) => Sum = 0.5 and Y2 = 1
It places a line
ARTIFICIAL NEURON
The input sum for the neuron of the 2nd layer = Y1 – Y2 + 1
For point (0,1) Y1 = 0, Y2 = 0, Sum = 1, Y3 = 0
For point (1,0) Y1 = 1, Y2 = 1, Sum = 1, Y3 = 0
For point (0,0) Y1 = 0, Y2 = 1, Sum = 0, Y3 = 1
For point (1,1) Y1 = 0, Y2 = 1, Sum = 0, Y3 = 1
Thus the neuron of the 2nd layer separates the data correctly
ARTIFICIAL NEURON
The first layer projects the input data (x1, x2) into another
dimension (Y1, Y2)
x2
Y2
x1
Y1
In dimension (Y1, Y2), the data becomes linearly seperable
ARTIFICIAL NEURON
Multi Layer Perceptron: Utilization
Hence a single neuron with unit step activation function can
classify the input into two categories
A
A=0
B=1
B
ARTIFICIAL NEURON
Multi Layer Perceptron: Utilization
Two neurons with unit step activation function can classify
the input into four categories
A=0
B=1
00
10
A=0
B=1
01
11
ARTIFICIAL NEURON
Multi Layer Perceptron: Utilization
Two layers of neurons can classify complex types of inputs
B
A
A