Transcript Data Mining Techniques 1

EE459 Neural Networks
Examples of using Neural Networks
Kasin Prakobwaitayakit
Department of Electrical Engineering
Chiangmai University
APPLICATIONS
• Two examples of real life applications of
neural networks for pattern classification:
– RBF networks for face recognition
– FF networks for handwritten recognition
FACE RECOGNITION
• The problem:
– Face recognition of persons of a known
group in an indoor environment.
• The approach:
– Learn face classes over a wide range of
poses using an RBF network.
Dataset
• Sussex database (university of Sussex)
– 100 images of 10 people (8-bit grayscale, resolution
384 x 287)
– for each individual, 10 images of head in different
pose from face-on to profile
– Designed to asses performance of face recognition
techniques when pose variations occur
Datasets (Sussex)
All ten images
for classes 0-3
from the Sussex
database, nosecentred and
subsampled to
25x25 before
preprocessing
Approach: Face unit RBF
• A face recognition unit RBF neural networks is
trained to recognize a single person.
• Training uses examples of images of the person to
be recognized as positive evidence, together with
selected confusable images of other people as
negative evidence.
Network Architecture
• Input layer contains 25*25 inputs which represent
the pixel intensities (normalized) of an image.
• Hidden layer contains p+a neurons:
– p hidden pro neurons (receptors for positive evidence)
– a hidden anti neurons (receptors for negative
evidence)
• Output layer contains two neurons:
– One for the particular person.
– One for all the others.
The output is discarded if the absolute difference of the two
output neurons is smaller than a parameter R.
RBF Architecture for one face recognition
Output units
Linear
Supervised
RBF units
Non-linear
Unsupervised
Input units
Hidden Layer
• Hidden nodes can be:
– Pro neurons:
Evidence for that person.
– Anti neurons:
Negative evidence.
• The number of pro neurons is equal to the positive
examples of the training set. For each pro neuron there is
either one or two anti neurons.
• Hidden neuron model: Gaussian RBF function.
The Parameters
• Centers:
– of a pro neuron: the corresponding positive example
– of an anti neuron: the negative example which is most similar to the
corresponding pro neuron, with respect to the Euclidean distance.
• Spread: average distance of the center vector from all
other centers. If , h hidden nodes, H total number of hidden nodes
then:
 
1
H
|| t

2

 t ||
h
h
• Weights: determined using the pseudo-inverse method.
• A RBF network with 6 pro neurons, 12 anti neurons, and R equal to
0.3, discarded 23 pro cent of the images of the test set and classified
correctly 96 pro cent of the non discarded images.
HANDWRITTEN DIGIT RECOGNITION
Dataset and NN architecture
7,291
for training
• 9,298 examples used
2,007
for testing
• Both training and test set contain ambiguous and
unclassifiable examples.
• 256 input neurons.
• 10 output neurons.
(16x16)
(each represents a numeral)
• A feedforward NN with 3 hidden layers is used.
The Idea
– Shared weights: all neurons in a feature share the same
weights (but not the biases). In this way all neurons detect
the same feature at different positions in the input image.
The detected features are combined to reach shift-invariant
feature detection.
– This is combined with layer implementing subsampling to
decrease the resolution and the sensitivity to distorsions.
NN Architecture
Convolutional NN
– If a neuron in the feature map fires, this corresponds to a
match with the template.
– Neurons of the feature map react to the same pattern at
different positions in the input image.
– For neurons in the feature map that are one neuron apart
(in the matrix representation of the feature map) their
templates in the input image are two pixels apart. Thus
the input image is undersampled, and some position
information is eliminated.
– A similar 2-to-1 undersampling occurs as one goes from
H1 to H2. The rationale is that although high resolution
may be needed to detect a feature, its exact position need
not be determined at equally high precision.
A Feature Map
A Sub-sampling Map
Architecture
• Input layer: 256 neurons with input values in
range [-1, 1].
• Hidden layer H1: consists of 12 feature maps
H1.1, … , H1.12.
• Feature map:
– It consists of 8x8 neurons.
– Each neuron in the feature map has the same
incoming weights , but is connected to a square at a
unique position in the input image. This square is
called a template.
Architecture
• Hidden layer H2:
consists
of 12 sub-sampling maps H2.1, … , H2.12.
• Sub-sampling map:
– Consists of 4x4 neurons.
– Each neuron of the sub-sampling map is
connected to a 5x5 square of H1.j, for each j in 8
of the 12 feature maps.
– All neurons of the sub-sampling map share the
same 25 weights.
Architecture
• Hidden layer H3:
– Consists of 30 neurons.
– H3 is completely connected to the sub-sampling
layer (H2).
• Output layer: consists of 10 neurons,
numbered 0, … , 9 and the neuron with the
highest activation value is chosen. The digit
recognized is equal to the cell number.
• A Dutch master thesis on Le Cun shared
weights NN:
master thesis of D. de Ridder:
“shared weights NN’s in image anlyses”, 1996
http://www.ph.tn.tudelft.nl/~dick
Le Cun NN Architecture
#Cells
L1
L2
L3
L4
L5
Input Layer
16 x 16
12 x (8 x 8) x (5 x 5 + 1)
12 x (8 x 8) x (1) + 12 x (5 x 5)
Feature Maps
12 x (8 x 8)
12 x (4 x 4) x (8 x (5 x 5) + 1)
12 x (4 x 4) x (1) 12 x (8 x (5 x 5) )
Subsampling maps
12 x (4 x 4)
30 x (12 x (4 x 4) + 1)
Hidden Layer
30
10 x (30 + 1)
Ouput layer
10
#Connections
#Weights
256
19968
1068
768
38592
2592
192
5790
5790
310
310
64660
9760
30
Total
10
1256
Le Cun NN Architecture
Training
– Backpropagation was used to learn the weights.
– The hyperbolic tangent was chosen as neuron model.
– After training the number of misclassified patterns was 10
on the training set and 102 on the test set.