Transcript ECE 173 Experimental Class Assignment 2

ECE 173
Assignment #3
1-out-of-m coding
Joshua Wortman
Mathematics/Psychology
Due April 29, 2004
Data Source
• Data represent four classes
inputs represented as
overlapping clusters in 2
dimensional space.
• Data are 100,000 examples
of {x1,x2} pairs associated
with one of four classes.
Clusters are not linearly
seperable.
• All clusters are nearly equally
represented n ≈ 25000.
Objective is 1 out of m MLP coding
Input space is 2 dimensional.
Network will trained with each
M = 4,8,16, & 24 hidden
neurons.
4 outputs to network exist, giving
1 (one) to correct output and 0
for the others.
z q  tanh

y p   z q v pq
q
n
r 1
u qr x r

Weight Updates
vpqnew= vpqold – 2(yp - yp’) zq
updates the weights for outputs from the
hidden neuron layer
uqrnew = uqrold + 2(1- zq2) ( i viq )xr
updates weights from inputs to hidden
layer; update of qth weight is weighted
relative to its contribution to the total error.
Procedure
• TrSet = 40000, TrTstSet = 20000, ValSet = 40000
• Output layer connected to all hidden layer neurons.
• Weights are updated using variable learning rate  near
10-6. After each epoch, Error on training test set is
measured.
– After 50 consecutive error decreases,  grows to 1.5
– After 2 consecutive error increases,  shrinks to 0.7.
(Note: 1.5x0.7 = 1.05)
• Network is trained for each M to find MSE achieved.
• Best MSE to accuracy to time cost result is selected.
M=4
min MSE = 0.5164
epochs = 900
Accuracy
TrSet
56.8%
TstSet
56.9%
alpha:
.00015 <  < .000001
M=8
min MSE = 0.420
epochs = 1000
Accuracy
TrSet
73.3%
TstSet
73.1%
alpha:
.000008 <  < .000003
M=16
min MSE = 0.324
epochs = 1200
Accuracy
TrSet
82.9%
TstSet
82.8%
alpha:
.000005 <  < .000002
M=24
min MSE = 0.2652
epochs = 1500
Accuracy
TrSet
85.1%
TstSet
85.1%
alpha:
.000005 <  < .0000018
Comparing Error plot curves
M=24 learns quickest before flattening
Choose M=24 as optimal condition
Applying Validation Set Yields: 84.9%
correct classifications
Number Hidden
Neurons M
4
8
16
24
Epochs
900
1000
1200
1500
MSE
0.52
0.42
0.32
0.27
Classification
Accuracy
56.8%
73.3%
82.9%
85.1%
MSE using Test Set:
85.12%
MSE using Validation Set:
84.87%
Example Netork Output
The Y1 output values for the
first 1000 examples in the
original data vector are
shown in blue.
Those values which are
actually members of class 1
are circled in red.
The image shows that
output Y1 is giving class 1
inputs substantial
preference.
Conclusions
• It is reasonable that more training time
would lead to higher accuracy.
• From Learning curve comparison graph, it
seems that more neurons may also
increase accuracy and decrease error.
This would have been experimented if not
for time constraints.