Transcript Slide 1

Neural Networks:
An Overview
There are many different kinds of Neural Network they
include to name a few:
The Back Propagating Networks
Kohonen (Self-Organising) Networks
Neural
Networks:
Hopfield Networks
AnMachine
Overview
The Boltzmann
All Neural Networks use the concept of Neurons which are gradually
trained to give a more appropriate response to input.
This project has used a Kohonen Network.
I shall explain the properties of such a network and why it is best for a
small scale project such as this.
What
So
what
is aisKohonen
a Neuron?
Network?
Neurons have inputs and outputs. The inputs are generally weighted
so that some inputs to the neuron may have a greater affect than
others. The inputs are added together and this result is used to
generate an output for the neuron.
x1
w1
Kohonen (Self-Organising)
Networks
x2
x3
w2
∑
U
w3
A neuron of such as this one is called an Adaptive Linear Element
(or Adaline) and may be found own it’s own …
or may be found as part of a larger “neural network” where the
output of a neuron may be the input for the next “layer”.
This project has used a Kohonen Network.
I shall explain the properties of such a network and why it is best for a
small scale project such as this.
Kohonen (Self-Organising) Networks
What is a Kohonen Network?
A Kohonen network uses these basic principles however in a
Kohonen network there is only one layer of neurons which are
arranged in a 2D lattice.
There is an input layer with inputs which feed into every neuron.
x1
x2
Kohonen (Self-Organising) Networks
How does training work?
What is a Kohonen Network?
The
A
Kohonen
key to the
network
neuraluses
network
theseare
basic
theprinciples
weights ashowever
it is theyinthat
a
Kohonen
dictate
which
network
neuron
there
will
is win
onlywhen
one layer
the network
of neurons
is presented
which arewith a
arranged ininput
particular
a 2D pattern.
lattice.
There is an input layer with inputs which feed
into every
neuron.
Individual
neuron
output
=∑(xi – wik) 2 (for all i)
x1
Where wik is the weight
between input i and
neuron k.
Each neuron evaluates the sum of the weighted inputs and
uses this as it’s own output. This is the Euclidian distance of
x2 from the “input pattern”.
the neuron
The neuron with the smallest Euclidian distance is the one
which best represents the input pattern and is known as the
winner.
Kohonen (Self-Organising) Networks
How does training work?
The
All the
use weights
of a Neighborhood
in the network
during
are initialised
training. to small random values.
An input pattern is presented to the network and the winning neuron
is found. The weights to the winning node are updated using the
following equation.
The updated
weight between
input i and
neuron k
The gain term
Input i
wik(t+1) = wik(t) + G(t) (xi(t) – wik(t))
The original weight between
input i and neuron k
The gain term is a value between 0 and 1 which starts high and as more
patterns are presented to the network is decreased. This promotes large
changes away from the initial state of the network but smaller changes later
when the network is some way through training. This is referred to as course
mapping and fine mapping.
Kohonen (Self-Organising) Networks
How does training work?
The use of a Neighborhood during training.
Usually the weights of more neurons than the winner are changed.
Once the winner has been identified a neighbourhood of a
predefined shape and size is drawn around it.
x1
x2
Kohonen (Self-Organising) Networks
How does training work?
The
A
real
useoutput
of a Neighborhood
lattice will typically
duringhave
training.
hundreds of neurons.
Usually the weights of more neurons than the winner are changed.
Once the winner has been identified a neighbourhood of a
predefined shape and size is drawn around it. All the neurons in the
neighbourhood have their weights updated to better represent the
input pattern. This promotes clustering of similar results.
The neighbourhood size starts very large (typically the entire
network) and is decreased during training in a similar way to the
gain term.
Kohonen (Self-Organising) Networks
How does training work?
A
Then
real all
output
that is
lattice
left iswill
to present
typically
some
haveinput
hundreds
patterns
of neurons.
to the network with
2) so that the defined areas of the
known
realthe
world
values
(such as
PCO
Over time
lattice
of neurons
will
organise
its self into areas of neurons
lattice
be found
the individual
problem
and toThis
giveself
the organisation
areas real
which can
respond
morefor
highly
to certain input
patterns.
world
values.
gives the
network it’s name and leads to the lattice of nodes often being
referred to as a self organising matrix.
The PCO2 problem on a
302.1
Kohonen network
320.8
295.6
254.3
Now when an input pattern is presented to the network the winning
neuron should be in or close to a defined area so a value can be assigned.
The PCO2 problem on a Kohonen network
The aim of this project is to program, implement and test a Kohonen network
designed to predict PCO2 levels at the surface of the ocean.
The
inputs
Lattice
size
The network has been mainly programmed in Fortran and is made up of
many console programs each with a particular function to perform on the
network. Where ever possible the programs are arbitrary and so such
properties as the number of input in an input pattern can be set for each
network. The latter networks have had 5 inputs in each pattern, these are:
2 for month , 1 latitude, 1 longitude and 1 sea surface temperature.
These where used as they where thought to have the greatest affect on the
PCO2 concentration. To encourage the network to distinguish between
months far apart and close in the calendar and to give a greater emphasis
on the month input it was included twice in each pattern. However one
was an input based on the Sine of the month and one on the Cosine
making the month cyclical.
The PCO2 problem on a Kohonen network
2 data
PCOaim
The
of in
this
the
project
NorthisAtlantic
to program,
drift area
implement
from a and
neural
testnetwork
a Kohonen network
designed to predict PCO2 levels at the surface of the ocean.
Lattice size
Typically a square lattice size of 70x70 neurons was used. Of the 4900
neurons in such a lattice only 18.06% of them will be given exact PCO2
values to report when they are found to be the winner. If an “unclassified”
neuron wins then a function must be employed to find the most
appropriate PCO2 value.
As the network must find the output of every neuron in the lattice to find
the winner the size of the lattice greatly affects the time the network takes
to train or run. To illustrate the properties of lattice sizes several sizes have
been tested. Lattice size
Classed
Apr
May
Jun
Jul
Aug
Sep
Oct Nov Average
This is a table
of Χ where
Χ2=Σ (model-data)
Number of
Data points
2
175x175 (30625)
6.13%
30.58
4.81
4.75
2.78
2.12
2.73
2.16
2.12
6.5080
70x70 (4900)
18.06%
35.68
4.74
6.93
3.33
2.52
3.13
2.24
2.22
7.6000
28x28 (784)
46.68%
31.82
6.16
6.57
3.75
2.62
4.27
2.36
2.31
7.4833
11x11 (121)
86.78%
11.54
8.86
10.70
4.62
2.91
5.00
4.07
2.82
6.3161
4x4 (16)
100.00%
30.49
13.22
14.19
5.58
11.43
10.98
16.52
3.59
13.2505
Using data as model
-
11.54
15.84
15.69
10.57
11.53
9.41
11.77
4.31
11.3331
All data is restricted to the north Atlantic drift area.
The PCO2 problem on a Kohonen network
PCO2 data in the North Atlantic drift area from a neural network
Using Levitus data (which gives a uniform grid of latitude and longitude
with temperature values for any month) a map of PCO2 values can be
constructed. This was done with MatLab.
Each point on the
maps has an inside
colour set by the
PCO2 value given
by the network
and an outside
colour set by the
data the network
was trained on,
this gives a visual
grade of how well
the network
predicts the PCO2
levels.
The network which
created these maps had a
lattice size of 70x70
Created
by a
70x70
lattice
Created
by a
175x175
lattice
Created
by a
28x28
lattice
Created
by a
11x11
lattice
The PCO2 problem on a Kohonen network
Training method
Created
by a
4x4
lattice
The PCO2 problem on a Kohonen network
Training method
Long
As mentioned
Train
3000000
before the 80.00
gain and80.00
the neighbourhood
100.00 75.00size change
during the training
3000000
process.
65.00
65.00
75.00is stepped
60.00 down while the
Usually
the gain
52.00 smoothly.
52.00 60.00
50.00 program looks
neighbourhood3000000
size gets smaller
The training
40.00
40.00these
50.00
33.00of the network.
for a file which3000000
tells it how
to change
properties
30.00
20.00 33.00
The file used in6000000
training the
last networks
was as 20.00
follows.
6000000
20.00 16.00
3000000 20.00
65.00 10.00
65.00 100.00
30.00
3000000
6000000
The number of runs.
This is the number of
times a random input
pattern is chosen and
presented to the
network.
40.00
20.00
40.00
10.00
30.00
20.00
Gain. This is expressed as
a percentage. The gain
goes from the first of
these to the second over
the number of runs.
Mostly it is fixed.
20.00
16.00
Neighbourhood diameter.
This is expressed as a
percent of the lattice
width. The size goes from
the first value to the
second smoothly over the
number of runs.
So two more tests where performed this time changing the training method.
The PCO2 problem on a Kohonen network
Training method
Long Train
3000000
3000000
3000000
3000000
6000000
6000000
80.00
65.00
52.00
40.00
30.00
20.00
80.00 100.00 75.00
65.00 75.00 60.00
52.00 60.00 50.00
40.00 50.00 33.00
20.00 33.00 20.00
10.00 20.00 16.00
Quick Train
1000000 65.00 65.00 100.00 35.00
500000 40.00 40.00 35.00 25.00
1500000 20.00 20.00 25.00 18.00
These are designed to do the same as original training method but with
more or less runs i.e. the patterns are presented to the network more or
less times. The networks both had a 70x70 output lattice so a comparison
with the 70x70 size network trained with the original method is valid.
This is a table
of Χ where
Classed
Apr
May
Jun
Jul
Aug
Sep
Oct
Nov
Average
Long
18.00%
34.83
4.80
6.08
3.45
2.37
3.37
2.07
2.26
7.4038
Original
18.06%
35.68
4.74
6.93
3.33
2.52
3.13
2.24
2.22
7.6000
Quick
19.84%
18.38
4.30
4.99
3.13
2.38
3.24
2.14
2.22
5.0963
Χ2=Σ (model-data)
Number of
Data points
2
Created
by a
70x70
lattice
using
the
Long
method
Created
by a
70x70
lattice
using
the
Original
method
Created
by a
70x70
lattice
using
the
Quick
method