Transcript document

Chapter 11
Neural Networks
http://www.msnbc.msn.com/id/10401930
http://www.msnbc.msn.com/id/15014341/
http://www.msnbc.msn.com/id/10401930
1
2
Neural Network
What is a neural network?
 Neural Networks are a different
paradigm for computing:
 von Neumann machines are based on
the processing/memory abstraction
of human information processing.
 neural networks are based on the
parallel architecture of animal brains.

3
Neural networks are a form of
multiprocessor computer system,
with
 simple processing elements
 a high degree of interconnection
 simple scalar messages
 adaptive interaction between
elements

4
A biological neuron may have as
many as 10,000 different inputs, and
may send its output (the presence or
absence of a short-duration spike) to
many other neurons.
 Neurons are wired up in a 3dimensional pattern.
 Real brains, however, are orders of
magnitude more complex than any
artificial neural network so far
considered.

5
Neural networks are being used:

in investment analysis:
 to attempt to predict the movement of stocks
currencies etc., from previous data. There, they
are replacing earlier simpler linear models.

in signature analysis:
 as a mechanism for comparing signatures
made (e.g. in a bank) with those stored. This is
one of the first large-scale applications of
neural networks in the USA, and is also one of
the first to use a neural network chip.
6

in process control:
 there are clearly applications to be made here: most
processes cannot be determined as computable
algorithms.
In monitoring:
 networks have been used to monitor
– the state of aircraft engines. By monitoring vibration levels
and sound, early warning of engine problems can be given.
– British Rail have also been testing a similar application
monitoring diesel engines.

in marketing:
 networks have been used to improve marketing mail
shots. One technique is to run a test mailshot, and look
at the pattern of returns from this. The idea is to find a
predictive mapping from the data known about the
clients to how they have responded. This mapping is
then used to direct further mailshots.
7
New Applications


New Application areas:
Pen PC's
 PC's where one can write on a tablet, and the writing will be
recognized and translated into (ASCII) text.

Speech and Vision recognition systems
 Not new, but Neural Networks are becoming increasingly part of
such systems. They are used as a system component, in
conjunction with traditional computers.

White goods and toys
 As Neural Network chips become available, the possibility of
simple cheap systems which have learned to recognize simple
entities (e.g. walls looming, or simple commands like Go, or
Stop), may lead to their incorporation in toys and washing
machines etc. Already the Japanese are using a related
technology, fuzzy logic, in this way. There is considerable
interest in the combination of fuzzy and neural technologies.
8
Chapter 11 Contents (1)





Biological Neurons
Artificial Neurons
Perceptrons
Multilayer Neural Networks
Backpropagation
9
Chapter 11 Contents (2)






Recurrent Networks
Hopfield Networks
Bidirectional Associative Memories
Kohonen Maps
Hebbian Learning
Evolving Neural Networks
10
Human Brains
11
12
13




The human brain has a huge number of
synapses. Each of the 1012 neurons (1,000
billion, i.e. 1 trillion) has on average 7,000
synaptic connections to other neurons.
It hast been estimated that the brain of a
three-year-old child has about 1016
synapses (10,000 trillion).
This number declines with age, stabilizing
by adulthood.
Estimates vary for an adult, ranging from
1015 to 5 x 1015 synapses (1,000 to 5,000
trillion).[3]
14
Biological Neurons

The human brain
is made up of
billions of simple
processing units
– neurons.

Inputs are received on dendrites, and if the
input levels are over a threshold, the neuron
fires, passing a signal through the axon to
the synapse which then connects to another
neuron.
15
True Neuron We Know Now
16
Functions of Neurons
Neurons are cells that send and
receive electro-chemical signals to
and from the brain and nervous
system.
 There are about 100 billion neurons
in the brain. There are many more
glial cells; they provide support
functions for the neurons, and are far
more numerous than neurons.

17
Neuron




Neurons are nerve cells that transmit
nerve signals to and from the brain at up to
200 mph.
The neuron consists of a cell body (or
soma) with branching dendrites (signal
receivers) and a projection called an axon,
which conducts the nerve signal.
At the other end of the axon, the axon
terminals transmit the electro-chemical
signal across a synapse (the gap between
the axon terminal and the receiving cell).
The word "neuron" was coined by the
German scientist Heinrich Wilhelm
Gottfried von Waldeyer-Hartz in 1891 (he
also coined the term "chromosome").
18
Types of Neurons
There are many type of neurons.
They vary in size from 4 microns
(.004 mm) to 100 microns (.1 mm) in
diameter.
 Their length varies from a fraction of
an inch to several feet.

19
Axon
The axon, a long extension of a nerve
cell, and take information away from
the cell body.
 Bundles of axons are known as
nerves or, within the CNS (central
nervous system), as nerve tracts or
pathways.
 Dendrites bring information to the
cell body.

20
Myelin
Myelin coats and insulates the axon
(except for periodic breaks called
nodes of Ranvier), increasing
transmission speed along the axon.
 Myelin is manufactured by
Schwann's cells, and consists of 7080% lipids (fat) and 20-30% protein.

21
Cell Body -- Soma
The cell body (soma) contains the
neuron's nucleus (with DNA and
typical nuclear organelles).
 Dendrites branch from the cell body
and receive messages.

22
Neuron

A typical neuron has about 1,000 to
10,000 synapses (that is, it
communicates with 1,000-10,000
other neurons, muscle cells, glands,
etc.).
23
Life Span of a Neuron
Unlike most other cells, neurons
cannot re-grow after damage (except
neurons from the hippocampus).
 Fortunately, there are about 100
billion neurons in the brain.

24
GLIAL CELLS




GLIAL CELLS Glial cells make up 90
percent of the brain's cells.
Glial cells are nerve cells that don't carry
nerve impulses.
The various glial (meaning "glue") cells
perform many important functions,
including: digestion of parts of dead
neurons, manufacturing myelin for
neurons, providing physical and nutritional
support for neurons, and more.
Types of glial cells include Schwann's
Cells, Satellite Cells, Microglia,
Oligodendroglia, and Astroglia.
25
Artificial Neurons
26
Artificial Neurons (1)




Artificial neurons are based on biological neurons.
Each neuron in the network receives one or more
inputs.
An activation function is applied to the inputs,
which determines the output of the neuron – the
activation level.
The charts on
the right show
three typical
activation
functions.
27
Artificial Neurons (2)

A typical activation function works as
follows:

Each node i has a weight, wi associated with
it. The input to node i is xi.
t is the threshold.
So if the weighted sum of the inputs to the
neuron is above the threshold, then the
neuron fires.


28
Perceptrons (1)



A perceptron is a single neuron that
classifies a set of inputs into one of two
categories (usually 1 or -1).
If the inputs are in the form of a grid, a
perceptron can be used to recognize visual
images of shapes.
The perceptron usually uses a step
function, which returns 1 if the weighted
sum of inputs exceeds a threshold, and –1
otherwise.
29
Perceptrons (2)

The perceptron is trained as follows:
First, inputs are given random weights
(usually between –0.5 and 0.5).
An item of training data is presented. If the
perceptron mis-classifies it, the weights are
modified according to the following:

e is the size of the error, and a is the learning
rate, between 0 and 1.
30
Perceptrons (3)



Perceptrons can only classify linearly
separable functions.
The first of the following graphs shows a
linearly separable function (OR).
The second is not linearly separable
(Exclusive-OR).
31
Multilayer Neural Networks



Multilayer neural networks can classify a range
of functions, including non linearly separable
ones.
Each input layer neuron
connects to all neurons
in the hidden layer.
The neurons in the
hidden layer connect to
all neurons in the output
A feed-forward network
layer.
32
Backpropagation (1)



Multilayer neural networks learn in the same way
as perceptrons.
However, there are many more weights, and it is
important to assign credit (or blame) correctly
when changing weights.
Backpropagation networks use the sigmoid
activation function, as it is easy to differentiate:
33
Backpropagation (2)



For node j, Xj is the output
Yj is the output
n is the number of inputs to node j
j is the threshold for j
After values are fed forward through the network,
errors are fed back to modify the weights in order
to train the network.
For each node, we calculate an error gradient.
34
Backpropagation (3)





For a node k in the output layer, the error ek is the
difference between the desired output and the
actual output.
The error gradient for k is:
Similarly, for a node j in the
hidden layer:
Now the weights are
updated as follows:
 is the learning rate, (a positive number
below 1)
35
Recurrent Networks



Feed forward networks do not have
memory.
Recurrent networks can have connections
between nodes in any layer, which enables
them to store data – a memory.
Recurrent networks can be used to solve
problems where the solution depends on
previous inputs as well as current inputs
(e.g. predicting stock market movements).
36
Hopfield Networks







A Hopfield Network is a recurrent network.
Use a sign activation function:
If a neuron receives a 0 as an input it does not
change state.
Inputs are usually represented as matrices.
The network is trained to represent a set of
attractors, or stable states.
Any input will be mapped to an output state
which is the attractor closest to the input.
A Hopfield network is autoassociative – it can
only associate an item with itself or a similar37
one.
Bidirectional Associative Memories
A BAM is a heteroassociative memory:
Like the brain, it can learn to
associate one item with another
completely unrelated item.
 The network consists of two fully
connected layers of nodes – every node
in one layer is connected to every node
in the other layer.

38
Kohonen Maps



An unsupervised learning system.
Two layers of nodes: an input layer and a
cluster (output) layer.
Uses competitive learning:
 Every input is compared with the weight vectors of each
node in the cluster node.
 The node which most closely matches the input, fires.
This is the classification of the input.
 Euclidean distance is used.
 The winning node has its weight vector modified to be
closer to the input vector.
39
Kohonen Maps
It is also called self-organizing feature
maps.
 It uses the winner-take-all algorithm.
 -- only one neuron provides the output
of the network in response to a given
input.
 It will be the neuron that has the
highest activation level.

40
The input and the weight
1 


wi  2 
 1
3 


xj    1


2


41
The input and the weight
di 
n
 wi  xj
2
=4
j 1
42
The neuron for which that has
the smallest di is the winner
This neuron will have its weight vector
updated as follows:
Wij  Wij + α(xj – Wij)
43
44
Kohonen Maps (example)
The nodes in the cluster layer are
arranged in a grid, as shown:
The diagram on the left
shows the training
data.
Initially, the weights are arranged
as shown here:
45
Kohonen Maps (example)
After training, the weight vectors
have been rearranged to match
the training data:
46
Hebbian Learning (1)
Hebb’s law:
“When an axon of cell A is near enough to
excite a cell B and repeatedly or persistently
takes part in firing it, some growth process
or metabolic change takes place in one or
both cells such that A’s efficiency, as one of
the cells firing B, is increased”.
 Hence, if two neurons that are connected
together fire at the same time, the weights
of the connection between them is
strengthened.

47
Hebbian Learning (2)

The activity product rule is used to modify the
weights of a connection between two nodes that
fire at the same time:

 is the learning rate; xi is the input to node i and yi
is the output of node i.
Hebbian networks usually also use a forgetting
factor, which decreases the weight of the
connection between if two nodes if they fire at
different times.

48
Evolving Neural Networks
Neural networks can be susceptible
to local maxima.
 Evolutionary methods (genetic
algorithms) can be used to determine
the starting weights for a neural
network, thus avoiding these kinds of
problems.

49