Transcript No Slide Title

Introduction to Neural Net
CS480 Computer Science Seminar
Fall 2002
Characteristics not present in von Neumann
or modern parallel computers
Typical tasks a neural net
is capable of performing
•
•
•
•
•
•
•
Pattern classification
Clustering/categorization
Function approximation
prediction/forecasting
Optimization
Content-addressable memory
control
Kohonen’s Self-Organizing Map
Neural network definition
• A machine that is designed to model the way in
which the brain performs a particular task or
function of interest.
• It is a massively parallel distributed processor that
has a natural propensity for storing experiential
knowledge and making it available for use. It
resembles brain in two respects: 1. Knowledge is
acquired through learning process, 2. Interneuron
connection strengths known as synaptic weights
are used to stored the knowledge.
Some interesting facts
•
•
•
•
Silicon gate speed: 10-9 sec
Neural event: 10-3 sec
# gates on Pentium III: < 107
# neurons in human brain: 1010 with 60x1012
synapses or connections
• Power consumption: for human brain---10-16
joules per operations per second; for today's best
computer---10-6 joules per operations per second
Some interesting facts (cont’d)
• The brain is a highly complex, nonlinear,
and parallel processor.
• Brain is superior in performing pattern
recognition, perception, and motor control),
e.g., it takes a brain 100-200 msec to
recognize a familiar face embedded in an
unfamiliar scene (will take days for the
computer to do the similar tasks)
Some interesting facts (cont’d)
• Brain has the ability to build up its own
rules through “experience”over the years
with most dramatic development in the first
two years from birth (about 106 synapses
formed per second)
Neural net architecture
• Feed-forward neural net
– Single layer
– Multi-layer
• Recurrent neural net
– There are closed loops (feedback) paths.
Training the neural net
• Training: setting the weights
– Supervised
– Unsupervised
Supervised learning
• Supervised training: presenting a sequence of
training vectors or patterns, each is associated with
an associated target output vector. The weight is
then adjusted according to a learning algorithm.
• Examples
– Pattern classification: the output is either 1 or –1
(belong or not belong to the category)
– Patten association: to associate a set of input vectors
with a corresponding set of output vectors (the neural
net is called associative memory). After training, the the
neural net can identify input vectors that are sufficiently
close to (but not necessary the same as the input
vectors).
History of neural net
• 1940s: McCulloch-Pitts neurons (perform
logic/combinational functions); introduced the
idea of “threshold”.
• 1949: Hebb learning algorithm---if two neurons
are active, the connection strength (weight) should
be increased.
• 1950s and 1960s: the first golden age of neural net
– Perceptrons (Rosenblatt) for pattern recognition
– Adaline (adaptive linear neurons).
• 1970s: the quiet years (research funding stopped)
– Kohonen, Anderson, Grossberg, and Carpenter
History of neural net (cont’d)
• 1980s:
– Backpropagation (of errors) learning algorithm
– Hopfield nets: capable of solving constraint satisfaction
problems such as traveling salesman problem
– Neocognitron: a self-organizing neural for pattern
recognition (position or rotation-distorted characters)
– Boltzmann machine: non-deterministic neural net
(weights or activations are changed on the basis of a
probabilistic density function)
– Hardware implementation: digital and analog
implementations using using VLSI technology
Neural net architecture
• Feed-forward neural net
– Single layer
– Multi-layer
• Recurrent neural net
– There are closed loops (feedback) paths.
Training the neural net
• Training: setting the weights
– Supervised
– Unsupervised
Supervised learning
• Supervised training: presenting a sequence of
training vectors or patterns, each is associated with
an associated target output vector. The weight is
then adjusted according to a learning algorithm.
• Examples
– Pattern classification: the output is either 1 or –1
(belong or not belong to the category)
– Patten association: to associate a set of input vectors
with a corresponding set of output vectors (the neural
net is called associative memory). After training, the net
is the neural net can identify input vectors that are
sufficiently close to (but not necessary the same as the
input vecotrs)
A simple example: McCulloch-Pitts
neurons that perform logic operations
• Presumption:
– Binary activation: neuron either fires or does not fire
– Neurons are connected by directed, weighted paths
– A connection path is excitatory if the weight on the path
is positive; otherwise, its inhibitory
– Each neuron has a fixed threshold such that if the net
input into a neuron is greater than the threshold, the
neuron fires.
– The threshold is set so that inhibition is absolute, i.e.,
any nonzero inhibitory input will prevent the neuron
from firing
– It takes one time step for a signal to pass over one
connection link
McCulloch-Pitts Neuron: AND
function
The threshold on y is 2
McCulloch-Pitts Neuron: OR
function
McCulloch-Pitts Neuron: OR
function
The threshold on y is 2
McCulloch-Pitts Neuron: ANDNOT function
McCulloch-Pitts Neuron: ANDNOT function
The threshold on y is 2
McCulloch-Pitts Neural net:
XOR function
The threshold on y is 2
McCulloch-Pitts Neural net:
XOR function
The threshold on y is 2
McCulloch-Pitts Neural net application:
modeling the perception of heat and cold
• Interesting physiological phenomenon: if
cold stimulus is applied to a person’s skin
for a very short period of time, the person
will perceive heat. If the same stimulus is
applied for a longer period, the person will
perceive cold.
McCulloch-Pitts Neural net: modeling the perception of heat
and cold (time step functions are included in the modeling)
A cold stimulus applied for one
time step
A cold stimulus applied for two
time steps
Another example: character
recognition
Noisy input
Possible topics for further
investigation
•
•
•
•
Current development and applications
Architectures
Training algorithms
Fuzzy logic and fuzzy-neural systems
Supervised learning
• Supervised training: presenting a sequence of
training vectors or patterns, each is associated with
an associated target output vector. The weight is
then adjusted according to a learning algorithm.
• Examples
– Pattern classification: the output is either 1 or –1
(belong or not belong to the category)
– Patten association: to associate a set of input vectors
with a corresponding set of output vectors (the neural
net is called associative memory). After training, the net
is the neural net can identify input vectors that are
sufficiently close to (but not necessary the same as the
input vecotrs)
McCulloch-Pitts Neuron: AND
function
McCulloch-Pitts Neuron: AND
function
The threshold on y is 2
McCulloch-Pitts Neuron: OR
function
McCulloch-Pitts Neuron: OR
function
The threshold on y is 2
McCulloch-Pitts Neuron: ANDNOT function
McCulloch-Pitts Neuron: ANDNOT function
The threshold on y is 2
McCulloch-Pitts Neural net:
XOR function
The threshold on y is 2
McCulloch-Pitts Neural net:
XOR function
The threshold on y is 2
McCulloch-Pitts Neural net application:
modeling the perception of heat and cold
• Interesting physiological phenomenon: if
cold stimulus is applied to a person’s skin
for a very short period of time, the person
will perceive heat. If the same stimulus is
applied for a longer period, the person will
perceive cold.
McCulloch-Pitts Neural net: modeling the perception of heat
and cold (time step functions are included in the modeling)
A cold stimulus applied for one
time step
A cold stimulus applied for two
time steps
Another example: character
recognition
Noisy input
Possible topics for further
investigation
•
•
•
•
Current development and applications
Architectures
Training algorithms
Fuzzy logic and fuzzy-neural systems