Transcript Introduction to Neural Networks

IE 585
Introduction to Neural Networks
Modeling Continuum
High Cost /
Low Error
Low Cost /
High Error
Unarticulated
Wisdom
Articulated
Qualitative
Models
Empirical
Categorical
Models
Empirical
Continuous
Models
Theoretic
(First Principles)
Models
2
Rise of Empirical Models
•
•
•
•
Sensoring - lots of data
Fast computing
Computing available on site
More complicated systems - do not
adhere to simple models
• Easy to use software
3
Typical Empirical Models
•
•
•
•
linear regression
splines
nearest neighbor clustering
neural networks
4
What is a Neural Net?
• An NN is a network of many simple processors
(“units, neurons”), each possibly having a
small amount of local memory. The units are
connected by communication channels
(“connections”) which usually carry numeric
data, encoded by any of various means. The
units operate only on their local data and on
the inputs they receive via the connections.
Usenet newsgroup comp.ai.neural-nets
5
What is a Neural Net?
• An NN is a massively parallel distributed processor
that has a natural propensity for storing experiential
knowledge and making it available for use. It
resembles the brain in two respects:
1. Knowledge is acquired by the network through a
learning process.
2. Interneuron connection strengths known as synaptic
weights are used to store the knowledge.
Haykin (1994)
6
Objectives of Neural Nets
•
•
•
•
High Computing Speed
Large Memory Capacity
Adaptive Learning
Fault Tolerance
7
Neural Network Predictive
Models - Advantages
• Can accommodate non-linear relationships
with interactions among variables
• Generalize well even for noisy and imprecise
data
• No assumption of analytical function or
theoretic relation needed
• User friendly software available
• Computationally very fast, once built
8
Neural Network Predictive
Models - Disadvantages
• Strongly data dependent
• No statistical interpretation of significance or
confidence
• Difficult to build and validate properly - too
many choices, too little general guidance,
misleading validation results
9
How Do Neural Networks Work?
• Inspired by the biological brain
• Consist of small, but numerous, parallel
elements - weighted connections (synapses)
and summing nodes (neurons)
• “Learn” relationships through repeated
calculations called “training”
• Remain fixed after training to be applied to
new data
10
Biological Neuron
11
How are Signals Transmitted?
12
Elements of Neural Networks
x1
w1
w2
x2
.
.
.
xn
y
y = f (  wixi )
i
wn
y
Simple Summing
Node (Neuron)
1
0.5
0
 wi xi
Non-linear
Transfer
Function
13
Typical Neural Network
Error Feedback
Weighted Synapses
During Training
I
N
P
U
Output
Layer
Neural
Network
Output
T
S
Hidden
Layer
Input
Layer
14
Terminology
• Neurons / nodes / units / cells / processing elements
(PEs)
• Transfer / activation function
• Connections / links / synapses
• Weights / bias (fixed input of 1)
• Feedforward / feedback
• Input / output vectors / patterns
• Self organizing (unsupervised) / supervised
• Training / testing data sets
15
Biological vs Artificial Neural
Networks
• Biological neurons are all excitatory
(positive) or inhibitory (negative) - ANN
neurons can be mixed
• Biological neurons operate
asynchronously - ANN neurons usually
synchronize by layer
• Biological neurons transmit signals at
varying rates but ANN use a single rate
16
Biological vs Artificial Neural
Networks
• There are many specialized biological
neurons - ANN neurons tend to be generic
• Biological neurons work through chemical /
electrical transmission (“wet ware”)
• Biological neurons are much slower but
there are many, many more of them (~
1011 neurons with 104 synapses per
neuron!)
17
Types of Neural Nets
•
•
•
•
Supervised
Unsupervised
Associate
Optimization
18
Common Neural Net Applications
• Pattern classification / recall
– medical
– defense
– manufacturing quality
– machine vision / postal
– speech recognition
– security detection
– noise removal
19
Common Neural Net Applications
• Clustering / compression
– data mining
– signal processing
– space exploration applications
– speech recognition
20
Common Neural Net Applications
• Prediction / simulation
– financial / stock market
– music composition
– utility usage
– fault / degradation detection
– sunspots
21
Common Neural Net Applications
• Control - real time / on line
– robots
– vehicles
– manufacturing
• Control - off line
– batch manufacturing
– process optimization
22
Common Neural Net Applications
• Optimization
– traveling salesman
– routing
– scheduling
– facility location
23
Cool Neural Net Web Sites
http://www.csse.monash.edu.au/~app/CSE5301/index.html
Detailed class notes and some matlab code.
http://www.geocities.com/CapeCanaveral/1624/
C source code for lots of neural nets.
24