Transcript CS-485: Capstone in Computer Science

CS-485: Capstone in
Computer Science
Artificial Neural Networks and their
application in Intelligent Image
Processing
Spring 2010
1
Organizational Details
• Class Meeting:
12:25-3:45pm Tuesday, SCIT213
• Class webpage http://www.eagle.tamut.edu/faculty/igor/CS-485.htm
• Instructor: Dr. Igor Aizenberg
• Office: Science and Technology Building, 104C
• Phone (903 334 6654)
• e-mail: [email protected]
• Office hours:
•
Monday, Thursday 10-30 – 6-30
• Tuesday, Wednesday 4-30 – 6-30
2
Text Book
1) I. Aizenberg,
“Advances in Neural Networks”,
University of Dortmund, 2005,
Class notes (available from the class
webpage)
2) Additional materials will also be
available from the class webpage
3
Introduction to Neural Networks
Artificial Intellect:
Who is stronger and why?
Applied Problems:
•Image, Sound, and Pattern recognition
•Decision making
Knowledge discovery
Context-Dependent Analysis
…
NEUROINFORMATICS
- modern theory about principles and new
mathematical models of information
processing, which based on the biological
prototypes and mechanisms of human
brain activities
4
Applied Problems
Natural language understanding
(Translation of the texts)
Decision Making
Recognition of Images
Learning and Adaptation
Reasoning and
Prediction
Knowledge Discovery
Cognitive analysis
Team behavior
Fuzzy Logic
5
The History of Neuroscience
Renaissance of connectionism from the papers by Hopfield, and
popularizing the back-propagation algorithm for multiplayer feedforward networks
End of Perceptron era:
Work “Perceptron” by Minsky and Papert
Frank Rosenblatt invented the modern
“perceptron” style of NN, composed of
trainable threshold units
1949
1948
1943
1969
1957
1952
1951
1982
Ashby puts the idea that intelligence
could be created by the use of
“homeostatic” devices which learn
through a kind of exhaustive search
Minsky’s builts the first actual neural network learning
system
Hebb hypothesis that human and animal long-term memory is
mediated by permanent alterations in the synapses.
Notion of Wiener about key role of connectionism and feedback loops
as a model for learning in neural networks
McCulloch and Pitts introduced the fundamental ideas of analyzing neural
activity via thresholds and weighted sums
6
ANN as a Brain-Like Computer
NN as an model of
brain-like Computer
Brain
The human brain is still not well
understood and indeed its
behavior is very complex!
There are about 10 billion
neurons in the human cortex and
60 trillion synapses of connections
The brain is a highly complex,
nonlinear and parallel computer
(information-processing system)
An artificial neural network (ANN) is a
massively parallel distributed processor that
has a natural propensity for storing
experimental knowledge and making it
available for use. It means that:

Knowledge is acquired by the network
through a learning (training) process;
 The strength of the interconnections
between neurons is implemented by means
of the synaptic weights used to store the
knowledge.
The learning process is a procedure of the
adapting the weights with a learning
algorithm in order to capture the knowledge.
On more mathematically, the aim of the
learning process is to map a given relation
between inputs and output (outputs) of the
7
network.
Intelligent Data Analysis in Engineering
Experiment
Interpretation
and
Decision Making
Data
Analysis
Data
Acquisition
Signals
&
parameters
Data
Acquisition
Rules
&
Knowledge
Productions
Characteristics
&
Estimations
Adaptive Machine Learning
via Neural Network
Data
Analysis
Decision
Making
Knowledge
Base
8
Mathematical Interpretation of
Classification in Decision Making
1. Quantization of pattern space
into p decision classes
xi
n
f : n   p
F   f t 
yi
p
m1
m2
mp
m3
2. Mathematical model of
quantization:
Input Patterns
 x11 
 1 
x
xi   2 
  
 1 
xn 
“Learning by Examples”
Response:
 y11 
 1 
y
yi   2 
  
 1 
 yn 
9
Learning via Self-Organization Principle
Self-organization – basic principle
of learning:
Structure reconstruction
Responce
Input Images
The learning
involves
change of
structure
Neuroprocessor
Learning Rule
Teacher
10
Applications of Artificial Neural
Networks
Advance
Robotics
Machine
Vision
Intelligent
Control
Technical
Diagnistics
Intelligent
Data Analysis
and Signal
Processing
Artificial
Intellect with
Neural
Networks
Image &
Pattern
Recognition
Intelligentl
Medicine
Devices
Intelligent
Security
Systems
Intelligent
Expert
Systems
11
What we will learn and do?
Artificial Neural Networks
And Its Applications
You will learn:
Contemporary theoretical principles and
paradigms of Neuroinformatics,
Mathematical models and algorithms of
neural network techniques for experimentation,
Applications of Neuroinformatics to
engineering and sciences problems,
Computer-Aided Technology for
Instrumentation
Theory
Practice
Self-Paced
Work
12
What we will learn and do?
• General principles of artificial neural networks
• General principles of learning algorithms
• Feedforward neural network and
backpropagation learning
• Multi-valued neurons and a feedforward neural
network based on multi-valued neurons
• Basic ideas of image processing
• Edge detection on noisy images using a neural
network
13
Symbol Manipulation or Pattern
Recognition ?
Ill-Formalizable Tasks:
•Sound and Pattern recognition
•Decision making
•Knowledge discovery
•Context-Dependent Analysis
What is difference
between
human
brain and traditional
computer via specific
approaches
to
solution
of
illformalizing
tasks
(those tasks that can
not be formalized
directly)?
Symbol manipulation
Which way of
imagination is
best for you ?
Pattern recognition
Dove flies
Lion goes
Tortoise scrawls
Donkey sits
Shark swims
14
Principles of Brain Processing
How our brain
manipulates with
patterns ?
A process of pattern
recognition and pattern
manipulation is based
on:
Massive parallelism
Connectionism
Brain computer as an information
or signal processing system, is
composed of a large number of a
simple processing elements, called
neurons. These neurons are
interconnected by numerous direct
links, which are called connection,
and cooperate which other to
perform a parallel distributed
processing (PDP) in order to soft a
desired computation tasks.
Brain computer is a highly
interconnected neurons system in
such a way that the state of one
neuron affects the potential of the
large number of other neurons
which are connected according to
weights or strength. The key idea
of such principle is the functional
capacity of biological neural nets
determs mostly not so of a single
neuron but of its connections
Associative distributed
memory
Storage of information in a
brain is supposed to be
concentrated
in
synaptic
connections of brain neural
network, or more precisely, in
the pattern of these connections
and strengths (weights) of the
synaptic connections.
15
Brain-like Computer
Artificial Neural Network – Mathematical
Paradigms of Brain-Like Computer
The new paradigm of computing
mathematics consists of the
combination of such artificial
Neurons and Neural Net
neurons into some artificial
neuron net.
Brain-Like Computer
Brain-like computer –
is a mathematical model of humane-brain
principles of computations. This computer consists
of those
elements which can be called the
biological neuron prototypes, which are
interconnected by direct links called connections
and which cooperate to perform parallel
distributed processing (PDP) in order to solve a
desired computational task.
?
16
Principles of Neurocomputing
Connectionizm
NN is a highly interconnected structure in such a way that the state of one
neuron affects the potential of the large number of another neurons to which
it is connected accordiny to weights of connections
Not Programming but Training
NN is trained rather than programmed to perform the given task
since it is difficult to separate the hardware and software in the
structure. We program not solution of tasks but ability of learning to
solve the tasks
 w11
w
 11
 w11

 w11
w11
w11
w11
w11
w11
w11
w11
w11
w11 
w11 
w11 

w11 
Distributed Memory
NN presents an distributed memory so that changing-adaptation of
synapse can take place everywhere in the structure of the network.
17
Principles of Neurocomputing
Learning and Adaptation
NN are capable to adapt themselves (the synapses connections
between units) to special environmental conditions by changing
their structure or strengths connections.
y   x
2

Non-Linear Functionality
Every new states of a neuron is a nonlinear function of the
input pattern created by the firing nonlinear activity of the
other neurons.
Robustness of Assosiativity
NN states are characterized by high robustness or
insensitivity to noisy and fuzzy of input data owing to use of
a highly redundance distributed structure
18