Transcript Neural Network

Neural Network
Ming-Feng Yeh (葉明豐)
Department of Electrical Engineering
Lunghwa University of Science and Technology
E-mail: [email protected]
Office: F412-III Tel: #5518
COURSE OBJECTIVE
This course gives an introduction to basic
neural network architectures and learning
rules.
Emphasis is placed on the mathematical
analysis of these networks, on methods of
training them and on their application to
practical engineering problems in such
areas as pattern recognition, signal
processing and control systems.
Ming-Feng Yeh
2
SYLLABUS
Textbook: Hagan, Demuth, Beale,
Neural Network Design,
PWS Publishing Company
Midterm Exam: 30%
Final Exam: 30%
Projects: 40%
Ming-Feng Yeh
3
CONTENTS
Ch 1.
Introduction
Ch 2.
Neuron Model & Neural Architecture
Ch 3&4. Perceptron (感知機) Learning Rule
Ch 7.
Supervised (監督式) Hebbian Learning
Ch 10. Widrow-Hoff Learning
Ch 11&12. Back-propagation (倒傳遞)
Ch 13. Associative (關聯) Learning
Ch 14. Competitive (競爭) Networks
Ch 15. Grossberg Networks
Ch 16. Adaptive Resonance (自適應) Theory
Ch 18. Hopfield Network
Ming-Feng Yeh
4
Information
Review
Ch 5 – Signal and Weight Vector Spaces
Ch 6 – Linear Transformations for
Neural Networks
Ming-Feng Yeh
5
CHAPTER 1
Introduction
Ming-Feng Yeh
6
Objectives
As you read these words you are using a
complex biological neural network. You
have a highly interconnected set of 1011
neurons to facilitate your reading, breathing,
motion and thinking.
In the artificial neural network, the neurons
are not biological. They are extremely simple
abstractions of biological neurons, realized as
elements in a program or perhaps as
circuits made of silicon.
Ming-Feng Yeh
7
History -1
Pre-1940: von Hemholtz, Mach & Pavlov


General theories of learning, vision, conditioning
No specific mathematical models of neuron operation
1940s: Hebb, McCulloch & Pitts


Mechanism for learning in biological neurons (Hebb)
Neural-like networks can compute any arithmetic or logical
function (McCulloch & Pitts)
1950s: Rosenblatt, Widrow & Hoff


First practical networks and learning rules: the perception
network and associated learning rule (Rosenblatt) &
Widrow-Hoff learning rule
Can not successfully modify their learning rules to train the
more complex networks.
Ming-Feng Yeh
8
History -2
1960s: Minsky & Papert


Demonstrated limitations of existing neural networks
Neural network research was largely suspended
1970s: Kohonen, Anderson & Grossberg


Kohonen and Anderson independently and separately
developed neural networks that could as memories
Self-organizing networks (Grossberg)
1980s: Hopfield, Rumelhart & McClelland


The use of statistical mechanics to explain the operation of
recurrent network: an associative memory (Hopfield)
Backpropagation algorithm (Rumelhart & McClelland)
Ming-Feng Yeh
9
Applications
The applications are expanding
because neural networks are good at
solving problems, not just in
engineering, science and
mathematics, but in medicine,
business, finance and literature as
well.
Ming-Feng Yeh
10
Biological Inspiration
Human brain consists of a large number
(about 1011) of highly interconnected
elements (about 104 connections per
element) called neurons (神經元).
Three principle components are the
dendrites, the cell body and the axon.
The point of contact is called a synapse.
Ming-Feng Yeh
11
Biological Neurons
Dendrites
Axon
Cell Body
Soma
Synapse
Ming-Feng Yeh
Dendrites(樹突): carry
electrical into the cell body
Cell Body(細胞體): sums and
thresholds these incoming
signals
Axon(軸突): carry the signal
from the cell body out to other
neurons
Synapse(突觸): contact
between an axon of one cell
and a dendrites of another cell
12
Neural Networks
Neural Networks: a promising new
generation of information processing systems,
usually operate in parallel, that demonstrate
the ability to learn, recall, and generalize
from training patterns or data.
Basic models, learning rules, and distributed
representations of neural networks will be
discussed.
Ming-Feng Yeh
13
補充資料
Artificial neural network可譯為類神經網路或人
工神經網路，是指模仿生物神經網路的一種資
訊處理系統。
類神經網路是一種計算系統，包括軟體與硬體，
它使用大量簡單的相連人工神經元來模仿生物
神經網路的能力。人工神經元是生物神經元的
簡單模擬，它從外界環境或其它人工神經元取
得資訊，並加以簡單的運算，並輸出其結果到
外界環境或其它人工神經元。
Ming-Feng Yeh
14
Fuzzy Logic
Fuzzy set theory was first proposed by
Lotfi Zadeh in 1965.
A mathematical way to represent
vagueness in linguistics
A generalization of classical set theory
Ming-Feng Yeh
15
Fuzzy Systems v.s.
Neural Networks
Fuzzy logic is based on the way the brain
deals with inexact information.
Neural networks are modeled after the
physical architecture of the brain.
Fuzzy systems and neural networks are both
numerical model-free estimator and
dynamical systems.
They share the common ability to improve the
intelligence of systems working in an
uncertain, imprecise and noisy environment.
Ming-Feng Yeh
16
Machine Intelligence
Neural networks provide fuzzy
systems with learning ability.
Fuzzy systems provide neural
networks with a structure framework
with high-level fuzzy IF-THEN rule
thinking and reasoning.
Ming-Feng Yeh
17
Fuzzy Neural
Integrated System
Neural fuzzy systems: use of neural
networks as tools in fuzzy models.
Fuzzy neural networks: fuzzification of
conventional neural network models.
Fuzzy-neural hybrid systems:
incorporation of fuzzy logic technology
and neural networks into hybrid
systems.
Ming-Feng Yeh
18
Soft / Hard Computing
Hard computing whose prime desiderata are
precision, certainty, and rigor.
Soft computing is tolerant of imprecision,
uncertainty, and partial truth. (Lotfi Zadeh)
The primary aim of soft computing is to exploit
such tolerance to achieve tractability, robustness,
a high level of machine intelligence, and a low
cost in practical applications.
Fuzzy logic, neural networks (including CMAC),
probabilistic reasoning (genetic algorithm,
evolutionary programming, and chaotic systems)
Ming-Feng Yeh
19
Soft Computing
Methodology
Strength
Neural network
Learning and adaptation
Fuzzy set theory
Knowledge representation
via fuzzy if-then rule
Systematic random search
Genetic algorithm
and simulated
annealing
Conventional AI
Ming-Feng Yeh
Symbolic manipulation
20
Computational Intelligence
Fuzzy logic, neural network, genetic algorithm,
and evolutionary programming are also
considered the building blocks of
computational intelligence. (James Bezdek)
Computational intelligence is low-level
cognition in the style of human brain and is
contrast to conventional (symbolic) artificial
intelligence (AI).
Ming-Feng Yeh
21