Transcript Associative Learning in Hierarchical Self

Associative Learning in
Hierarchical Self Organizing
Learning Arrays
Janusz A. Starzyk, Zhen Zhu, and Yue Li
School of Electrical Engineering and
Computer Science
Ohio University, Athens, OH 45701,
U.S.A.
Organization
 Introduction
• Network structure
• Associative learning
• Simulation results
• Conclusions and future work
2
Introduction - SOLAR
• SOLAR –Self Organizing Learning
Array
– A concept inspired by the structure of
biological neural networks
– Regular, two or three-dimensional
array of identical processing cells,
connected to programmable routing
channels
– Self-organizing in individual cells and
the network structure
3
Introduction - SOLAR
16
14
• SOLAR vs. ANN
– Deep multi-layer structure
with sparse connections
– Self organized neuron
functions
– Dynamic selection of
interconnections
12
10
8
6
4
2
0
0
1
2
3
4
5
6
7
A 15 X 7 SOLAR
– Hardware efficiency
– Online learning
A 15 X 3 ANN
4
Introduction - Associative Learning
• Hetero-associative (HA)
– To associate different
types of input signals
e. g. a verbal command
with an image
• Auto-associative (AA)
– To recall a pattern from a
fractional part
e. g. an image with
missing part
•The proposed approach:
An associative learning network in a hierarchical SOLAR
structure - HA and AA
www.idi.ntnu.no/~keithd/ classes/advai/lectures/assocnet.ppt
5
Introduction
 Network structure
Associative learning
Simulation results
Conclusions and future work
6
Network Structure
• Two or three dimensional
multi-layer regular structure
– 2 D networks:
Input span – rows and network
depth – columns
– 3 D networks, better for image
applications
• “Small world” network
connection
– More local connections with short
Euclidean distance
(as in biological neural networks)
– Few distant connections
7
Network Structure
• Hierarchical network connection
– Each neuron only connects to the preceding layer
• Neuron connections:
– Redundant initial inputs
Inputs
to be refined in learning
– 2 inputs (I1 / I2) and
1 output O
– Feed-forward
and feed-back links
Depth
8
Introduction
Network structure
 Associative learning
Simulation results
Conclusions and future work
9
Associative learning – feed-forward
• Semi-logic inputs and internal signals:
– scaling from 0 to 1, 0.5 = unknown;
– 0 = determinate low, 1 = determinate high;
– > 0.5 = weak high, < 0.5 = weak low.
• The I1/I2 relationship is are found with:
– P(I1 is low), P(I1 is high), P(I2 is low) & P(I2 is high)
– The joint probabilities, e.g. P(I2 is low | I1 is low)
– The conditional probabilities, e.g.
P(I 2 is low | I1 is low) 
P(I 2 is low & I1 is low)
P(I1 is low)
10
Associative learning – feed-forward
• Compare the conditional probabilities against a
confidence interval:
CI 
2(1  P(I 2 | I1 ))
N
, where N is # samples.
• If P(I2 | I1) – CI > threshold, I2 can be implied from I1
11
Associative learning – feed-forward
A neuron is an associative neuron if I1 can be
implied from I2 and I2 can be implied from I1,
otherwise it is a transmitting neuron
Six possible I1/I2
distributions for
associative neurons.
f1
f2
f3
f4
f5
f6
A semi-logic function
is designed for each
one.
12
Associative learning – feed-forward
• In an associative neuron:
– Functions are designed for data transformation and
feedback calculation.
– f1 to f4 – for data centered in one dominant quadrant.
– f5 to f6 – for data mainly in two quadrants
0, if I1  0.5
1, if I1  0.5 & I 2  0.5
f1 (I1 , I 2 )  
f 5 (I1 , I 2 )  
0, else
1, if I1  0.5
– Neuron output is 0.5 with an unknown input.
0.5, if I1  0.5 or I 2  0.5
O
 f (I1 , I 2 ), otherwise
13
Associative learning – feed-forward
• In a transmitting neuron:
– The input with higher entropy (dominant input)
is transmitted to O, with the other is ignored.
– I1 is the dominant input if
abs(P(I1 is low) - P(I1 is high))  abs(P(I 2 is low) - P(I 2 is high))
• O may be an input to other neurons.
• O receives feedback Of from connected neurons,
which generate feedback to its inputs.
14
Associative learning – feedback
• The network generates feedback to the unknown
inputs through association.
Inputs
unknown
N2
N1
N3
N4
known
depth
a
b
15
Associative learning – feedback
• N1 -> transmitting neurons
– Of is passed back to the input.
• N2 -> associative neurons with determined inputs
– Feedback takes no effect and information passes forward.
• N3 -> associative neurons with active feedback and inactive input(s)
– Of creates feedbacks I1f and I2f through the function f,
– These neurons only pass information backwards.
• N4 -> actively associating neurons with inactive feedback
– If one of their inputs is inactive, it will be overwritten based on its
association with the other input and the neuron’s function f.
16
Associative learning – feedback
• Calculation of the feedback (using f5):
 w 5 , if Of  0.5
1-w , if O  0.5

f
I1f   5
I 2 , if I1  0.5
I1 , otherwise
With an active output
feedback, I1f is determined
based on f5 and weighted
using w5.
w 5  P(I1  high, I 2  high)
 P(I1  low, I 2  low)
w5 measures the quality of
learning.
17
Introduction
Network structure
Associative learning
 Simulation results
Conclusions and future work
18
Simulation results
• The TAE database (from University WisconsinMadison)
– 151 instances, 5 features and 3 classes
• The Iris plants database
– 150 instances 4 features and 3 classes
• The glass identification Database
– 214 instances, 9 features and 6 classes
• Image Recovery
– Two letters: B and J
19
Simulation results - TAE database
Features coded into binary
format with sliding bars and
classified using orthogonal
codes:
Not hierarchical; Connections
distribution Gaussian; vertically
(STD = 30) and horizontally (STD
= 5); correct rate = 62.33 %
N
V-Min
L
3M
Class 1
3M
Class 2
3M
Class 3
20
Simulation results - Iris database
Not hierarchical; Connections distribution Gaussian;
vertically (STD = 30) and horizontally (STD = 5);
correct rate = 73.74 %
21
Simulation results - Iris database
Hierarchical; vertical connections 80% Gaussian (STD = 2)
and 20% uniform; correct rate improved to 75.33 %
22
Simulation results - Iris database
The hierarchical structure appears
advantageous.
Using equal number of bits for features and
class IDs gives better rate.
Performance further improved to 86% with
mixed feature/classification bits.
23
Simulation results – glass ID database
• The depth of learning is
related to the complexity of
the target problem.
• With more classes, more
actively associating
neurons and more layers
are needed.
Average number of actively
associating neurons per
layer, with 3 / 6 classes
24
Simulation results - Image Recovery
A 2-D image recovery task.
200 patterns are generated by adding
random noise to 2 black-white images of
letters B and J. The network was trained with
190 patterns and tested using 10 patterns.
Mean correct classification rate: 100%
Training patterns
An Average image of training patterns
25
Simulation results - Image Recovery
Testing result and recovered images
Testing result and recovered image using input redundancy
26
Introduction
Network structure
Associative learning
Simulation results
 Conclusions and future work
27
Conclusion and Future Work
SOLAR has a flexible and sparse interconnect structure
designed to emulate the organization of a human cortex
It handles a wide variety of machine learning tasks including
image recognition, classification and data recovery, and is
suitable for online learning
The associative learning SOLAR is adaptive network with
feedback and inhibitory links
It discovers the correlation between inputs and establishes
associations inside the neurons
It can perform auto associative and hetero associative
learning
It can be modified to perform value driven interaction with
environment
28