Transcript 6 November, 13 November

Bioinspired Computing
Lecture 7
Alternative
Neural Networks
M. De Kamps, Netta Cohen
Attractor networks:
Two examples
• Jets and Sharks network
– Weights set by hand
– Demonstrates recall
•
•
•
•
Generalisation
Prototypes
Graceful degradation
Robustness
• Kohonen networks
– Unsupervised learning: Self-Organizing Maps
2
3
+1
-1
4
Dynamics
• o: output of a node
act > 0: o = act
act <=0: o = 0
• act: activity of a node
– i >0:
– i <=0:
Δau = (max – au)*i – decay*(au-rest)
Δau = (au - min)*i – decay*(au-rest)
• i: input of a node
– iu = 0.1Σwuioi + 0.4 extu
5
Jets and Sharks
• Units
• Weights (excitatory: +1; inhibitory -1)
• Activation -0.2, 1.0
– Resting activation: -0.1
• Dynamics
6
7
+1
-1
8
Activate “ART”
9
Jets and Sharks
Name
Gang
Age
Education
Mar.
Occupation
Art
Jets
40s
J.H
Sing.
Pusher
Al
Jets
30s
J.H.
Mar.
Burglar
Clyde
Jets
40s
J.H
Sing.
Bookie
Mike
Jets
30s
J.H.
Sing.
Bookie
Phil
Sharks
30s
Col.
Mar
Pusher
Don
Sharks
30s
Col
Mar.
Burglar
Dave
Sharks
30s
H.S.
Div.
Pusher
10
Properties
• Retrieving a name from other properties
– Content Addressable Memory
• Categorisation and prototype formation
– Activating sharks will activate person units
of shark members
– Phil is quintessential shark:
• 30s
• Pusher (wins out in the end!)
11
Activate “Shark”
12
Properties
• Can activate 20s and pusher and find
persons who match best
• Robust
– Graceful degradation
– Noise
• Weight set by hand
13
Last time
Today
Attractor neural nets:
Other Neural Nets
• Biologically inspired
associative memories
• moves away from biorealistic model
• Unsupervised learning
• Working examples and
applications
• Pros, Cons & open
questions
•
•
•
•
SOM (Competitive) Nets
Neuroscience applications
GasNets.
Robotic control
14
Spatial Codes
Natural neural nets often code similar things close together.
The auditory and visual cortex provide examples.
Neural Material
Low
Freq.
Frequency
Sensitivity
Neural Material
High
Freq.
0°
Orientation
Sensitivity
Another example: touch receptors in the human
body. "Almost every region of the body is
represented by a corresponding region in both
the primary motor cortex and the somatic
sensory cortex" (Geschwind 1979:106). "The
finger tips of humans have the highest density of
receptors: about 2500 per square cm!" (Kandel
and Jessell 1991:374). This representation is
often dubbed the homunculus (or little man in the
brain)
Picture from http://www.dubinweb.com/brain/3.html
359°
15
Kohonen Nets
Lattice
In a Kohonen net, a number
of input neurons feed a
single lattice of neurons.
Input Nodes
Fully
Connected
Output
Pattern
The output pattern is
produced across the lattice
surface.
Large volumes of data are compressed using spatial/
topological relationships within the training set. Thus
the lattice becomes an efficient distributed
representation of the input.
16
Kohonen Nets
also known as self-organising maps (SOMs)
Important features:
• Self-organisation of a distributed representation of inputs.
• This is a form of unsupervised learning:
• The underlying learning principle: competition among
nodes known as “winner takes all”. Only winners get to
“learn” & losers decay. The competition is enforced by the
network architecture: each node has a self-excitatory
connection and inhibits all its neighbours.
• Spatial patterns are formed by imposing the learning rule
throughout the local neighbourhood of the winner.
17
Training Self-Organising Maps
A simple training algorithm might look like this:
1. Randomly initialise the network input weights
2. Normalise all inputs so they are size-independent
3. Define a local neighbourhood and a learning rate
4. For each item in the training set
• Find the lattice node most excited by the input
• Alter the input weights for this node and those
nearby such that they more closely resemble the
input vector, i.e., at each node, the input weight
update rule is: w = r (x-w)
5. Reduce the learning rate & the neighbourhood size
6. Goto 2 (another pass through the training set)
18
Training Self-Organising Maps (cont)
Gradually the net self-organises into a map of the
inputs, clustering the input data by recruiting areas of
the net for related inputs or features in the inputs.
The size of the neighbourhood roughly corresponds to
the resolution of the mapped features.
19
How Does It Work?
Imagine a 2D training set with clusters of data points
The nodes in the lattice are
initially randomly sensitive.
Horizontal
Gradually, they will “migrate”
towards the input data.
Blue
Red
Vertical
Nodes that are neighbours in the
lattice will tend to become
sensitive to similar inputs.
Effective resource allocation: dense
parts of the input space recruit
more nodes than sparse areas.
Another example: The travelling salesman problem
20
Applet from http://www.patol.com/java/TSP/index.html
The U-Matrix
• High dimensional clusters can only be visualized in
2D
• The U-height facilitates the recognition of clusters
– Define a distance in weight space
– For a given node n, take weight wn.
• For each neighbour of n, take weight wm
• Calculate d(wn,wm)
• Add all of these distances for node n
– This is the U-height of node n
• Colour the node map according to its U-height
• Clusters are areas of low U-height, separated by
boundaries of high U-height
21
The U-Matrix
22
How does the brain perform
classification?
One area of the cortex (the inferior temporal cortex or IT)
has been linked with two important functions:
• object recognition
• object classification
These tasks seem to be shape/colour specific but
independent of object size, position, relative motion or
speed, brightness or texture.
Indeed, category-specific impairments have been linked
to IT injuries.
23
How does the brain perform
classification (cont)?
Questions:
How do IT neurons encode objects/categories? e.g.,
• local versus distributed representations/coding
• temporal versus rate coding at the neuronal level
Can we recruit ANNs to answer such questions?
Can ANNs perform classification as well given similar data?
Recently, Elizabeth Thomas and colleagues performed
experiments on the activity of IT neurons during an
exercise of image classification in monkeys and used a
Kohonen net to analyse the data.
24
The experiment
Monkeys were trained to distinguish between a training set of
pictures of trees and various other objects. The monkeys were
considered trained when they reached a 95% success rate.
Trained monkeys were now shown new images of trees and other
objects. As they classified the objects, the activity in IT neurons in
their brains was recorded. All in all 226 neurons were recorded on
various occasions and over many different images.
The data collected was the mean firing rate of each neuron in
response to each image. 25% of neurons responded only to one
category, but 75% were not category specific. All neurons were
image-specific.
Problem: Not all neurons were recorded for all images &
No images were tested across all neurons.
In fact, when a Table of neuronal responses for each image was
created, it was more than 80% empty.
25
E. Thomas et al, J. Cog. Neurosci. (2001)
Experimental Results
Question: Given the partial data, is there sufficient
information to classify images as trees or non-trees?
Answer: A 2-node Kohonen net trained on the Table of
neuronal responses was able to classify new images with an
84% success rate.
Question: Are categories encoded by category-specific
neurons?
Answer: Delete data of category-specific neuron responses
from Table. The success rate of the Kohonen net was
degraded but only minimally. A control set with random data
deletions yielded similar results. Conclusion: Categoryspecific neurons are not important for categorisation!
26
E. Thomas et al, J. Cog. Neurosci. (2001)
Experimental Results (cont.)
Question: Which neurons are important, if any?
Answer: An examination of the weights that contribute most
to the output in the Kohonen net revealed that a small
subset of neurons (<50) that are not category-specific yet
respond with different intensities to different categories are
crucial for correct classification.
Conclusions: The IT employs a distributed representation to
encode categories of different images. The redundancy in
this encoding allows for graceful degradation so that even
with 80% of data missing and many neurons deleted,
sufficient information is present for classification purposes.
The fact that only rate information was used suggests that
temporal information is less important here.
27
E. Thomas et al, J. Cog. Neurosci. (2001)
From Biology to ANNs & Back
Neuroscience and studies of animal behaviour have led
to new ideas for artificial learning, communication,
cooperation & competition. Simplistic cartoon models of
these mechanisms can lead to new paradigms and
impressive technologies.
• Dynamic Neural Nets are helping us understand real-time
adaptation and problem-solving under changing conditions.
• Hopfield nets shed new insight on mechanisms of
association and the benefits of unsupervised learning.
• Thomas’ work helps unravel coding structures in the cortex.
28
Next time…
• Hopfield networks
Reading
• Elizabeth Thomas et al (2001) “Encoding of categories by noncategoryspecific neurons in the inferior temporal cortex”, J. Cog. Neurosci. 13: 190200.
• Phil Husbands, Tom Smith, Nick Jakobi & Michael O’Shea (1998). “Better
living through chemistry: Evolving GasNets for robot control”, Connection
Science, 10:185-210.
• Ezequiel Di Paolo (2003). Organismically-inspired robotics: Homeostatic
adaptation and natural teleology beyond the closed sensorimotor loop, in: K.
Murase & T. Asakura (Eds) Dynamical Systems Approach to Embodiment and
Sociality, Advanced Knowledge International., Adelaide, pp 19 - 42.
• Ezequiel Di Paolo (2000) “Homeostatic adaptation to inversion of the visual
29
field and other sensorimotor disruptions”, SAB2000, MIT Press.