Transcript Lecture7 Associative Memory

Neural Networks
Associative Model of
Memory



Learning is the process of forming associations
between related patterns.
Human memory connects items (ideas, sensations,
etc.) that are similar, that are contrary, that occur
in close proximity, or that occur in close succession
(Kohonen, 1987).
We cannot remember an event before it happens.
– Therefore an event happens, some change takes
place in our brains so that subsequently we can
remember the event.
– So memory is inherently bound up in the learning
process
Associative Model of
Memory



In a neurobiological context, memory refers to the
relatively enduring neural alterations induced by
the interactions of an organism with its
environment (Tayler, 1986).
– Without such a change, there can be no
memory.
Furthermore, for the memory to be useful, it must
be accessible to the nervous system so as to
influence future behavior.
When a particular activity pattern is learned, it is
stored in the brain, from which it can be recalled
later when required.
Short- and Long-Term
Memories



Memory may be divided into “short-term” and
“long-term” memory, depending on the
retention time (Arbib, 1989).
Short-term memory refers to a compilation of
knowledge representing the “current” state of
the environment. Any discrepancies between
knowledge stored in short-term memory and a
“new” state are used to update the short-term
memory.
Long-term memory, on the other hand, refers
to knowledge stored for a long time or
permanently.
Fundamental Property of
Associative Memory





A fundamental property of the associative memory
is that “it maps an output pattern of neural activity
onto an input pattern of neural activity”.
In particular, during the learning phase, a “key
pattern” is presented as stimulus, and the memory
transforms it into a “memorized” or “stored
pattern”.
The storage takes place through specific changes in
the synaptic weights of the memory.
During the retrieval or recall phase, the memory is
presented with a stimulus that is a noisy version or
incomplete description of a key pattern originally
associated with a stored pattern.
Despite imperfections in the stimulus, the
associative memory has the capability to recall the
stored pattern correctly.
Some Characteristics of the
Associative Memory
The memory is distributed.
1. Both the stimulus (key) pattern and the response (stored)
pattern of an associative memory consist of data vectors.
2. Information is stored in memory by setting up a spatial
pattern of neural activities across a large number of
neurons.
3. Information contained in a stimulus not only determines
its storage location in memory but also an address for its
retrieval.
4. Despite the fact that the neurons do not represent reliable
and low-noise computing cells, the memory exhibits a high
degree of resistance to noise and damage of a diffusive
kind.
5. There may be interactions between individual patterns
stored. (Otherwise, the memory would have to be
exceptionally large for it to accommodate the storage of a
large number of patterns in perfect isolation from each
other.) There is therefore, the distinct possibility of the
memory making errors during the recall process.
Auto- Versus Heteroassociative Memory


There are two types of association:
Auto-association: A key vector (pattern) is
associated with itself in memory.
– This is most useful for pattern completion where a
partial pattern (a pair of eyes) or a noisy pattern (a
blurred image) is associated with its complete and
accurate representation (the whole face).
– The input and output signal (data) spaces have the
same dimensionality.

Hetero-association: A vector is associated with
another vector which may have different
dimensionality.
– We may still hope that a noisy or partial input vector will
retrieve the complete output vector.
Linear Versus Non-linear
Associative Memory




An associative memory may also be classified as linear
or non-linear, depending upon the model adopted for its
neurons.
Let the data vectors a and b denote the stimulus (input)
and the response (output) of an associative memory,
respectively.
Linear Associative Memory: Input-output relationship
is:
b=Ma
where M is called the “memory matrix”.
Nonlinear Associative Memory: Here the input-output
relationship is of the form:
b = j( M; a ) a
where in general, j(. ; .) is a nonlinear function of the
memory matrix and the input vector.
Block Diagram of
Associative Memory
Stimulus
a
Memory Matrix
M
Response
b
A Simple Network for
Holding Associative Memory
Weights
Inputs
Outputs
Input
Neurons
Output
Neurons