Transcript The Hopfield Model - Jonathan Amazon Neural Network Neural

The Hopfield
Model
- Jonathan Amazon
Neural Network
Neural Network

Can be modeled as a spin glass.

Each neuron is a spin.



The synapses between neurons are spin couplings.



Can be in an excited state (s = +1)
or a quiet state (s = -1)
Can be excitory (ferromagnetic, J > 0)
or inhibitory (anti-ferromagnetic, J < 0)
Experimentally observed: Neurons spend most time in
quiet state due to activation threshold.

External field (H < 0) captures this behavior
Spin Glass


Ising model with
non-uniform
coupling
strength.
Couplings are
usually
quenched
variables drawn
from distribution.
Hopfield Network

Spin glass neural network


Completely connected
 Couplings are not chosen from a distribution
Pre-defined memory states are encoded into the
coupling strengths.




Hebbian rule fixes couplings.
Memory states become minimal energy
configurations (mostly).
Gives network associative memory properties.
Memory states are randomly generated by
uniform probability of up or down spin.
Dynamics

Method of Decent.


Calculate local field from all other spins
 Compare to activation threshold.
 H = 0 for my simulation, Implies inversion
symmetry of hamiltonian.
 Flip accordingly.
Total energy is monotonically decreasing and system
tends to a local energy well.
Associative Memory

Relaxation from arbitrary starting state to nearest energy
minimum.


Hebbian rule: Local minimum will be memory state
most closely resembling starting state.
 Or its inverse (two fold degeneracy).
Memory capacity



Extensively measured as p/N (memory density).
Critical Memory threshold above which your 'brain
explodes'
How much is too much?
Percent of misaligned spins
Thermodynamic Limit
D. Amit
H. Gutfreund
H Sompolinsky
Memory Density

Critical memory density at p/N ~ 0.138. All energy minima are
null correlated with desired memory states.
Memory Reliability

Testing reliability of memory storage.


Initialize in pure state.
 Relax lattice to ground state.
 All or nothing. Does relaxed state match initial
state perfectly?
Measures the percentage of times the lattice
successfully retained the memory state.
Memory Density
Percent chance of recovering pure state
Memory Degradation


Testing how memory degrades as memory density
increases.
CASE1:


Start in pure state.
 Relax network and record percent of spins that differ
from initial state.
CASE2:



Start in random state.
Relax network and determine closest pure state
(prone to bias when null correlated).
Record percent of spins that differ from closest
memory state.
Percent of misaligned spins
FINITE SIZE EFFECTS?
SAMPLING BIAS?
Memory Density
Memory Density
Percent of misaligned spins
Applications

Facial recognition (secutiry cameras, digital cameras...)

Hand writing recognition (scanners, LateX help...)

Numerical/graphical operations