#### Transcript Pattern Recognition using Hebbian Learning and Floating-Gates

Pattern Recognition using Hebbian Learning and Floating-Gates Certain pattern recognition problems have been shown to be easily solved by Artificial neural networks and many neural network chips have been made and sold. Most of these are not terribly biologically realistic. Output layer neurons weights Hidden layer neurons weights Input layer neurons A 2-dimensional example… x (-10:10) y(-10:10) w1 = 0.3, w2 = =0.7 x y Output Decision Space 50 50 100 100 Y Y Summation of weighted inputs 150 +1, -1 150 200 200 50 100 X 150 200 50 100 X 150 200 x (-10:10) y(-10:10) w1 = 0.5, w2 = 0.11 x y Outputs over the Input Space 50 50 100 100 Y Y Weighted Summation over the Input Space 150 +1, -1 150 200 200 50 100 X 150 200 50 100 X 150 200 Putting the two together… x y 50 -.53 y 0.53 -0.25 We respond to a smaller region of this 2-D input space. +1, -1 100 150 200 50 100 x 150 200 So in general, we can apply this type of operation on an N-dimensional input With the hidden units defining hyperplanes in this input space. The individual output units combine these hyperplanes to create specific subregions of this N-dim space. This is what pattern recognition is about. 100 x 77 pixels = 7700 dimensional input space As you might expect, these two images live very far apart from each other in this very high dimensional space. 10 20 30 40 50 60 70 unit1 80 90 Easy Task 100 10 20 30 40 50 60 70 unit2 10 20 30 But if we had a set of 100 faces that we wanted to recognize, this might be harder. 40 50 60 70 80 90 100 10 20 30 40 50 60 70 What happens if the faces are rotated, shifted, or scaled? How do I pick the weight matrices to solve these tasks?? One way is to present inputs and adjust the weights if the output is not what we want. wiknew wikold wik where input k, example wik 2 i k = 0 Learning rate if i Oi otherwise Output unit i, example Target output for unit i example /- 1 This is known as the perceptron learning rule A training set of examples with target output values is defined and presented one by one, adjusting the weight matrix after each evaluation. The learning rule Assigns large value weights to components of the inputs that allow discrimination between the classes of inputs. e.g., many faces and many helicopters Face vs. Helicopter Example 5000 Red = unit 1 response to face 4000 Blue = unit 1 response to helicopter 3000 Green = unit 2 response to face 2000 Black = unit 2 response to helicopter 1000 0 -1000 -2000 -3000 -4000 -5000 0 10 20 30 40 50 60 70 80 90 100 Associative Memory and Energy Functions Si inputs 10 10 20 20 30 30 40 40 50 50 60 60 70 70 80 80 90 90 100 100 10 wij matrix 20 30 40 50 60 70 10 20 30 40 50 60 70 The Hopfield Recurrent Network The concept of an energy function of a recurrent neural network was introduced by Hopfield (1982) to describe the state of a network. By studying the dynamics, it is possible to show that the activity in the network will always decrease in energy, evolving towards a "local minima". H - 1 wijSiSj 2 ij The network defines an 'energy landscape' in which the state of the network settles. By starting close to minima (stored patterns) compared to other points in the landscape The network will settle towards the minima and 'recall' the stored patterns. This view of neural processing has its merits, provides insight into this type of computational structure and has spawned new fields on its own, but does not describe the current neurobiological state of knowledge very well. In particular, neurons communicate with spikes and the backpropagation learning rule Is not a good match to what has been found. So what do we know about neurobiological learning? Hebbian learning If both cells are active, strengthen the synapse If only the post-synaptic cell is active, weaken the synapse In fact, learning at some synapses seems to be even more specific. Temporal ordering seems to play a role in determining the change in the synapse. strengthen weaken Dw Time between pre-syn and post-syn spikes Abbott and Blum, 1996 Chip Idea: 1. Design a spiking neural network that can learn using the spike-timing rule to solve a particular temporal pattern recognition problem 2. Design a floating-gate modification circuit that can implement the learning rule