Transcript USC Brain Project Specific Aims

Michael Arbib: CS564 - Brain Theory and Artificial Intelligence
The Aims of the Course:
We will use the challenge of understanding the mechanisms of visuomotor
coordination and action recognition in the monkey brain to provide a
structured set of goals for our mastery of
Brain Theory: modeling interactions of components of the brain,
especially more or less realistic biological neural networks localized in
specific brain regions
and
Connectionism in both Artificial intelligence (AI) and Cognitive
psychology: modeling artificial neural networks -- networks of trainable
“quasi-neurons” -- to provide “parallel distributed models” of intelligence in
humans, animals and machines
Tools: Gaining an understanding of the use of NSLJ (Neural Simulation
Language in Java) for simulating and analyzing neural networks
Arbib: CS564 - Brain Theory and Artificial Intelligence, USC, Fall 2001. Lecture 1. Introduction and Overview
1
CS 564: Brain Theory and Artificial Intelligence
URL: http://www-scf.usc.edu/~csci564/ for syllabus, instructor and TA
information, handouts, homework and grades
DEN URL: http://den.usc.edu/academic/fa2001/courses/csci564/csci564.asp
Instructor:
Michael Arbib; [email protected] (Office hours: 11-12 Tuesdays, HNB 03)
TAs: Erhan Oztop, [email protected], Salvador Marmol, smarmol@rana
This course provides a basic understanding of brain function, of artificial neural
networks which provide tools for a new paradigm for adaptive parallel
computation, and of the Neural Simulation Language NSLJ which allows us to
simulate biological and artificial neural networks
No background in neuroscience is required, nor is specific programming
expertise, but knowledge of Java will enable students to extend the NSLJ
functionality in interesting ways
Arbib: CS564 - Brain Theory and Artificial Intelligence, USC, Fall 2001. Lecture 1. Introduction and Overview
2
Texts
Texts:
MA Arbib, 1989, The Metaphorical Brain 2:
Neural Networks and Beyond, Wiley-Interscience
TMB2 is being sent over to "The paper clip" for duplication. Students
can purchase it there for $16 plus tax.
A Weitzenfeld, MA Arbib and A Alexander, 2002, NSL Neural Simulation
Language, MIT Press (in press)[
An old version is at http://www-hbp.usc.edu/_Documentation/NSL/ Book/TOC.htm. A
new version will be posted in a week or two.]
Other Required Reading:
will be posted on the Website – starting with the Mirror Neuron Proposal
Supplementary reading:
MA Arbib, Ed, 1995, The Handbook of Brain Theory and Neural Networks,
MIT Press (paperback)
Michael A Arbib, and Jeffrey Grethe, Editors, 2001, Computing the Brain: A
Guide to Neuroinformatics, San Diego: Academic Press (in press)
Arbib: CS564 - Brain Theory and Artificial Intelligence, USC, Fall 2001. Lecture 1. Introduction and Overview
3
Grading
One mid-term and a final will cover the entire contents
of the readings as well as the lectures
Students will be organized into 5 groups, each working together
on a semester-long project
The final exam will cover all of the course, but emphasizing
material not covered in the mid-term
Distribution of Grades:
NSL assignments and other homework: 25%;
 Mid-term: 20%;
 Project 30%;
 Final Exam: 25%

Arbib: CS564 - Brain Theory and Artificial Intelligence, USC, Fall 2001. Lecture 1. Introduction and Overview
4
5 Specific Aims for the Mirror Neuron Proposal
5 Specific Group Projects for CS 564
Development of the Mirror System:
1. Development of Grasp Specificity in F5 Motor and Canonical
Neurons
2. Visual Feedback for Grasping: A Possible Precursor of the Mirror
Property
Recognition of Novel and Compound Actions and their Context:
3. The Pliers Experiment: Extending the Visual Vocabulary
4. Recognition of Compounds of Known Movements
5. From Action Recognition to Understanding: Context and Expectation
Arbib: CS564 - Brain Theory and Artificial Intelligence, USC, Fall 2001. Lecture 1. Introduction and Overview
5
Send email to [email protected]
by Noon on Tuesday September 4, 2001
Line 1:
Your name then a colon (:) then your email address
Line 2:
Your Department (and your location if you are off-campus)
Line 3:
Any special skills you bring to the course
Lines 4+:
Your top 2 or 3 choices for a Project topic from the list:
1. Development of Grasp Specificity
2. Visual Feedback for Grasping
3. The Pliers Experiment
4. Recognition of Compound Movements
5. Context and Expectation
Arbib: CS564 - Brain Theory and Artificial Intelligence, USC, Fall 2001. Lecture 1. Introduction and Overview
6
Michael Arbib:CS564 - Brain Theory and Artificial Intelligence
Lecture 1 Introduction and Brain Overview
Reading Assignments:
TMB2: Chapter 1
Mirror Neuron Proposal*
* Read this “lightly” the first week of class We will master it and much more as
the class proceeds Your initial task is to review the 5 specific aims, and pick
the 2 you find most interesting One of these will probably become your
project goal for the semester
Arbib: CS564 - Brain Theory and Artificial Intelligence, USC, Fall 2001. Lecture 1. Introduction and Overview
7
Opposition Spaces and Virtual Fingers
The goal of a successful
preshape, reach and grasp
is to match the opposition
axis defined by the virtual
fingers of the hand with
the opposition axis defined
by an affordance of the
object
(Iberall and Arbib 1990)
Arbib: CS564 - Brain Theory and Artificial Intelligence, USC, Fall 2001. Lecture 1. Introduction and Overview
8
Visual Control of Grasping in Macaque Monkey
A key theme of
visuomotor coordination:
parietal affordances
(AIP)
drive
frontal motor
schemas
(F5)
F5 - grasp
commands in
premotor cortex
Giacomo Rizzolatti
AIP - grasp
affordances
in parietal cortex
Hideo Sakata
Arbib: CS564 - Brain Theory and Artificial Intelligence, USC, Fall 2001. Lecture 1. Introduction and Overview
9
Grasp Specificity in an F5 Neuron
Precision pinch (top)
Power grasp (bottom)
(Data from Rizzolatti et
al.)
Arbib: CS564 - Brain Theory and Artificial Intelligence, USC, Fall 2001. Lecture 1. Introduction and Overview
10
FARS (Fagg-Arbib-Rizzolatti-Sakata) Model Overview
AIP extracts affordances features of the object relevant
to physical interaction with it.
Prefrontal cortex provides
“context” so F5 may select
an appropriate affordance
AIP
AIP
Dorsal
Stream:
dorsal/ventral
Affordances
streams
Ways to grab
this “thing”
TaskConstra
Constraints
Task
ints ( F6)
(F6)
Working
Memory
W
orking Me
mory (46)
(46?)
Instruction
Stimuli
Instruction
Stim
uli (F2)
(F2)
PFC
F5
F5
“It’s a mug”
Ventral
Stream:
Recognition
IT
Arbib: CS564 - Brain Theory and Artificial Intelligence, USC, Fall 2001. Lecture 1. Introduction and Overview
11
Syllabus Overview 1
Introduction [Proposal] {Background: TMB Chapter 1}
Charting the Brain 1 [TMB 2.4]
The Brain as a Network of Neurons [TMB Section 2.3]
Visual Preprocessing [TMB 3.3]
Adaptive networks: Hebbian learning, Perceptrons; Landmark learning
[TMB 3.4] [NSLbook]
Hebbian Learning and Visual plasticity; Self-organizing feature maps;
[NSLJ] Kohonen maps
Higher level vision 1: object recognition {Background TMB 5.2}
Introduction to NSL: modules; SCS schematic capture system;
Maxselector model[NSLbook] {Homework}

Arbib: CS564 - Brain Theory and Artificial Intelligence, USC, Fall 2001. Lecture 1. Introduction and Overview
12
Syllabus Overview 2
Schemas for Reaching and Grasping; Affordances [TMB 2.2, 5.3]
{Background TMB 2.1, 52}
Charting the Brain 2
The FARS model of control of grasping 1: Population coding
The FARS model of control of grasping 2: Sequence learning and the
basal ganglia
The FARS model of control of grasping 3: Working Memory

Arbib: CS564 - Brain Theory and Artificial Intelligence, USC, Fall 2001. Lecture 1. Introduction and Overview
13
Mirror Neurons
Rizzolatti, Fadiga, Gallese, and Fogassi, 1995:
Premotor cortex and the recognition of motor
actions
Mirror neurons form the subset
of grasp-related premotor
neurons of F5 which discharge
when the monkey observes
meaningful hand movements
made by the experimenter or
another monkey.
F5 is endowed with an
observation/execution matching
system
[The non-mirror grasp neurons of F5 are
called F5 canonical neurons.]
Arbib: CS564 - Brain Theory and Artificial Intelligence, USC, Fall 2001. Lecture 1. Introduction and Overview
14
Computing the Mirror System Response
The FARS Model:
Recognize object affordances and determine appropriate grasp.
The Mirror Neuron System (MNS) Model:
We must add recognition of
 trajectory and
 hand preshape
to
 recognition of object affordances
and ensure that all three are congruent.
There are parietal systems other than AIP adapted to this task.
Arbib: CS564 - Brain Theory and Artificial Intelligence, USC, Fall 2001. Lecture 1. Introduction and Overview
15
The Mirror Neuron System (MNS) Model
Object features
cIPS
PF
Object
affordance
extraction
Hand
motion
detection
STS
AIP
Object affordance
-hand state
association
Hand
shape
recognition
Motor
program
(Grasp)
Integrate
temporal
association
Action
Mirror
Feedback recognition
Hand-Object
spatial relation
analysis
7a
F5canonical
(Mirror
Neurons)
F5mirror
Object
location
Motor
execution
Motor
program
(Reach)
M1
F4
work with
Erhan Oztop
Arbib: CS564 - Brain Theory and Artificial Intelligence, USC, Fall 2001. Lecture 1. Introduction and Overview
16
STS hand shape recognition
Color Coded Hand
Feature Extraction
Step 1 of hand shape
recognition: system
processes the color-coded
hand image and generates a
set of features to be used by
the second step
Model Matching
Step 2: The feature vector
generated by the first step is used
to fit a 3D-kinematics model of the
hand by the model matching
module. The resulting hand
configuration is sent to the
classification module.
Precision grasp
Hand Configuration
Classification
Arbib: CS564 - Brain Theory and Artificial Intelligence, USC, Fall 2001. Lecture 1. Introduction and Overview
17
Virtual Hand/Arm and Reach/Grasp Simulator
A precision pinch
A power grasp
and
a side grasp
Arbib: CS564 - Brain Theory and Artificial Intelligence, USC, Fall 2001. Lecture 1. Introduction and Overview
18
Core Mirror Circuit
Object affordance
Association
(7b) Neurons
Mirror Neurons
(F5mirror)
Mirror Neuron
Output
Hand state
Motor Program
(F5 canonical)
Object
Affordances
Object affordance hand state association
Integrate
temporal
association
Mirror Feedback
Motor
program
F5canonical
Hand shape
recognition
& Hand
motion
detection
Mirror
Feedback
Hand-Object
spatial relation
analysis
Action
recognition
(Mirror
Neurons)
Motor
program
Motor
execution
F5mirror
Arbib: CS564 - Brain Theory and Artificial Intelligence, USC, Fall 2001. Lecture 1. Introduction and Overview
19
Power and precision grasp resolution
(a)
(b)
Precision Pinch
Mirror Neuron
Power Grasp
Mirror Neuron
Arbib: CS564 - Brain Theory and Artificial Intelligence, USC, Fall 2001. Lecture 1. Introduction and Overview
20
Syllabus Overview 3
Reinforcement learning and motor control; [NSLJ] Conditional motor
learning
Adaptive networks: Gradient descent and backpropagation [TMB 82]
[NSLJ] Backprop: a How to run the model; b How to write the model
[NSLbook]
NeuroBench and the NeuroHomology Database

Arbib: CS564 - Brain Theory and Artificial Intelligence, USC, Fall 2001. Lecture 1. Introduction and Overview
21
Syllabus Overview 4
The MNS1 Model 1: Hand Recognition
Systems concepts; Feedback and the spinal cord [TMB 31, 32]
The MNS 1 Model 2: Simulating the kinematics and biomechanics of
reach and grasp
The MNS1 Model 3: Modeling the Core Mirror Neuron Circuit

Arbib: CS564 - Brain Theory and Artificial Intelligence, USC, Fall 2001. Lecture 1. Introduction and Overview
22
5 Specific Aims for the Mirror Neuron Proposal
5 Specific Group Projects for CS 564
Development of the Mirror System:
1. Development of Grasp Specificity in F5 Motor and Canonical
Neurons
2. Visual Feedback for Grasping: A Possible Precursor of the Mirror
Property
Recognition of Novel and Compound Actions and their Context:
3. The Pliers Experiment: Extending the Visual Vocabulary
4. Recognition of Compounds of Known Movements
5. From Action Recognition to Understanding: Context and Expectation
Arbib: CS564 - Brain Theory and Artificial Intelligence, USC, Fall 2001. Lecture 1. Introduction and Overview
23
A Human Mirror System
Rizzolatti, Fadiga, Matelli, Bettinardi, Perani, and Fazio: Broca's region is activated by observation of
hand gestures: a PET study.
PET study of human brain with 3 experimental conditions:
Object observation (control condition)
 Grasping observation
A key language area!!!
 Object prehension.
The most striking result was highly significant activation in the rostral
part of Broca's area.

Arbib: CS564 - Brain Theory and Artificial Intelligence, USC, Fall 2001. Lecture 1. Introduction and Overview
24
A New Approach to the Evolution of Human Language
Rizzolatti, G, and Arbib, M.A., 1998, Language Within Our Grasp, Trends
in Neuroscience, 21(5):188-194:
The Mirror System Hypothesis: Human Broca’s area contains a mirror system
for grasping which is homologous to the F5 mirror system of monkey, and this
provides the evolutionary basis for language parity - i.e., an utterance means
roughly the same for both speaker and hearer.
This adds a neural “missing link” to the tradition that roots speech in a prior
system for communication based on manual gesture.

Beyond the Mirror: Seeing F5 as part of a larger mirror system, then extending
our understanding via imitation to language-readiness.
 This topic will be the jumping off point for the Spring 2002 version of CS
664, which will be taught by Professors Arbib and Itti.
Arbib: CS564 - Brain Theory and Artificial Intelligence, USC, Fall 2001. Lecture 1. Introduction and Overview
25
Syllabus Overview 5
Control of eye movements [TMB 6.2]
Basal Ganglia and Control of eye movements - Dominey [NSLbook]
Dopamine and Sequence Learning
Abstract models of Sequence Learning
Extending the FARS model to mirror neurons and language
Project Reports 1, 2,3
Project Reports 4,5; Concluding Discussion

Arbib: CS564 - Brain Theory and Artificial Intelligence, USC, Fall 2001. Lecture 1. Introduction and Overview
26
A Conceptual Restructuring of the Syllabus 1
Overview + Basic Concepts of Neurons and Schemas
1. Modeling the Mirror System: Setting Goals for the Course [Proposal]
{Background: TMB Chapter 1}
2. Charting the Brain 1 [TMB 2.4]
3. The Brain as a Network of Neurons [TMB Section 2.3]
9. Schemas for Reaching and Grasping; Affordances [TMB 2.2, 5.3]
{Background TMB 2.1, 5.2}
10. Charting the Brain 2
Arbib: CS564 - Brain Theory and Artificial Intelligence, USC, Fall 2001. Lecture 1. Introduction and Overview
27
A Conceptual Restructuring of the Syllabus 2
Vision
Preprocessing
4. Visual Preprocessing [TMB 3.3]
6. Hebbian Learning and Visual plasticity; Self-organizing feature maps;
[NSLJ] Kohonen maps
Low-Level
Vision
Supplementary reading not covered in lectures: Perceptual and motor
schemas: Discussion of visual segmentation based on, e.g., edge and
texture cues; Stereoscopic vision [TMB 7.1]; Motion perception and
optic flow [TMB 7.2].
High-Level Vision
7. Higher level vision: object recognition {Background TMB 5.2}
18. The MNS1 Model 1: Hand Recognition
Arbib: CS564 - Brain Theory and Artificial Intelligence, USC, Fall 2001. Lecture 1. Introduction and Overview
28
A Conceptual Restructuring of the Syllabus 3
Motor Control
19. Systems concepts; Feedback and the spinal cord [TMB 3.1, 3.2]
20. The MNS 1 Model 2: Simulating the kinematics and biomechanics
of reach and grasp
Visuo-Motor Integration
11. The FARS model of control of grasping 1: Population coding
12. The FARS model of control of grasping 2: Sequence learning and
the basal ganglia
13. The FARS model of control of grasping 3: Working Memory
21. The MNS1 Model 3: Modeling the Core Mirror Neuron Circuit
22. Control of eye movements [TMB 6.2]
23. Basal Ganglia and Control of eye movements - Dominey
[NSLbook]
Arbib: CS564 - Brain Theory and Artificial Intelligence, USC, Fall 2001. Lecture 1. Introduction and Overview
29
A Conceptual Restructuring of the Syllabus 4
Adaptive Networks
5. Adaptive networks: Hebbian learning, Perceptrons; Landmark learning
[TMB 3.4] [NSLbook]
14. Reinforcement learning and motor control; [NSLJ] Conditional motor
learning
15. Adaptive networks: Gradient descent & backpropagation [TMB 8.2]
24. Dopamine and Sequence Learning
25. Abstract models of Sequence Learning
Higher Cognitive Functions
26. Extending the FARS model to mirror neurons and language
Supplementary reading: Memory and Consciousness [TMB 8.3]
Simulation and Neuroinformatics
8. Introduction to NSL: modules; SCS schematic capture system; Maxselector
model [NSLbook]
16. [NSLJ] Backprop: a. How to run and write the model [NSLbook]
17. NeuroBench and the NeuroHomology Database
Arbib: CS564 - Brain Theory and Artificial Intelligence, USC, Fall 2001. Lecture 1. Introduction and Overview
30