Transcript No Slide Title

Cognitive and Semantic Forms (CF & SF)
Grammaticization: From bag of tricks to systematic syntax
Karine Megerdoomian: Unlocking the CF of verbs
Unpacking syntactic categories: e.g., mass nouns versus
count nouns.
From scene components to constituents
Heine: in front of/behind
PF
Language-Specific
SF
CF
“Almost” Language-Independent
I will use the term SF for Semantic Form (not San Francisco!)
The idea is that this occupies the same place as LF in
the approach of many linguists, but emphasizes that Logic is more likely
to be a useful descriptive tool rather than a strict match for neural representations.
Arbib and Itti: CS 664 (University of Southern California, Spring 2002) Integrating Vision, Action and Language
1
Performance Revisited
Observations:
Each form is distributed across
multiple brain regions.
Binding of subrepresentations is
required both within and across
Forms.
The Problem of Serial Order:
Linking hierarchical constituents to
ordered expressive gestures.
Hypothesis:
Linkages of the Basal Ganglia to
multiple levels play a crucial role.
Cognitive Structures
(Schema Assemblages)
CF
Semantic Structures
(Hierarchical Constituents
SF
expressing objects,
actions and relationships)
“Phonological” Structures
PF
(Ordered Expressive
Gestures)
Arbib and Itti: CS 664 (University of Southern California, Spring 2002) Integrating Vision, Action and Language
2
Extending the Mirror System
A subtle issue: We start with a clear distinction between the representation of the grasp
(the action/proto-verb) and the raisin (the thing/proto-noun) in the brain, but in the
spoken language both noun and verb are uttered by actions. Given this, how are we to
maintain the distinction between verb and noun?
Review the imaging data in this light.
Arbib and Itti: CS 664 (University of Southern California, Spring 2002) Integrating Vision, Action and Language
3
A Key to Distinguishing CF from Linguistic Forms
Distinguishing the sign from the affordance or the schema
Recall: Two Roles for Imitation in the Evolution of Manual-Based Communication
1. Extending imitation from imitation of hand movements by hand movements to
pantomime which uses the degrees of freedom of the hand (and arm and body) to
imitate degrees of freedom of objects and actions other than hand movements.
 Distinguishing the neural representation of the action or object per se (CF) from
the gesture which represents it (PF)
2. Extending these pantomime movements to to provide ad hoc gestures that may
convey to the observer information which is hared to pantomime in an “obvious”
manner. This requires extending the mirror system from the grasping repertoire to
mediate imitation of gestures to support the transition from ad hoc gestures to
conventional signs which can reduce ambiguity and extend semantic range.
 The beginning of morphology - modifying a gesture to provide shadings of
meaning.
 Such modifications may be ad hoc, yet become more systematic as historical
evolution regularizes certain of these constructions.
Arbib and Itti: CS 664 (University of Southern California, Spring 2002) Integrating Vision, Action and Language
4
Activity of F5 canonical neurons is part of the code for
Command: Grasp-A(Object)
Object features
Object
affordance
extraction
cIPS
7b: PF/PG Object affordance
-hand state
association
Hand
shape
recognition
Hand
motion
detection
STS
AIP
Integrate
temporal
association
Mirror Action
Feedback recognition
Hand-Object
spatial relation
analysis
7a
(Mirror
Neurons)
F5mirror
The Mirror Neuron
System (MNS) Model
F5canonical
Motor
program
(Grasp)
Motor
program
(Reach)
F4
Motor
execution
M1
Object
location
The full neural representation of the “Cognitive Form” (CF): Grasp-A(Object)
requires not only the regions AIP and F5canonical shown in the MNS diagram, but also
inferotemporal cortex (IT) which holds the identity of the object.
How are these representations bound together?
NOTE: This is only the Cognitive Form. There are no “Linguistic Forms” in the monkey.
How are these in humans linked to the CF (assumed homologous to monkey’s)?
Arbib and Itti: CS 664 (University of Southern California, Spring 2002) Integrating Vision, Action and Language
5
Activity of F5 mirror neurons is part of the code for
Declarative: Grasp-A(Agent, Object)
Object features
Object
affordance
extraction
cIPS
7b: PF/PG Object affordance
-hand state
association
Hand
shape
recognition
Hand
motion
detection
STS
Mirror
Feedback
Hand-Object
spatial relation
analysis
7a
AIP
Integrate
temporal
association
Action
recognition
F5mirror
The Mirror Neuron
System (MNS) Model
F5canonical
Motor
program
(Grasp)
Motor
program
(Reach)
F4
Motor
execution
M1
Object
location
The full neural representation of the “Cognitive Form” (CF): Grasp-A(Agent, Object)
requires not only the regions AIP, STS, 7a, 7b and F5miirror shown in the MNS diagram, but
also inferotemporal cortex (IT) which holds the identity of the object and regions of STS (?)
not included in MNS which hold the identity of the agent.
How are these representations bound together?
NOTE: This is only the Cognitive Form. There are no “Linguistic Forms” in the monkey.
How are these in humans linked to the CF (assumed homologous to monkey’s)?
Arbib and Itti: CS 664 (University of Southern California, Spring 2002) Integrating Vision, Action and Language
6
Beyond the Mirror to Neurolinguistics
Object features
cIPS
Object
affordance
extraction
7b: PF/PG Object affordance
-hand state
association
Hand
shape
recognition
Hand
motion
detection
STS
AIP
Integrate
temporal
association
Mirror Action
Feedback recognition
Hand-Object
spatial relation
analysis
7a
(Mirror
Neurons)
F5mirror
The Mirror Neuron
System (MNS) Model
F5canonical
Motor
program
(Grasp)
Motor
program
(Reach)
F4
Motor
execution
M1
Object
location
If the monkey needs so many brain regions for the mirror system for
grasping, how many more brain regions will we need for
an account of the language-ready brain that goes beyond the mirror
that goes far beyond the F5  Broca’s area homology to develop a full
neurolinguistic model linking CF, SF and PF ??
Arbib and Itti: CS 664 (University of Southern California, Spring 2002) Integrating Vision, Action and Language
7
Towards a Computational Neurolinguistics
Cooperative computation in the brain: to make sense of data relating
different brain regions to different aspects of language.

Do these data reflect the brain's genetic prespecification and/or the
results of the self-organization of the infant brain when the infant
develops within a particular language community?

Arbib and Itti: CS 664 (University of Southern California, Spring 2002) Integrating Vision, Action and Language
8
Cooperative Computation
The HEARSAY Paradigm for Speech Understanding
Woodja … ?
Arbib and Itti: CS 664 (University of Southern California, Spring 2002) Integrating Vision, Action and Language
9
HEARSAY II (1976)
A Serial Implementation of
a Distributed Architecture:
Consider how it might
relate to the interaction of
multiple brain regions
Arbib and Itti: CS 664 (University of Southern California, Spring 2002) Integrating Vision, Action and Language
10
A Simplistic View of Perceptual Schemas:
Constraint Satisfaction
Arbib and Itti: CS 664 (University of Southern California, Spring 2002) Integrating Vision, Action and Language
11
Cooperative Computation
The VISIONS Paradigm for Visual Scene Analysis
Arbib and Itti: CS 664 (University of Southern California, Spring 2002) Integrating Vision, Action and Language
12
A 20-year-old Overview of
Neurolinguistics:Luria (Arbib & Caplan)
A
B
A summary of diagrams
developed by Arbib and
Caplan (1979) based on
Luria's (1973) analyses of
Sel ect ive Naming
____
Vis ual P ercept ion
____
Terti ary L. P arietoOcci pit al Zone
L. Temporo-Occipit al
Zones
D
C
Swit chi ng Cont rol
____
Arti culat ory Syst em
____
Inferior Zone of
L. P remot or Cortex
Inferior Zone of
L. P os tcentral Cort ex
Vis ual Input
Audit ory Input
 Naming of objects;
 Verbal expression of
motives;
 Speech understanding;
 Speech repetition
E
F"
Updat ing t he P l an
of t he Expres si on
____
Frontal Lobes
F
...
H
Secondary Zone of
L. Temporal Cortex
G
P lan Formati on
____
Frontal Lobes
Lexi cal Anal ysi s
____
P honemic Analys is
____
P ost erior Zone of
L. Temporo-Occipit al
Regi on
I
Speech Memory
____
Mi ddl e Zones of
L. Temporal Region
Deep Zones of
L. Temporal Lobe
Formati on of the
Linear Scheme
____
Inferior Zone of
L. Front o-Temporal
Cort ex
F'
Acti ve A nalys is of
Mos t Si gni fi cant
Element s
____
Frontal Lobes
Arbib and Itti: CS 664 (University of Southern California, Spring 2002) Integrating Vision, Action and Language
J
Logical Scheme
____
L. P arieto-TemporoOcci pit al Zones
13
Naming of objects
A
B
C
Switching Control
____
Selective Naming
____
Visual Perception
____
Tertiary L. ParietoOccipital Zone
L. Temporo-Occipital
Zones
D
Articulatory System
____
Visual Input
Inferior Zone of
L. Premotor Cortex
Inferior Zone of
L. Postcentral Cortex
Arbib and Caplan (1979)
based on Luria's (1973) analysis
Arbib and Itti: CS 664 (University of Southern California, Spring 2002) Integrating Vision, Action and Language
14
Speech repetition
E
F"
Updating the Plan
of the Expression
____
Frontal Lobes
Phonemic Analysis
...
____
Secondary Zone of
L. Temporal Cortex
Arbib and Caplan (1979)
based on Luria's (1973) analysis
Arbib and Itti: CS 664 (University of Southern California, Spring 2002) Integrating Vision, Action and Language
15
Speech understanding
H
Lexical Analysis
____
Posterior Zone of
L. Temporo-Occipital
Region
I
Speech Memory
____
Middle Zones of
L. Temporal Region
Deep Zones of
L. Temporal Lobe
F'
Active Analysis of
Most Significant
Elements
____
Frontal Lobes
J
Logical Scheme
____
Arbib and Caplan (1979)
based on Luria's (1973) analysis
L. Parieto-TemporoOccipital Zones
Arbib and Itti: CS 664 (University of Southern California, Spring 2002) Integrating Vision, Action and Language
16
Verbal expression of motives
F
G
Plan Formation
____
Frontal Lobes
Formation of the
Linear Scheme
____
Inferior Zone of
L. Fronto-Temporal
Cortex
Arbib and Caplan (1979)
based on Luria's (1973) analysis
Arbib and Itti: CS 664 (University of Southern California, Spring 2002) Integrating Vision, Action and Language
17
Arbib and Caplan (1979)
B
based on Luria's (1973) analysis
C
D
Switching
Control
Selective
Naming
A
Visual
Perception
Articulatory
System
Visual Input
Auditory Input
E
F"
Updating
the Plan of
the Expr’n
...
Phonemic
Analysis
G
F
F'
Plan
Formation
Analysis of
Significant
Elements
Formation
of the
Linear
Scheme
H
Lexical
Analysis
I
J
Speech
Memory
Logical
Scheme