Transcript The Neural Basis of Thought and Language

The Neural Basis of
Thought and Language
Final
Review Session
Administrivia
• Final in class next Tuesday, May 9th
• Be there on time!
• Format:
– closed books, closed notes
– short answers, no blue books
• And then you’re done with the course!
The Second Half
Motor
Control
Grammar
Metaphor
Cognition and Language
abstraction
Computation
Bayes
Nets
Bailey KARMA
Model
Structured Connectionism
ECG
SHRUTI
Bayesian
Model of
HSP
Computational Neurobiology
Biology
Midterm
Final
Overview
• Bailey Model
• Grammar Learning
– feature structures
– parsing
– Bayesian model merging
– construction grammar
– recruitment learning
– learning algorithm
• KARMA
• SHRUTI
– X-schema, frames
• FrameNet
– aspect
• Bayesian Model of Human
Sentence Processing
– event-structure metaphor
– inference
Full Circle
Neural System &
Development
Metaphor
Psycholinguistics
Experiments
Grammar
Embodied
Representation
Structured
Connectionism
Probabilistic
algorithms
Converging
Constraints
Motor Control & Visual
System
Verbs & Spatial
Relation
Spatial
Relation
Q&A
How can we capture the difference between
“Harry walked into the cafe.”
“Harry is walking into the cafe.”
“Harry walked into the wall.”
“Harry walked into the café.”
Utterance
Constructions
General
Knowledge
Analysis Process
Semantic
Specification
Belief State
Simulation
The INTO construction
construction INTO
subcase of Spatial-Relation
form
selff .orth ← “into”
meaning: Trajector-Landmark
evokes Container as cont
evokes Source-Path-Goal as spg
trajector ↔ spg.trajector
landmark ↔ cont
cont.interior ↔ spg.goal
cont.exterior ↔ spg.source
The Spatial-Phrase construction
construction SPATIAL-PHRASE
constructional
constituents
sr : Spatial-Relation
lm : Ref-Expr
form
srf before lmf
meaning
srm.landmark ↔ lmm
The Directed-Motion construction
construction DIRECTED-MOTION
constructional
constituents
a : Ref-Exp
m: Motion-Verb
p : Spatial-Phrase
form
af before mf
mf before pf
meaning
evokes Directed-Motion as dm
selfm.scene ↔ dm
dm.agent ↔ am
dm.motion ↔ mm
dm.path ↔ pm
schema Directed-Motion
roles
agent : Entity
motion : Motion
path : SPG
What exactly is simulation?
• Belief update plus X-schema execution
at goal
ready
time
of day
hungry
start
ongoing
meeting
iterate
cafe
WALK
finish
done
“Harry walked into the café.”
ready
walker=Harry
walk
done
goal=cafe
“Harry is walking to the café.”
Utterance
Constructions
General
Knowledge
Analysis Process
Semantic
Specification
Belief State
Simulation
“Harry is walking to the café.”
suspended
interrupt
ready
resume
start
finish
ongoing
abort
done
iterate
cancelled
walker=Harry
WALK
goal=cafe
“Harry has walked into the wall.”
Utterance
Constructions
General
Knowledge
Analysis Process
Semantic
Specification
Belief State
Simulation
Perhaps a different sense of INTO?
construction INTO
subcase of spatial-prep
form
selff .orth ← “into”
meaning
evokes Trajector-Landmark as tl
evokes Container as cont
evokes Source-Path-Goal as spg
tl.trajector ↔ spg.trajector
tl.landmark ↔ cont
cont.interior ↔ spg.goal
cont.exterior ↔ spg.source
construction INTO
subcase of spatial-prep
form
selff .orth ← “into”
meaning
evokes Trajector-Landmark as tl
evokes Impact as im
evokes Source-Path-Goal as spg
tl.trajector ↔ spg.trajector
tl.landmark ↔ spg.goal
im.obj1 ↔ tl.trajector
im.obj2 ↔ tl.landmark
“Harry has walked into the wall.”
suspended
interrupt
ready
resume
start
finish
ongoing
abort
done
iterate
cancelled
walker=Harry
WALK
goal=wall
Map down to timeline
start
ready
finish
ongoing
done
consequence
E
S
R
further questions?
What about…
“Harry walked into trouble”
or for stronger emphasis,
“Harry walked into trouble, eyes wide open.”
Metaphors
• metaphors are mappings from a source domain to a
target domain
• metaphor maps specify the correlation between
source domain entities / relation and target domain
entities / relation
• they also allow inference to transfer from source
domain to target domain (possibly, but less
frequently, vice versa)
<TARGET> is <SOURCE>
Event Structure Metaphor
• Target Domain: event structure
• Source Domain: physical space
•
States are Locations
•
Changes are Movements
•
Causes are Forces
•
Causation is Forced Movement
•
Actions are Self-propelled Movements
•
Purposes are Destinations
•
Means are Paths
•
Difficulties are Impediments to Motion
•
External Events are Large, Moving Objects
•
Long-term Purposeful Activities are Journeys
KARMA
• DBN to represent
target domain
knowledge
• Metaphor maps link
target and source
domain
• X-schema to
represent source
domain knowledge
Metaphor Maps
1. map entities and objects between embodied and
abstract domains
2. invariantly map the aspect of the embodied domain
event onto the target domain
by setting the evidence for the status variable based
on controller state (event structure metaphor)
3. project x-schema parameters onto the target
domain
further questions?
How do you learn…
the meanings of spatial relations,
the meanings of verbs,
the metaphors, and
the constructions?
How do you learn…
the meanings of spatial relations,
the meanings of verbs,
the metaphors, and
the constructions?
That’s the Regier model.
(first half of semester)
How do you learn…
the meanings of spatial relations,
the meanings of verbs,
the metaphors, and
the constructions?
VerbLearn
schema
elbow jnt
posture
accel
slide 0.9
extend 0.9
palm 0.7
0.9
[6]- 8]
[6
grasp 0.3
data #1
data #2
data #3
data #4
schema
elbow jnt
posture
accel
depress 0.9
fixed 0.9
index 0.9
[2]
schema
elbow jnt
posture
accel
slide
extend
palm
6
schema
elbow jnt
posture
accel
slide
extend
palm
8
schema
elbow jnt
posture
accel
depress
fixed
index
2
schema
elbow jnt
posture
accel
slide
extend
grasp
2
Computational Details
• complexity of model + ability to explain data
• maximum a posteriori (MAP) hypothesis
argmax P(m | D)
wants the best model
given data
m
 argmax P( D | m) P(m) by Bayes' rule
m
how likely is the data
given this model?
penalize complex models –
those with too many word senses
How do you learn…
the meanings of spatial relations,
the meanings of verbs,
the metaphors, and
the constructions?
conflation hypothesis
(primary metaphors)
How do you learn…
the meanings of spatial relations,
the meanings of verbs,
the metaphors, and
the constructions?
construction learning
Usage-based Language Learning
Reorganize
(Utterance, Situation)
(Comm. Intent, Situation)
Constructions
Analyze
Partial Analysis
Comprehension
Generate
Hypothesize
Utterance
Acquisition
Production
Main Learning Loop
while <utterance, situation> available and cost > stoppingCriterion
analysis = analyzeAndResolve(utterance, situation, currentGrammar);
newCxns = hypothesize(analysis);
if cost(currentGrammar + newCxns) < cost(currentGrammar)
addNewCxns(newCxns);
if (re-oganize == true) // frequency depends on learning parameter
reorganizeCxns();
Three ways to get new constructions
• Relational mapping
– throw the ball
THROW < BALL
• Merging
– throw the block
– throwing the ball
THROW < OBJECT
• Composing
– throw the ball
– ball off
– you throw the ball off
THROW < BALL < OFF
Minimum Description Length
• Choose grammar G to minimize cost(G|D):
– cost(G|D) = α • size(G) + β • complexity(D|G)
– Approximates Bayesian learning;
cost(G|D) ≈ posterior probability P(G|D)
• Size of grammar = size(G) ≈ 1/prior P(G)
– favor fewer/smaller constructions/roles; isomorphic mappings
• Complexity of data given grammar ≈ 1/likelihood P(D|G)
– favor simpler analyses
(fewer, more likely constructions)
– based on derivation length + score of derivation
further questions?
Connectionist Representation
How can entities and relations be represented at the
structured connectionist level?
or
How can we represent
Harry walked to the café
in a connectionist model?
SHRUTI
• entity, type, and predicate focal clusters
• An “entity” is a phase in the rhythmic activity.
• Bindings are synchronous firings of role and entity cells
• Rules are interconnection patterns mediated by coincidence
detector circuits that allow selective propagation of activity
• An episode of reflexive processing is a transient propagation of
rhythmic activity
“Harry walked to the café.”
entity
type
predicate
Harry
+
+e
+v
cafe
?
?e
?v
• asserting that
walk(Harry, café)
• Harry fires in phase
with agent role
• cafe fires in phase
with goal role
+
-
walk ?
agt goal
“Harry walked to the café.”
entity
type
predicate
Harry
+
+e
+v
cafe
?
?e
?v
• asserting that
walk(Harry, café)
• Harry fires in phase
with agent role
• cafe fires in phase
with goal role
+
-
walk ?
agt goal
Activation Trace for walk(Harry, café)
+: walk
walk-agt
walk-goal
+: Harry
+e: cafe
1
2
3
4
further questions?
Human Sentence Processing
Can we use any of the mechanisms we just discussed
to predict reaction time / behavior
when human subjects read sentences?
Good and Bad News
• Bad news:
– No, not as it is.
– ECG, the analysis process and simulation process are
represented at a higher computational level of
abstraction than human sentence processing (lacks
timing information, requirement on cognitive
capacity, etc)
• Good news:
– we can construct bayesian model of human sentence
processing behavior borrowing the same insights
Bayesian Model of Sentence
Processing
• Do you wait for sentence boundaries to interpret the meaning
of a sentence? No!
• As words come in, we construct
– partial meaning representation
– some candidate interpretations if ambiguous
– expectation for the next words
• Model
– Probability of each interpretation given words seen
– Stochastic CFGs, N-Grams, Lexical valence probabilities
SCFG + N-gram
Reduced Relative
Main Verb
S
S
Stochastic CFG
NP
NP
VP
NP
D
The
VP
N
VBN
cop arrested
D
the detective
The
VP
N
VBD
cop arrested
PP
by
SCFG + N-gram
Reduced Relative
Main Verb
S
S
NP
NP
VP
NP
D
The
VP
N
VBN
cop arrested
D
the detective
N-Gram
The
VP
N
VBD
cop arrested
PP
by
SCFG + N-gram
Different
Interpretations
Reduced Relative
Main Verb
S
S
NP
NP
VP
NP
D
The
VP
N
VBN
cop arrested
D
the detective
The
VP
N
VBD
cop arrested
PP
by
Predicting effects on reading time
• Probability predicts human disambiguation
• Increase in reading time because of...
– Limited Parallelism
• Memory limitations cause correct interpretation to be pruned
• The horse raced past the barn fell
– Attention
• Demotion of interpretation in attentional focus
– Expectation
• Unexpected words
Open for questions