Transcript Analysis of ASL motion capture data towards

The Biological Bases of Syntax-Semantics
Interface in Natural
Languages: Cognitive Modeling and
Empirical Evidence.
• event structure
Evguenia Malaia
Ronnie B. Wilbur
SLHS Department, Purdue University
[email protected]
1
WORLD-to-LANGUAGE
events in real world 
parsed out and expressed in sentences 
Each sentence centered around a
predicate/verb
…and this is what we want to model
Parsing observable events:
Human languages parse and formulate observable
events in a logically restricted fashion
(e.g. Son and Cole, 2008, Borer 1994, Ritter and Rosen 1998, Davis and Demidarche 2000, van Hout
2000, Hale and Keyser 1993, Van Valin 2007)
Vendler (1967)
four syntactically relevant semantic types of predicates:
– homogenous // atelic States and Activities
sleep (for an hour)
walk (for an hour)
– transitions // telic Achievements and Accomplishments
die (in an hour)
paint a portrait (in two weeks)
3
Model of Event Structure
• Pustejovsky 2001
Sub-event type
Subevent
States
S
Processes
P
Transitions: Achievements
S→S
Transitions: Accomplishments
P→S
Atelic
Telic transition
to end state
Model of Event Structure
• Ramchand (2008) - Divide events into
– Initiation phrase
– Process phrase
– Result phrase
InitP
ProcP
ResP
event structure
• Participant involved:
– Initiation phrase:
– Process phrase:
– Result phrase:
Initiator
Undergoer
Resultee
argument structure
5
Ramchand Event Structure Model Tree
initP (vP)
procP
Initiator
init
Undergoer
resP
proc
Resultee
XP
res
(Rheme)
- verbal morphology of individual languages can represent
individuated elements of event structure…
- which allows use of a single verbal root in multiple event
structures…
- to yield telic or atelic meanings.
6
Linguistic universals meet language processing
Do we have empirical evidence that telicity affects the
way human languages are processed?
Options:
• complete analysis of all known languages of the world
with respect to their event structure
• evidence from psycholinguistic and neurolinguistic
research
7
Linguistic universals meet language processing
• Reaction times:
- O’Bryan (2003) - independent effects of telicity and transitivity
:word maze experiment with reduced relative clauses.
:reaction time advantage on the preposition “by” for sentences with telic
verbs in the relative clause, as compared to those with atelic verbs;
:independent advantage for the second argument in sentences with
obligatorily transitive verbs (both telic and atelic).
- Friedmann, Taranto, Shapiro, and Swinney (2008)
priming effect for obligatorily intransitive telic verbs, but not for unergatives
obligatorily intransitive atelics.
• ERPs
- Malaia, Wilbur, Weber-Fox (in press) ERP evidence for telicity effects on
syntactic processing in garden-path sentences. Brain and Language.
The baby nursed/ burped by the mother rolled over.
The customer shaved / cheated by the barber left no tip.
The prisoner taught/ halted by the agent tried to escape.
8
ERPs on the RRC: HP group
verb
FC3
by
the
noun
µV
AN
FZ
P200
FC4
VEOG
-2.5
The actress awakened by the writer left in a hurry. (telic condition)
The actress worshipped by the writer left in a hurry. (atelic condition)
μV
100
ms
500
7.5
9
ERPs on the RRC: NP group
verb
by
the
noun
FC3
N100
FCZ
P200
FC4
VEOG
-2.5
The actress awakened by the writer left in a hurry. (telic condition)
The actress worshipped by the writer left in a hurry. (atelic condition)
μV
100
ms
500
7.5
10
ERPs on the RRC: HP and NP groups
HP group
by
NP group
the Agent
by
the Agent
CZ
-2.5 µV
ms
500 1000
-2.5 µV
ms
500 1000
2000
2000
7.5
7.5
Telic condition
Atelic condition
processing of syntactic information interacts with the previous semantic context.
(cf. Yamada & Neville, 2007; Osterhout, Holcomb, Swinney, 1994)
11
Frame alternations: telic verbs
New argument does not require re-assignment of thematic roles
awakened (intransitive)
actress
awakened (transitive)
writer
actress
12
Frame alternations: atelic verbs
New argument requires thematic role re-assignment
worshipped (intr.)
actress
worshipped (transitive)
writer
actress
13
Linguistic universals, language processing – unified
account?
Where did
grammatically relevant semantic
features
come from?
Research on Signed languages:
 tied to the visual modality
 in both production and perception
 provide the missing link
14
structure
in American
Sign Language
Telic Event
and atelic
predicates
in signed
languages
Telic event signs
a. change of aperture
handshape change
SEND
b. orientation change
HAPPEN
c. setting change
proximal/distal
POSTPONE
d. change of location
with contact
HIT
Atelic event signs
(a) RUN [tracing: straight]
(b) PLAY (tracing + TM)
(c) READ (tracing + TM)
15
Event Structure and SLs
• We already know that telicity plays a role in sign
morphology:
– [delayed completive] aspect only applies to telic stems
(Brentari 1998).
– Durative and continuative aspects cannot apply to telic
predicates (Wilbur 2005, in press).
– Some mouth non-manuals are distributed according to
predicate telicity type (Schalber 2004, 2006; Schalber &
Grose 2006) in Austrian Sign Language and American Sign
Language.
:: production differences reflecting semantic distinction of event
type
16
Event Visibility Hypothesis
• The Event Visibility Hypothesis (EVH) argues that
the semantics of event structure (subevents) are
visible in predicate sign formation.
• Can see if an end state is intended by way sign
movement comes to a stop:
- rapidly for end states
- regularly for no end state.
17
Motion Analysis
• ASL signs representing telic events appear to
contain ‘perceptually significant rapid
deceleration to a stop’.
• Hypothesis:
Signs representing telic events will have steeper
deceleration slopes than those representing atelic
events, because it will provide an end-marking to
indicate the final State.
18
Methodology
•
Stimuli:
–
•
Conditions
–
–
–
–
–
•
29 telic and 21 atelic signs chosen, randomized and
presented to native bilingual ASL signer.
Signs in isolation (done twice)
Signs in carrier phrase SIGN X AGAIN
Signs in medial sentence position IX3 X TODAY
Signs in final position TODAY IX3 X (NB: data analysis not
yet finished).
Each sign produced 4 times by signer
x 29 for telic = 116 cases for telics
x 21 for atelic = 84 cases for atelics.
Equipment
–
–
–
Gypsy 3.0 wired motion capture suit
Pair of 18-sensor Metamotion Cybergloves.
Six special motion capture ceiling mounted cameras.
19
Deceleration Results
Metric: the Slope from the velocity peak to the next velocity
minimum, reflecting the deceleration to(ward) a stop.
Slope
Telic/Atelic
Isolation 1 *
Isolation 2 *
Carrier Phrase**
Sentence 1 *
Atelic
-.09
-.12
-.12
-.14
Telic
-.14
-.18
-.23
-.23
Ratio
1.46
1.46
1.97
1.62
e.g. Mean telic
slope is 1.46
times steeper
than mean
atelic slope in
isolation
Difference between atelic and telic slopes is significant at
*p<.05 **p<.001
Sign movements have internal structure based on velocity
changes
Signers are making production distinctions
20
What is recipient doing for processing?
ASL production studies - left hemispheric activation of Broca’s area
(cf. Corina et al. 1999, Horowitz et al. 2003, Emmorey 2002, 2003, 2004).
ASL comprehension studies - bilateral activation in Broca’s area
(Levänen et al, 2001, Neville et al., 1998, etc)
- also typical for audiovisual stimuli comprehension in spoken
languages (cf. Capek et al., 2004).
Broca’s area is activated during both syntactic and semantic
processing in spoken languages - integrates syntactic and
semantic information during sentence comprehension
(Hagoort, 2005).
:: spoken and signed languages appear to be highly correlated
in the use of Broca’s area in the left hemisphere for
integration of structural and lexical linguistic information.
21
Hypotheses on ASL predicates
Telic predicates in ASL incorporate the end-state and
internal argument
Atelic predicates do not  simpler  smaller load?
Hypothesis: ASL predicate signs with distinct types of event
structure (telic vs. atelic) would elicit differentiated
activation patterns in Broca’s area (BA 44/45) of left IFG.
Pilot fMRI study of ASL predicates
Participants: 5 healthy adults - native ASL signers:
Age
Eye dominance
Hand dominance
Linguistic status
48
Right
Right
DEAF
20
Right
Left
DEAF
22
Right
Right
Child of Deaf Adult (hearing CODA)
21
Right
Right
Child of Deaf Adult (hearing CODA)
22
Right
Right
Child of Deaf Adult (hearing CODA)
Subjects were presented
with visual stimuli
consisting of telic and
atelic ASL signs in a
block paradigm, with
non-ASL gesture as a
baseline condition.
23
Data acquisition and fixed effects analysis
Subtraction paradigm was
used to compare
differential activation
during processing of telic
vs. atelic predicates.
Fixed effects analysis of the
pilot data from 5 subjects
(p<.001, uncorrected for
multiple
comparisons)
demonstrates
activation
clusters in Broca’s area
(BA 44/45, cluster size 15
voxels),
and
V5/MT+,
cluster size 5 voxels.
24
Interpreting the empirical data
• experiments demonstrate semantics/grammar
interaction in spoken and signed languages
• syntax of human languages cross-modally is
grounded in what can be construed as biological
perception.
The way events are perceived and conceptualized
is explicitly coded in syntax-semantics interface.
25
References
Bornkessel, I., & Schlesewsky, M. (2006). The Extended Argument Dependency Model: A Neurocognitive Approach to Sentence Comprehension Across Languages. Psychological Review, 113(4), 787821.
Brentari, D. (1998). Prosodic Model of ASL. Cambridge: MIT Press.
Capek, C.M., Bavelier, D., Corina, D., Newman, A.J., Jezzard, P., & Neville, H.J., 2004. The cortical organization of audio-visual sentence comprehension: An fMRI study at 4 Tesla. Cognitive Brain
Research, 20, 111-119.
Corina, D.P., San Jose-Robertson, L., Guillemin, A., High, J., & Braun, A. 2003. Language lateralization in a bimanual language. Journal of Cognitive Neuroscience, 15, 718-730.
Corina, D.P., McBurney, S.L., Dodrill, C., Hinshaw, K., Brinkley, J., & Ojemann, G. 1999. Functional roles of Broca’s area and supramarginal gyrus: Evigence from cortical stimulation mapping in a deaf
signer. NeuroImage, 10, 570-581.
Emmorey, K. 2002. Language, cognition, and the brain: Insights from sign language research. Mahwah, NJ: Lawrence Erlbaum and Associates.
Emmorey, K., Grabowki, T., 2004. Neural organization for sign versus speech production. Journal of Cognitive Neuroscience Supplement, F69, 205.
Emmorey, K., Herzig, M., 2003. Categorical versus gradient properties of classifier constructions in ASL. In K. Emmorey (Ed)., Perspectivees on classifier constructions in sign languages. Mahwah, NJ:
Lawrence Erlbaum and Associates.
Frazier, L., & Rayner, K. (1982). Making and correcting errors during sentence comprehension: eye movements in the analysis of structurally ambiguous sentences. Cognitive Psychology, 14, 178210.
Frisch, S., & Schlesewsky, M. (2001). The N400 indicates problems of thematic hierarchising. Neuroreport, 12, 3391-3394.
Just, M. A., & Carpenter, P. A. (1992). A capacity theory of comprehension: Individual differences in working memory. Psychological Review(99), 122-149.
Just, M. A., Carpenter, P. A., Keller, T. A., Eddy, W.F., & Thulborn, K.R. (1996). Brain Activation Modulated by Sentence Comprehension. Science, 274(5284), 114-116.
Hagoort, P. (2005). On Broca, brain, and binding: a new framework. Trends in Cog. Sci. 9, 416–423.
Hopf, J., Bayer, J., Bader, M., & Meng, M. (1998). Event-Related Brain Potentials and Case Information in Syntactic Amgibuities. Journal of Cognitive Neuroscience, 10(2), 264-280.
Kaan, E., & Swaab, T.Y. (2003). Repair, revision, and complexity in syntactic analysis: an electrophysiological differentiation. Journal of Cognitive Neuroscience, 15(1), 98-110.
Kaan, E., Wijnen, F., & Swaab, T.Y. (2004). Gapping: Electrophysiological evidence for immediate processing of “missing” verbs in sentence comprehension. Brain and Language, 89, 584-592.
King, J., & Kutas, M. (1995). Who did what and when? Using word- and causal- level ERPs to monitor working memory usage in reading. Journal of Cognitive Neuroscience, 7(3), 376-395.
Levänen, S., Uutela, K., Salenius, S., & Hari, R. 2001. Cortical representation of sign lanugage: Comparison of deaf signers and hearing non-signers. Cerebral Cortex, 11, 506-512.
Neville, H., Bavelier, D., Corina, D., Rauschecker, J., Kami, A., Lalwani, A., et al. 1998. Cerebral organization for language in deaf and hearing subjects: Biological constraints and effects of
experience. Proceedings of the National Academy of Science USA, 95, 922-929.
Malaia, E., Borneman, J., Wilbur, R.B. (2008) Analysis of ASL motion capture data towards identification of verb type. Symposium on semantics in systems for text processing, Venice, Italy,
September 22-24.
Malaia, E., Wilbur, R., Weber-Fox, C. (accepted) Effect of telicity on syntax-semantics integration in sentence processing: ERP evidence. Brain and Language
O'Bryan, E. (2003). Event Structure in Language Comprehension. Unpublished manuscript, University of Arizona.
Osterhout, L., Holcomb, P. J., & Swinney, D. A. (1994). Brain potentials elicited by garden-path sentences: Evidence of the application of verb information during parsing. . Journal of Experiment
Psychology: Learning, Memory, & Cognition, 20, 786-803.
Pustejovsky, J. (1991). The syntax of event structure. Cognition, 41(1-3): 47-81.
Ramchand, G. (2008). Verb Meaning and the Lexicon: A First Phase Syntax. Cambridge: Cambridge University Press.
Rathmann, C. (2005).Event Structure in ASL. Doctoral dissertation, University of Texas-Austin.
Sanz, M. (2000). Events and predication. A new approach to syntactic processing in English and Spanish. Amsterdam: John Benjamins.
Seegmiller, M., Ingraffea, K., & Townsend D. (2003). Role of telicity in sentence comprehension. Presentation at the “It’s About Time” workshop on Tense, Aspect, Modality, and Events. Michigan
State University, Lansing.
Streb, J., Henninghausen, E., & Rösler, F. (2004). Different Anaphoric Experssions are Investigated by Event-Related Brain Potentials. Journal of Psycholinguistic Research, 33(3), 175-201.
Vendler, Z. (1967). Linguistics in Philosophy, Cornell University Press, New York.
Van Valin (2007) Some universals of verb semantics. In Linguistic Universals, Mairal, R., Gil, J. eds. Cambridge University Press.
Weber-Fox, C., & Neville, H. J. (2001). Sensitive periods differentiate processing of open- and closed-class words: An ERP study of bilinguals. Journal of Speech Language and Hearing Research,
44(6), 1338-1353.
Yamada, Y., Neville, H.J. (2007). An ERP study of syntactic processing in English and nonsense sentences. Brain Research, 1130(1), 167-180.
Wilbur, R. B. (2003). Representations of telicity in ASL. Chicago Linguistic Society 39 (1): 354-368.
26
THANK YOU
We are grateful to Robin Shay, Gabriel Masters, Nicoletta
Adamo-Villani, Greg Tamer, Javier Gonzalez-Castillo, and
Purdue and Indianapolis sign language community for their
ongoing support of Purdue Sign Language Lab research.
This work was supported by NSF Research in Disabilities
Education grant, and by NIH grant DC00524 to R.B. Wilbur.
27
Ramchand’s factorical typology of predicates
Class
Syntactic status
Arguments involved
Examples
Transitive
Initiator, Undergoer
drive, push, paint
Transitive
Initiator, Path
eat, read, paint
Intransitive
Initiatori, Undergoeri
run
Transitive
Initiator, Undergoeri, Resulteei
throw, defuse
Transitive
Initiatori, Undergoeri, Result-Rheme
enter
IV
Intransitive
Initiatori, Undergoeri, Resulteei
arrive, jump
V
Ditransitive
Initiator, Undergoer, Resultee
give, throw
Undergoer
melt, roll, freeze
Undergoeri, Resulteei
break, tear
Initiatori, Undergoeri
dance, sleep
Undergoer
dry, clear
[init, proc]
I
II
[init, proc, res]
III
[proc]
VI
Intransitive
[proc, res]
VII
Intransitive
[init, proc, N]
VIII
N-conflation
[init, proc, A]
IX
A-conflation