Transcript slides
The Relevance of a Cognitive Model of the Mental Lexicon to Automatic Word Sense Disambiguation Martha Palmer and Susan Brown University of Colorado August 23, 2008 HJCL /Coling, Manchester, UK 1 Outline Sense Distinctions Annotation Sense Inventories created by human judgments An hierarchical model of sense distinctions OntoNotes WordNet PropBank and VerbNet Mappings to VerbNet and FrameNet Groupings Empirical evidence of replicable sense distinctions Reading response experiments based on the hierarchical model CLEAR – Colorado HJCL2008 2 Word sense in Machine Translation Different syntactic frames John left the room Juan saiu do quarto. (Portuguese) John left the book on the table. Juan deizou o livro na mesa. Same syntactic frame? Same sense? John left a fortune. Juan deixou uma fortuna. CLEAR – Colorado HJCL2008 3 Word sense in Machine Translation – not just syntax Different syntactic frames John left the room Juan saiu do quarto. (Portuguese) John left the book on the table. Juan deizou o livro na mesa. Same syntactic frame? Same sense? John left a fortune to the SPCA. Juan deixou uma fortuna. CLEAR – Colorado HJCL2008 HJCL2008--4 WordNet – Princeton (Miller 1985, Fellbaum 1998) On-line lexical reference (dictionary) Nouns, verbs, adjectives, and adverbs grouped into synonym sets Other relations include hypernyms (ISA), antonyms, meronyms Typical top nodes - 5 out of 25 (act, action, activity) (animal, fauna) (artifact) (attribute, property) (body, corpus) CLEAR – Colorado HJCL2008 5 WordNet – Princeton – leave, n.4, v.14 (Miller 1985, Fellbaum 1998) Limitations as a computational lexicon Contains little syntactic information No explicit lists of participants Sense distinctions very fine-grained, Definitions often vague Causes problems with creating training data for supervised Machine Learning – SENSEVAL2 Verbs > 16 senses (including call) Inter-annotator Agreement ITA 71%, Automatic Word Sense Disambiguation, WSD 64% Dang & Palmer, SIGLEX02 CLEAR – Colorado HJCL2008 6 PropBank – WSJ Penn Treebank Palmer, Gildea, Kingsbury., CLJ 2005 have been expecting Arg1 Arg0 Analysts a GM-Jaguar pact Arg0 *T*-1 Analysts have been expecting a GM-Jaguar pact that would give the U.S. car maker an eventual 30% stake in the British company. that would give Arg2 the US car maker Arg1 an eventual 30% stake in the British company expect(Analysts, GM-J pact) give(GM-J pact, US car maker, 30% stake) CLEAR – Colorado HJCL2008 PropBank - Palmer 7 Lexical Resource - Frames Files: give Roles: Arg0: giver Arg1: thing given Arg2: entity given to Example: double object The executives gave the chefs a standing ovation. Arg0: The executives REL: gave Arg2: the chefs Arg1: a standing ovation CLEAR – Colorado HJCL2008 PropBank - Palmer 8 Word Senses in PropBank Orders to ignore word sense not feasible for 700+ verbs Mary left the room Mary left her daughter-in-law her pearls in her will Frameset leave.01 "move away from": Arg0: entity leaving Arg1: place left Frameset leave.02 "give": Arg0: giver Arg1: thing given Arg2: beneficiary How do these relate to traditional word senses in VerbNet and WordNet? CLEAR – Colorado HJCL2008 PropBank - Palmer 9 Limitations to PropBank as a Sense Inventory Sense distinctions are very coarse-grained – only 700 verbs High ITA, > 94%, High WSD,> 90% CLEAR – Colorado HJCL2008 PropBank - Palmer 10 Limitations to Levin Classes as a Sense Inventory Dang, Kipper & Palmer, ACL98 Coverage of only half of the verbs (types) in the Penn Treebank (1M words,WSJ) Usually only one or two basic senses are covered for each verb Confusing sets of alternations Different classes have almost identical “syntactic signatures” or worse, contradictory signatures CLEAR – Colorado HJCL2008 VerbNet – Kipper, Dang, Palmer11 Intersective Levin Classes Kipper, Dang & Palmer, IJCAI00, Coling00 More syntactically and semantically coherent sets of syntactic patterns explicit semantic components relations between senses Multiple class memberships viewed as base classes and regular sense extensions VERBNET CLEAR – Colorado HJCL2008 12 VerbNet – Karin Kipper Class entries: Capture generalizations about verb behavior Organized hierarchically Members have common semantic elements, semantic roles and syntactic frames Verb entries: Refer to a set of classes (different senses) each class member linked to WN synset(s) (not all WN senses are covered) CLEAR – Colorado HJCL2008 VerbNet 13 14 HJCL2008 Mapping from PropBank to VerbNet (similar mapping for PB-FrameNet) Frameset id = leave.02 Sense = give VerbNet class = future-having 13.3 Arg0 Giver Agent/Donor* Arg1 Thing given Theme Arg2 Benefactive Recipient Baker, Fillmore, & Lowe, COLING/ACL-98 Fillmore & Baker, WordNetWKSHP, 2001 *FrameNet Label CLEAR – Colorado HJCL2008 VerbNet 15 Mapping from PB to VerbNet verbs.colorado.edu/~semlink CLEAR – Colorado HJCL2008 VerbNet 16 Mapping PropBank/VerbNet http://verbs.colorado.edu/~mpalmer/verbnet Extended VerbNet now covers 80% of PropBank tokens. Kipper, et. al., LREC-04, LREC-06 (added Korhonen and Briscoe classes) Semi-automatic mapping of PropBank instances to VerbNet classes and thematic roles, hand-corrected. VerbNet class tagging as automatic WSD Run SRL, map Arg2 to VerbNet roles, Brown performance improves Yi, Loper, Palmer, NAACL07 CLEAR – Colorado HJCL2008 VerbNet 17 Limitations to VN/FN as sense inventories Concrete criteria for sense distinctions Distinct semantic roles Distinct frames Distinct entailments But…. Limited coverage of lemmas For each lemma, limited coverage of senses CLEAR – Colorado HJCL2008 18 Sense inventory desiderata Coverage of WordNet Sense distinctions captured by concrete differences in underlying representations as in VerbNet and FrameNet Distinct semantic roles Distinct frames Distinct entailments Start with WordNet and be more explicit Groupings CLEAR – Colorado HJCL2008 19 WordNet: - leave, 14 senses, grouped WN1, WN5,WN8 Depart, a job, a room, a dock, a country WN6 WN10 WN2 WN 4 WN9 WN11 WN12 WN14 Wnleave_off2,3 WNleave_behind1,2,3 Leave behind, leave alone WNleave_alone1 WN13 Create a WN3 State WN7 WNleave_out1, Wnleave_out2 WNleave_off1 “leave off” stop, terminate exclude CLEAR – Colorado HJCL2008 Groupings 20 WordNet: - leave, 14 senses, groups, PB WN1, WN5,WN8 4 Depart, a job, a room, a dock, a country (for X) 1 2 WN6 WN10 WN2 WN 4 WN9 WN11 WN12 WN14 WNleave_off2,3 WNleave_behind1,2,3 Leave behind, leave alone WNleave_alone1 WN13 5 WN7an effect: Create a WN3 State /cause Left us speechless, leave a stain WNleave_out1, WNleave_out2 exclude CLEAR – Colorado 3 HJCL2008 WNleave_off1 stop, terminate: the road leaves off, not leave off your jacket, the results Groupings 21 Overlap between Groups and PropBank Framesets – 95% Frameset2 Frameset1 WN1 WN2 WN3 WN4 WN6 WN7 WN8 WN5 WN 9 WN10 WN11 WN12 WN13 WN19 WN 14 WN20 develop Palmer, Dang & Fellbaum, NLE 2007 CLEAR – Colorado HJCL2008 Groupings 22 Sense Hierarchy (Palmer, et al, SNLU04 - NAACL04, NLE07, Chen, et. al, NAACL06) PropBank Framesets – ITA >90% coarse grained distinctions 20 Senseval2 verbs w/ > 1 Frameset Maxent WSD system, 73.5% baseline, 90% Sense Groups (Senseval-2) - ITA 82% Intermediate level (includes Levin classes) – 71.7% WordNet – ITA 73% fine grained distinctions, 64% CLEAR – Colorado Tagging w/groups, ITA 90%, 200@hr, Taggers - 86.9% Semeval07 Chen, Dligach & Palmer, ICSC 2007 HJCL2008 Groupings 23 Groupings Methodology – Human Judges (w/ Dang and Fellbaum) Double blind groupings, adjudication Syntactic Criteria (VerbNet was useful) Distinct subcategorization frames call him a bastard call him a taxi Recognizable alternations – regular sense extensions: play an instrument play a song play a melody on an instrument CLEAR – Colorado HJCL2008 SIGLEX01, SIGLEX02, JNLE07 Groupings 24 Groupings Methodology (cont.) Semantic Criteria Differences in semantic classes of arguments Differences in the number and type of arguments Change of prior entity or creation of a new entity? Differences in types of events Often reflected in subcategorization frames John left the room. I left my pearls to my daughter-in-law in my will. Differences in entailments Abstract/concrete, human/animal, animate/inanimate, different instrument types,… Abstract/concrete/mental/emotional/…. Specialized subject domains CLEAR – Colorado HJCL2008 Groupings 25 Creation of coarse-grained resources Unsupervised clustering using rules (Mihalcea & Moldovan, 2001) Clustering by mapping WN senses to OED (Navigli, 2006). OntoNotes - Manually grouping WN senses and annotating a corpus (Hovy et al., 2006) Supervised clustering WN senses using OntoNotes and another set of manually tagged data (Snow et al., 2007) . CLEAR – Colorado HJCL2008 Groupings 26 OntoNotes Goal: Modeling Shallow Semantics DARPA-GALE AGILE Team: BBN, Colorado, ISI, Penn Skeletal representation of literal meaning Synergistic combination of: Text Syntactic structure Propositional structure Word Sense Word sense wrt Ontology Coreference CLEAR – Colorado HJCL2008 Treebank PropBank OntoNotes Annotated Text Co-reference 27 Empirical Validation – Human Judges the 90% solution (1700 verbs) Leave 49% -> 86% CLEAR – Colorado HJCL2008 Groupings 28 Question remains: Is this the “right” level of granularity? “[Research] has not directly addressed the problem of identifying senses that are distinct enough to warrant, in psychological terms, a separate representation in the mental lexicon.” (Ide and Wilks, 2006) Can we determine what type of distinctions are represented in people’s minds? Will this help us in deciding on sense distinctions for WSD? CLEAR – Colorado HJCL2008 29 Sense Hierarchy PropBank Framesets – ITA >90% coarse grained distinctions 20 Senseval2 verbs w/ > 1 Frameset Maxent WSD system, 73.5% baseline, 90% Sense Groups (Senseval-2) - ITA 82% Intermediate level (includes Levin classes) – 71.7% WordNet – ITA 73% fine grained distinctions, 64% CLEAR – Colorado HJCL2008 Groupings 30 Computational model of the lexicon based on annotation Hypothesis: Syntactic structure overtly marks very coarse-grained senses Subsequently subdivided into more and more fine-grained distinctions. A measure of distance between the senses The senses in a particular subdivision share certain elements of meaning. CLEAR – Colorado HJCL2008 31 Psycholinguistic theories on semantic representations – Susan Brown Discrete-senses theory (Klein and Murphy, 2001) Each sense has its own separate representation. Related senses and unrelated senses are stored and processed in the same way Shared-representation theory (Rodd et al., 2002) Related senses share a portion of their meaning representation CLEAR – Colorado HJCL2008 Brown, ACL08 32 Hypotheses Broke the radio (WN 3) Broke the vase (WN 5) DiscreteSenses CLEAR – Colorado HJCL2008 Brown, ACL08 33 Hypotheses Broke the radio (WN 3) DiscreteSenses CLEAR – Colorado Broke the vase (WN 5) Slow access HJCL2008 Brown, ACL08 34 Hypotheses Broke the radio (WN 3) DiscreteSenses CLEAR – Colorado Broke the vase (WN 5) Broke the horse (WN 12) Slow access HJCL2008 Brown, ACL08 35 Hypotheses Broke the radio (WN 3) DiscreteSenses CLEAR – Colorado HJCL2008 Broke the vase (WN 5) Broke the horse (WN 12) Slow access Slow access Brown, ACL08 36 Hypotheses Broke the radio (WN 3) DiscreteSenses Broke the vase (WN 5) Broke the horse (WN 12) Slow access Slow access SharedRepresentn CLEAR – Colorado HJCL2008 Brown, ACL08 37 Hypotheses Broke the radio (WN 3) Broke the vase (WN 5) Broke the horse (WN 12) DiscreteSenses Slow access Slow access SharedRepresentn Fast access CLEAR – Colorado HJCL2008 Brown, ACL08 38 Hypotheses Broke the radio (WN 3) Broke the vase (WN 5) Broke the horse (WN 12) DiscreteSenses Slow access Slow access SharedRepresentn Fast access Slow access CLEAR – Colorado HJCL2008 Brown, ACL08 39 Procedure Semantic decision task Judging semantic coherence of short phrases banked the plane “makes sense” hugged the juice doesn’t “make sense” Pairs of phrases with the same verb Primed with a sense in the first phrase Sense in the second phrase was one of 4 degrees of relatedness to the first CLEAR – Colorado HJCL2008 Brown, ACL08 40 Materials – stimuli based on groupings Prime Target Unrelated banked the plane banked the money Distantly related ran the track ran the shop Closely related broke the glass broke the radio Same sense cleaned the cup cleaned the shirt CLEAR – Colorado HJCL2008 Brown, ACL08 41 Mean response time (in ms) 1500 1400 1300 1200 1100 1000 900 800 700 Sam e CLEAR – Colorado Close HJCL2008 Distant Unrelated Brown, ACL08 42 Mean accuracy (% correct) 100 90 80 70 60 50 40 Sam e CLEAR – Colorado Close HJCL2008 Distant Unrelated Brown, ACL08 43 Broke the radio Broke the vase Broke the horse DiscreteSenses Slow access Slow access SharedRepresentn Fast access Slow access CLEAR – Colorado HJCL2008 Brown, ACL08 44 Broke the radio Broke the vase Broke the horse DiscreteSenses Slow access Slow access SharedRepresentn Fast access 1157 ms Slow access 1330 ms Closely related senses were processed more quickly (t32=5.85; p<.0005) CLEAR – Colorado HJCL2008 Brown, ACL08 45 Shared-representation theory Highly significant linear progression for response time (F1,32=95.8; p<.0001) and accuracy (F1,32=100.1; p<.0001). Overlapping portions of meaning representation Closely related senses share a great deal Distantly related senses share only a little Unrelated senses share nothing CLEAR – Colorado HJCL2008 Brown, ACL08 46 Implications for WSD Enumerating discrete senses may be a convenient (necessary?) construct for computers or lexicographers Little information loss when combining closely related senses Distantly related senses are more like homonyms, so they are more important to keep separate CLEAR – Colorado HJCL2008 47 Computational model of the lexicon based on annotation Hypothesis: Syntactic structure overtly marks very coarse-grained senses Subsequently subdivided into more and more fine-grained distinctions. A measure of distance between the senses The senses in a particular subdivision share certain elements of meaning. Computational model provided us with the stimuli and the factors for the experiments. CLEAR – Colorado HJCL2008 48 Synergy between cognitive models and computational models of the lexicon Hierarchical computational model provides stimuli and factors for response time experiment Confirmation of shared representation theory provides support for computational model and suggestions for richer representations All based on Human Judgments! CLEAR – Colorado HJCL2008 49 Future work in psycholinguistics Further analysis of categories of relatedness Closely related pairs had 2 literal senses Distantly related pairs often had 1 literal and 1 metaphoric sense Corpus study of literal and metaphoric uses of verbs and their frequency Use for further experiments on how people store and process sense distinctions Annotation study looking at how well people can make fine-grained sense distinctions in context CLEAR – Colorado HJCL2008 50 Future work in NLP Improvements to WSD Semi-supervised clustering of word senses Dynamic dependency neighbors (Dligach & Palmer, ACL08, ICSC 08) Using language models for sentence selection Hierarchical clustering? Soft clustering? Against PB, VN, FN, Groupings, WN, etc. Clustering of verb classes….. CLEAR – Colorado HJCL2008 51 Acknowledgments We gratefully acknowledge the support of the National Science Foundation Grant NSF0415923, Consistent Criteria for Word Sense Disambiguation and DARPA-GALE via a subcontract from BBN. We thank Walter Kintsch and Al Kim for their advice on the psycholinguistic experiments. CLEAR – Colorado HJCL2008 52 Leave behind, leave alone… John left his keys at the restaurant. We left behind all our cares during our vacation. They were told to leave off their coats. Leave the young fawn alone. Leave the nature park just as you found it. I left my shoes on when I entered their house. When she put away the food she left out the pie. Let's leave enough time to visit the museum. He'll leave the decision to his wife. When he died he left the farm to his wife. I'm leaving our telephone and address with you. CLEAR – Colorado HJCL2008 53 Category criteria WordNet OED Relatedness Rating (0-3) Unrelated pairs Different Unrelated 0-0.3 Distantly related different Related 0.7-1.4 Closely related Different 1.7-2.4 Same sense same 2.7-3.0 CLEAR – Colorado HJCL2008 Brown, ACL08 54 Broke the radio Broke the vase Broke the horse DiscreteSenses Slow access Slow access Shared-Repr Fast access 1157 ms 91% Slow access 1330 ms 71% Closely related senses were processed more quickly (t32=5.85; p<.0005) and accurately (t32=8.65; p<.0001) than distantly related and unrelated senses. CLEAR – Colorado HJCL2008 Brown, ACL08 55 Broke the radio Broke the vase Broke the horse DiscreteSenses Slow access Slow access Shared-Repr Fast access 1157 ms 91% Slow access 1330 ms 71% Closely related senses were processed more quickly (t32=5.85; p<.0005) and accurately (t32=8.65; p<.0001) than distantly related and unrelated senses. Distantly related senses were processed more quickly (t32=2.38; p<.0001) and accurately (t32=5.66; p<.0001) than unrelated senses. CLEAR – Colorado HJCL2008 Brown, ACL08 56