Transcript WN Domains
Departament de Llenguatges i Sistemes Informàtics Universitat Politècnica de Catalunya Automatic Assignment of Domain Labels to WordNet Mauro Castillo V. Francis Real V. German Rigau C. GWC 2004 Outline •Introduction •WordNet •WN Domains •Experimentation •Evaluation and results •Discussion •Conclusions Introduction • To semantically enrich any WN version with the semantic domain labels of MultiWordNet Domains • WN is an standard resource for semantic processing • Effectiveness of Word Domain Disambiguation • The work presented explores the automatic and sistematic assignment of domain labels to glosses • Proposed Method can be used to correct and verify the suggested labeling WordNet • The version WN1.6 was used because of the availability of WN Domains WN Domains TOP pure_science mathematics geometry statistics biology WordNet Domain hierarchy botany developed at IRST zoology (Magnini and Cavagliá, 2000) entomology anatomy ... ... ... WN Domains • The synsets have been annotated semiautomatically with one or more labels • Most of synsets it has single a label # 1 2 3 4 5 nom 56458 8104 1251 210 2 verb 11287 743 88 8 1 adj 16681 1113 113 8 0 adv 3460 109 6 0 0 % 88,2020 10,1050 1,4632 0,2268 0,0030 Distribution of domain labels for synset Average labels for synset noun = 1.170 verb = 1.078 adj = 1.076 adv = 1.033 WN Domains • A domain may include synsets of different syntactic categories : e.g. MEDICINE doctor#1 (n) operar#7 (v) medical#1 (a) clinically#1 (r) • A domain label may also contain senses from different Wn subhierarchies. e.g. SPORT athleta#1 game-equipment#1 sport#1 playing-field#1 life-form#1 physical-object#1 act#2 location#1 WN Domains • Synsets that have more than one label, do not seem to follow any pattern • sultana#n#1 (pale yellow seedless grape used for raisins and wine) Botany Gastronomy • morocco#n#2 (a soft pebble-grained leather made from goatskin; used for shoes and book bindings etc.) Anatomy Zoology • canicola_fever#n#1(an acute feverish disease in people and in dogs marked by gastroenteritis and mild jaundice) Medicine Physiology Zoology • blue#n#1, blueness#n#1 (the color of the clear sky in the daytime; "he had eyes of bright blue") Color Quality WN Domains • FACTOTUM : Used to mark the senses of WN that do not have a specific domain • STOP Senses: The synsets that appear frequently in different contexts, for instance: numbers, colours, etc. Applications of WN Domains • Word Sense Disambiguation • Word Domain Disambiguation • Text Categorization, etc. Experimentation • Process to automatically assign domain labels to WN1.6 glosses • Validation procedures of the consistency of the domains assignment in WN1.6, and especially, the automatic assignment of the factotum labels POS noun verb adj adv FAC no FAC 66025 58252 12127 4425 17915 6910 3575 1039 %FAC 11,77 63,51 61,42 70,93 Distribution of synset with and without the domain label factotum in WN1.6 Experimentación Test set was randomly selected (around 1%) and the other synsets were used as a training set POS noun verb FAC no FAC 572 647 43 121 %FAC 11,90 60,33 Corpus test for nouns and verbs Experimentation castle#n#4, castling#n#1 CHESS SPORT castle castling | interchanging the positions of the king and a rook castle chess castle sport castling chess castling sport interchanging chess interchanging sport interchanging chess interchanging sport interchanging chess interchanging sport king chess king sport rook chess rook sport Calculation of frequency castle chess 68 castle sport 27 castle hystory 18 castle archictecture 57 castle law 12 castle tourism 24 … Experimentation Measures M1: Square root formula c(w,D) - 1/N*c(w)c(D) c(w,D) M2: Association Ratio Ar(w,D) = Pr(w|D)log2(Pr(w|D) / Pr(w)) M3: Logarithm formula log2(N*c(w,D) / c(w)c(D)) Experimentation TRAINING CALCULATION orange orange orange orange orange orange orange orange orange orange botany gastronomy color jewellery entomology quality hunting geology chemistry biology 10.1739451057135 4.98225066954225 3.28232334801756 1.49369255002054 1.23243498322359 1.17822271128967 0.412524764820793 0.293707167933641 0.166183492890361 0.110492358490017 MATRIX OF WEIGHTS VALIDATION Experimentation 06950891 leader#n#1 PERSON politics history religion person mythology commerce person 19.94 law 8.01 economy 4.74 religion 4.24 anthropology 3.74 sexuality 3.53 politics 3.49 4.30 3.33 2.19 1.78 1.17 1.11 leader | a person law factotum computer_science mathematics grammar play linguistics politics who variant VD = weigth(wi,dj)*percentage 2.70 2.09 2.05 1.83 1.68 1.57 1.54 1.35 rules gloss tourism industry person mechanics factotum occultism pedagogy or 1.64 1.54 1.46 1.26 1.24 0.98 0.93 psychology 0.96 factotum 0.82 guides or inspires others POSITION 1: person POSITION 2: politics POSITION 3: law ... ... person = 30.23 = 13.40 = 11.08 Evaluation y Results: nouns AP: Accuracy first label AT: Accuracy all labels P : Precision MiD : Measures the success of each formula (M1, M2 or M3) when the first proposed label is correct (or subsumed as correct one in the domain hierarchy). R : Recall F1 : 2PR/(P+R) N M1A M1D M2A M2D M3A M3D AP 70,94 74,50 45,75 52,09 66,77 71,56 AT 79,75 84,85 50,39 57,50 74,50 81,45 MiA : Measures the success of each formula (M1, M2 or M3) when the first proposed label is correct P 64,74 68,88 42,73 48,75 60,86 66,54 R 68,25 72,62 43,12 49,21 63,76 69,71 F1 66,45 70,70 42,92 48,98 62,27 68,09 Results for nouns with factotum CF N M1A M1D M2A M2D M3A M3D AP 73,95 78,50 52,45 59,44 74,48 78,85 AT 81,82 87,24 57,52 65,21 82,69 88,64 P 66,81 71,24 49,32 55,94 68,41 73,33 R 68,68 73,24 48,24 54,71 69,41 74,41 F1 67,73 72,23 48,77 55,32 68,91 73,87 Results for nouns without factotum SF Evaluation y Results: verbs AP: Accuracy first label AT: Accuracy all labels P : Precision MiD : Measures the success of each formula (M1, M2 or M3) when the first proposed label is correct (or subsumed as correct one in the domain hierarchy). R : Recall F1 : 2PR/(P+R) V M1A M1D M2A M2D M3A M3D AP 51,24 51,24 13,22 16,53 23,14 24,79 AT 57,02 57,02 14,88 19,83 28,10 29,75 MiA : Measures the success of each formula (M1, M2 or M3) when the first proposed label is correct P 47,26 47,26 12,68 16,90 21,94 23,23 R 50,74 50,74 13,24 17,65 25,00 26,47 F1 48,94 48,94 12,95 17,27 23,37 24,74 Results for verbs with factotum CF V M1A M1D M2A M2D M3A M3D AP 69,77 74,72 20,93 41,86 41,86 53,49 AT 76,74 83,72 25,58 51,16 55,81 67,44 P 64,71 69,23 19,64 38,60 39,34 46,77 R 55,93 61,02 18,64 37,29 40,68 49,15 F1 60,00 64,86 19,13 37,93 40,00 47,93 Results for verbs without factotum SF Evaluation y Results • On average, the method assigns: Noun : 1.23 domains labels (1.170) Verb : 1.20 domains labels (1.078) • We obtain better results with nouns • The best average results were obtained with the M1 measure • The first proposed label (noun): 70% accuracy • The results of verbs are worse than nouns, one of the reasons may be the high number of verbal synsets labels with factotum domain Discussion Monosemic words: credit application#n#1 (an application for a line of credit) Domains: SCHOOL Proposal 1. Banking Proposal 2. Economy Banking economy banking Discussion Relation between labels: Academic_program#n#1 (a program of education in liberal arts and sciences (usually in preparation for higher education)) Domains: PEDAGOGY Proposal 1. School Proposal 2. University pedagogy school university Discussion Relation between labels: shopping#n#1 (searching for or buying goods or services: "went shopping for a reliable plumber"; "does her shopping at the mall rather than down town") Domains: ECONOMY Proposal 1. Commerce social_science commerce economy Discussion Relation between labels: Fire_control_radar#n#1 (radar that controls the delivery of fire on a military target) Domains: MERCHANT_NAVY Proposal 1. Military social_science transport merchant_navy military Discussion Uncertain cases: birthmark#n#1 (a blemish on the skin formed before birth) Domains: QUALITY Proposal 1. Medicine bardolatry#n#1 (idolization of William Shakespeare) Domains: RELIGION Proposal 1. History Proposal 1. Literature Conclusions • The procedure to assign automatically domain labels to WN gloss seems to be dificult • The proposal process is very reliable with the first proposal labels • The proposal labels are ordered by priority • It is posible to add new correct labels or validate the old ones Departament de Llenguatges i Sistemes Informàtics Universitat Politècnica de Catalunya Automatic Assignment of Domain Labels to WordNet Mauro Castillo V. Francis Real V. German Rigau C. GWC 2004