Transcript Slide 1
Googleology is bad science Adam Kilgarriff Lexical Computing Ltd Universities of Sussex, Leeds 1 Web as language resource Replaceable or replacable? check 2 Very very large Most languages Most language types Up-to-date Free Instant access 3 How to use the web? Google or other commercial search engines (CSEs) not 4 Using CSEs No setup costs Start querying today Methods Hit counts ‘snippets’ Metasearch engines, WebCorp Find pages and download 5 Googleology CSE hit counts for language modelling 36 queries to estimate freq(fulfil, obligation) to each of Google and Altavista (Keller & Lapata 2003) finding noun-noun relations “we issue exact phrase Google queries of type noun2 THAT * noun1” Nakov and Hearst 2006 Small community of researchers Corpora mailing list Very interesting work Intense interest in query syntax Creativity and person-years 6 The Trouble with Google not enough instances max 1000 not enough queries max 1000 per day with API not enough context 10-word snippet around search term ridiculous sort order search term in titles and headings untrustworthy hit counts limited search syntax No regular expressions linguistically dumb lemmatised aime/aimer/aimes/aimons/aimez/aiment … not POS-tagged not parsed not 7 Appeal Zero-cost entry, just start googling Reality High-quality work: high-cost methodology 8 Also: No replicability Methods, stats not published At mercy of commercial corporation 9 Also: No replicability Methods, stats not published At mercy of commercial corporation Bad science 10 The 5-grams A present from Google All 1-, 2-, 3-, 4-, 5-grams with fr>=40 in a terabyte of English A large dataset 11 Prognosis Next 3 years Exciting new ideas Dazzlingly clever uses Drives progress in NLP 12 Prognosis Next 3 years Exciting new ideas Dazzlingly clever uses After 5+ years A chain round our necks Cf Penn Treebank (others? Brickbats?) Resource-led vs. ideas-led research 13 How to use the web? Google or other commercial search engines (CSEs) not 14 Language and the web Web is mostly linguistic Text on web << whole web (in GB) Not many TB of text Special hardware not needed We are the experts 15 Community-building ACL SIGWAC WAC Kool Ynitiative (WaCKY) Mailing list Open source WAC workshops WAC1, Birmingham 2005 WAC2, Trento (EACL), April 2006 WAC3, Louvain, Sept 15-16 2007 16 Proof of concept: DeWaC, ItWaC 1.5 B words each, German and Italian Marco Baroni, Bologna (+ AK) 17 What is out there? What text types? some are new: chatroom proportions is it overwhelmed by porn? How much? Hard question 18 What is out there The web a social, cultural, political phenomenon new, little understood a legitimate object of science mostly language we are well placed a lot of people will be interested Let’s study the web source of language data apply our tools for web use (dictionaries, MT) use the web as infrastructure 19 How to do it: Components 1. web crawler 2. filters and classifiers de-duplication 3. linguistic processing • Lemmatise, pos-tag, parse 4. Database • • Indexing user interface 20 1. Crawling How big is your hard disk? When will your sysadmin ban you? DeWaC/ItWaC Open source crawler: heritrix 21 1.1 Seeding the crawl Mid-frequency words Spread of text types Formal and informal, not just newspaper DeWaC Words from newspaper corpus Words from list with “kitchen” vocab Use Google to get seeds for crawls 22 2. Filtering non ‘running-text’ stripping Function word filtering Porn filtering De-duplication 23 2.1 Filtering: Sentences What is the text that we want? Lists? Links? Catalogues? … For linguistics, NLP in sentences Use function words 24 2.2 Filtering: CLEANEVAL “Text cleaning” Lots to be done, not glamorous Many kinds of dirt needing many kinds of filter Open Competition/shared task Who can produce the cleanest text?! Input: arbitrary web pages “gold standard” paragraph-marked plain text Prepared by people Workshop Sept 2007. do join us! http://cleaneval.sigwac.org.uk 25 3. Linguistic processing Lemmatise, POS-tag, parse Find leading NLP group for each language Be nice to them Use their tools 26 Database, interface Solved problem (at least for 1.5 BW) Sketch Engine 27 “Despite all the disadvantages, it’s still so much bigger” 28 How much bigger? Method Sample words 30 Mid-to-high freq Not common words in other major lgs Min 5 chars Compare freqs, Google vs ItWaC/DeWaC 29 Google results (Italian) Arbitrariness Repeat identical searches 9/30: > 10% difference 6/30: > 100% difference API: typically 1/18th ‘manual’ figure Language filter mista bomba clima mostly non-Italian pages use MAX and MIN of 6 lg-filtered results 30 Clima= Computational logic in multi-agent systems Centre for Legumes in Mediterranean Agriculture (5-char limit too short) 31 Ratios, Google:DeWaC WORD MAX MIN RAW CLEAN -------------------------------------------------------------besuchte 10.5 3.8 81840 18228 stirn 3.38 0.62 32320 11137 gerufen 7.14 3.72 66720 27187 verringert 6.86 3.46 52160 15987 bislang 24.4 11.6 239000 90098 brach 4.36 2.26 44520 19824 -------------------------------------------------------------MAX/MIN: max/min of 6 Google values (millions) RAW: DeWaC document frequency before filters, dedupe CLEAN: DeWaC document frequency after filters, dedupe 32 ItWaC:Google ratio, best estimate For each of 30 words Calculate ratio, max:raw Calculate ratio, min:raw Take mid-point and average: 1:33 or 3% Calculate raw:vert Average = 4.4 half (for conservativeness/uncertainty) = 2.2 3% x 2.2 = 6.6% ItWaC:Google = 6.6% 33 Italian web size ItWaC = 1.67b words Google indexes 1.67/.066 = 25 bn words sentential non-dupe Italian 34 German web size Analysis as for Italian DeWaC: 3% Google DeWaC = 1.41b words Google indexes 1.41/.03 = 44 bn words sentential non-dupe German 35 Effort ItWac, DeWac Less than 6 person months Developing the method (EnWaC: in progress) 36 Plan ACL adopts it (like ACL Anthology) (LDC?) Say: 3 core staff, 3 years Goals could be: English: 2% G-scale (still biggest part) 6 other major languages: 30% G-scale 30 other languages: 10% G-scale Online for Searching as in SkE Specifying, downloading subcorpora for intensive NLP “corpora on demand” Don’t quote me 37 Logjams Cleaning See CLEANEVAL Text type “what kind of page is it?” Critical but under-researched WebDoc proposal (with Serge Sharoff, Tony Hartley) (a different talk) 38 Moral Google, CSEs are wonderful Start today but bad science Not Good science, reliable counts We (the NLP community) have the skills With collective effort, mid-sized project Google-scale is achievable 39 Thank you http://www.sketchengine.co.uk 40 Scale and speed, LSE Commercial search engines banks of computers highly optimised code but this is for performance no downtime instant responses to millions of queries This proposal crawling: once a year downtime: acceptable not so many users 41 …but it’s not representative The web is not representative but nor is anything else Text type variation under-researched, lacking in theory Atkins Clear Ostler 1993 on design brief for BNC; Biber 1988, Baayen 2001, Kilgarriff 2001 Text type is an issue across NLP Web: issue is acute because, as against BNC or WSJ, we simply don’t know what is there 42 Oxford English Corpus Method as above Whole domains chosen and harvested control over text type 1 billion words Public launch April 2006 Loaded into Sketch Engine 43 Oxford English Corpus 44 Oxford English Corpus 45 Examples DeWaC, ItWaC Baroni and Kilgarriff, EACL 2006 Serge Sharoff, Leeds Univ UK English Chinese Russian English French Spanish, all searchable online Oxford English corpus 46 Options for academics Give up Niche markets, obscure languages Leave the mainstream to the big guys Work out how to work on that scale Web is free, data availability not a problem 47