Transcript poojax
Rotten or Fresh? NLP applied to Sentiment Analysis Two NLP tasks • Part-of-speech tagging: marking up a word in a text (corpus) as corresponding to a particular part of speech, based on both its definition, as well as its context • Negation scope analysis: determining which word a “negation word” (such as “no” or “not”) binds to Sentiment Analysis • Aims to determine the attitude of a speaker or a writer with respect to some topic • Well studied topic in both ML and NLP • Important not only from research perspective, but also for many companies. Question • Can part-of-speech tagging and negationscope analysis be used to improve sentiment analysis tools? Rotten Tomatoes • Aggregate movie and TV database • Includes many reviews for each movie, including a “fresh” or “rotten” indicator for each review • Ideal for training sentiment analysis classifier, since “ground truth” is given. • Offers API which gives “quote” of movie review and “rotten/fresh” Sentiment Analysis • Simple open source PHP tool (phpInsight) • Uses positive/neutral/negative word dictionary • Assigns positive/neutral/negative score to entire quote, proportional to number of positive/neutral/negative words contained POS tagger • Open source PHP interface to Stanford tagger, which we discussed in class • Generally accepted as one of the most widely used POS tagger Negation scope analysis • Start with dictionary of negation words: “'not', 'isn\'t', 'wasn\'t', 'aren\'t', 'neither', 'nor‘” • Use heuristics from relevant paper [1] to decide which word to bind to • Negation word is concatenated to bound word and added to opposite sentiment dictionary • For example, “notbad” would be added to positive dictionary Negation scope analysis • Negations tend to bind to adjectives/verbs • Also tend to bind to words proceeding negation word • Search window of five words after negation word, looking for first adjective/verb • Simple, but effective heuristic Naïve Bayes Classifier • Used Naïve Bayes classifier from Python’s NLTK toolkit to predict “rotten/fresh”-ness from sentiment scores • 500 pos/neg/neutral scores along with their rotten/freshness • Split into 400 training and 100 test examples • Data reshuffled 200 times and average precision was taken Methods • Method 1: Quote taken directly from RottenTomatoes • Method 2: Quote preprocessed using POS tagger, only adjectives passed to Sentiment Analysis • Method 3: Quote preprocessed using POS, adjectives/verbs found and used to assist in negation scope analysis Comparison of precision by method • 64.2% • 65.2% • 66.6% Rotten Tomatoes POS tagger Negation scope analysis Sentiment Analysis Freshness Classifier Conclusions • Adjectives-only gave us best results. We hypothesize this is due to higher information content of adjectives, resulting in less noise in data • Negation scope analysis increase not as much as adjective. Hypothesize this is due to relatively rare appearance of negations within sample quote. Conclusions • Although increases not very large, precision was aggregated over 200 runs, so confidence is still large • We conclude that POS tagging and negation scope analysis can improve sentiment analysis using simple “bag-of-words” model Fun Examples/Demo! References • 1.http://www.researchgate.net/publication/4 8340764_Scope_of_negation_detection_in_se ntiment_analysis