Transcript Python for NLP and the Natrual Language Toolkit

Python for NLP and the Natural
Language Toolkit
CS1573: AI Application Development, Spring 2003
(modified from Edward Loper’s notes)
Outline
•
•
•
Review: Introduction to NLP (knowledge of
language, ambiguity, representations and
algorithms, applications)
HW 2 discussion
Tutorials: Basics, Probability
Python and Natural Language Processing
• Python is a great language for NLP:
– Simple
– Easy to debug:
• Exceptions
• Interpreted language
– Easy to structure
• Modules
• Object oriented programming
– Powerful string manipulation
Modules and Packages
•
Python modules “package program code and
data for reuse.” (Lutz)
–
•
•
Similar to library in C, package in Java.
Python packages are hierarchical modules
(i.e., modules that contain other modules).
Three commands for accessing modules:
1.
2.
3.
import
from…import
reload
Modules and Packages: import
• The import command loads a module:
# Load the regular expression module
>>> import re
• To access the contents of a module, use dotted
names:
# Use the search method from the re module
>>> re.search(‘\w+’, str)
• To list the contents of a module, use dir:
>>> dir(re)
[‘DOTALL’, ‘I’, ‘IGNORECASE’,…]
Modules and Packages
from…import
• The from…import command loads individual
functions and objects from a module:
# Load the search function from the re module
>>> from re import search
• Once an individual function or object is loaded
with from…import, it can be used directly:
# Use the search method from the re module
>>> search (‘\w+’, str)
Import vs. from…import
Import
from…import
• Keeps module
functions separate
from user functions.
• Requires the use of
dotted names.
• Works with reload.
• Puts module functions
and user functions
together.
• More convenient
names.
• Does not work with
reload.
Modules and Packages: reload
• If you edit a module, you must use the reload
command before the changes become visible in
Python:
>>> import mymodule
...
>>> reload (mymodule)
• The reload command only affects modules that
have been loaded with import; it does not
update individual functions and objects loaded
with from...import.
Introduction to NLTK
• The Natural Language Toolkit (NLTK) provides:
– Basic classes for representing data relevant to natural
language processing.
– Standard interfaces for performing tasks, such as
tokenization, tagging, and parsing.
– Standard implementations of each task, which can be
combined to solve complex problems.
NLTK: Example Modules
• nltk.token: processing individual elements of text,
such as words or sentences.
• nltk.probability: modeling frequency distributions
and probabilistic systems.
• nltk.tagger: tagging tokens with supplemental
information, such as parts of speech or wordnet sense tags.
• nltk.parser: high-level interface for parsing texts.
• nltk.chartparser: a chart-based implementation of
the parser interface.
• nltk.chunkparser: a regular-expression based
surface parser.
NLTK: Top-Level Organization
• NLTK is organized as a flat hierarchy of packages
and modules.
• Each module provides the tools necessary to
address a specific task
• Modules contain two types of classes:
– Data-oriented classes are used to represent information
relevant to natural language processing.
– Task-oriented classes encapsulate the resources and
methods needed to perform a specific task.
To the First Tutorials
• Tokens and Tokenization
• Frequency Distributions
The Token Module
• It is often useful to think of a text in terms of
smaller elements, such as words or sentences.
• The nltk.token module defines classes for
representing and processing these smaller
elements.
• What might be other useful smaller elements?
Tokens and Types
•
The term word can be used in two different ways:
1.
2.
•
•
To refer to an individual occurrence of a word
To refer to an abstract vocabulary item
For example, the sentence “my dog likes his dog”
contains five occurrences of words, but four vocabulary
items.
To avoid confusion use more precise terminology:
1.
2.
Word token: an occurrence of a word
Word Type: a vocabulary item
Tokens and Types (continued)
•
•
In NLTK, tokens are constructed from their
types using the Token constructor:
>>> from nltk.token import *
>>> my_word_type = 'dog'
'dog'
>>> my_word_token
=Token(my_word_type) ‘dog'@[?]
Token member functions include type and
loc
Text Locations
• A text location @ [s:e] specifies a region of a text:
– s is the start index
– e is the end index
• The text location @ [s:e]specifies the text beginning
at s, and including everything up to (but not including)
the text at e.
• This definition is consistent with Python slice.
• Think of indices as appearing between elements:
I
saw
a
man
0
1
2
3
4
• Shorthand notation when location width = 1.
Text Locations (continued)
• Indices can be based on different units:
– character
– word
– sentence
• Locations can be tagged with sources (files, other
text locations – e.g., the first word of the first
sentence in the file)
• Location member functions:
–
–
–
–
start
end
unit
source
Tokenization
• The simplest way to represent a text is with a
single string.
• Difficult to process text in this format.
• Often, it is more convenient to work with a list of
tokens.
• The task of converting a text from a single string
to a list of tokens is known as tokenization.
Tokenization (continued)
• Tokenization is harder that it seems
I’ll see you in New York.
The aluminum-export ban.
• The simplest approach is to use “graphic words”
(i.e., separate words using whitespace)
• Another approach is to use regular expressions to
specify which substrings are valid words.
• NLTK provides a generic tokenization interface:
TokenizerI
TokenizerI
• Defines a single method, tokenize, which
takes a string and returns a list of tokens
• Tokenize is independent of the level of
tokenization and the implementation
algorithm
Example
• from nltk.token import WSTokenizer
from nltk.draw.plot import Plot
#Extract a list of words from the corpus
corpus = open('corpus.txt').read()
tokens = WSTokenizer().tokenize(corpus)
# Count up how many times each word length
occurs wordlen_count_list = []
for token in tokens:
wordlen = len(token.type())
# Add zeros until wordlen_count_list is long
enough while wordlen >= len(wordlen_count_list):
wordlen_count_list.append(0)
# Increment the count for this word length
wordlen_count_list[wordlen] += 1
Plot(wordlen_count_list)
Next Tutorial: Probability
• An experiment is any process which leads to
a well-defined outcome
• A sample is any possible outcome of a
given experiment
• Rolling a die?
Outline
Review Basics
Probability
Experiments and Samples
Frequency Distributions
Conditional Frequency Distributions
Review: NLTK Goals
•
•
•
Classes for NLP data
Interfaces for NLP tasks
Implementations, easily combined (what is an
example?)
Accessing NLTK
•
What is the relation to Python?
Words
•
•
•
Types and Tokens
Text Locations
Member Functions
Tokenization
•
•
•
•
TokenizerI
Implementations
>>> tokenizer = WSTokenizer()
>>> tokenizer.tokenize(text_str)
['Hello'@[0w], 'world.'@[1w], 'This'@[2w],
'is'@[3w], 'a'@[4w], 'test'@[5w],
'file.'@[6w]]
Word Length Freq. Distribution Example
•
from nltk.token import WSTokenizer
from nltk.probability import SimpleFreqDist
# Extract a list of words from the corpus
corpus = open('corpus.txt').read()
tokens = WSTokenizer().tokenize(corpus)
# Construct a frequency distribution of word lengths
wordlen_freqs = SimpleFreqDist()
for token in tokens:
wordlen_freqs.inc(len(token.type()))
# Extract the set of word lengths found in the
corpus wordlens = wordlen_freqs.samples()
Frequency Distributions
• A frequency distribution records the
number of times each outcome of an
experiment has occurred
• >>> freq_dist = FreqDist()
>>> for token in document:
...
freq_dist.inc(token.type())
• Constructor, then initialization by storing
experimental outcomes
Methods
• The freq method returns the frequencey of a given
sample.
• We can find the number of times a given sample
occured with the count method
• We can find the total number of sample outcomes
recorded by a frequency distribution with the N
method
• The samples method returns a list of all samples
that have been recorded as outcomes by a
frequency distribution
• We can find the sample with the greatest number
of outcomes with the max method
Examples of Methods
• >>> freq_dist.count('the')
6
• >>> freq_dist.freq('the')
0.012
• >>> freq_dist.N()
500
• >>> freq_dist.max()
‘the’
Simple Word Length Example
• >>> from nltk.token import WSTokenizer
>>> from nltk.probability import FreqDist
>>> corpus = open('corpus.txt').read()
>>> tokens =
WSTokenizer().tokenize(corpus)
#
What is the distribution of word lengths in a
corpus? >>> freq_dist = FreqDist()
>>> for token in tokens:
...
freq_dist.inc(len(token.type()))
– What is the "outcome" for our experiment?
Simple Word Length Example
• >>> from nltk.token import WSTokenizer
>>> from nltk.probability import FreqDist
>>> corpus = open('corpus.txt').read()
>>> tokens = WSTokenizer().tokenize(corpus)
# What is the distribution of word lengths in a
corpus? >>> freq_dist = FreqDist()
>>> for token in tokens:
...
freq_dist.inc(len(token.type()))
– This length is the "outcome" for our experiment, so we use
inc() to increment its count in a frequency distribution.
Complex Word Length Example
• # define vowels as "a", "e", "i", "o", and "u"
>>> VOWELS = ('a', 'e', 'i', 'o', 'u')
# distribution for words ending in vowels?
>>> freq_dist = FreqDist()
>>> for token in tokens:
...
if token.type()[-1].lower() in VOWELS:
...
freq_dist.inc(len(token.type()))
• What is the condition?
More Complex Example
• # What is the distribution of word lengths for
# words following words that end in vowels?
>>> ended_in_vowel = 0 #Did last word end in
vowel?
>>> freq_dist = FreqDist()
>>> for token in tokens:
...
if ended_in_vowel:
...
Freq_dist.inc(len(token.type()))
...
ended_in_vowel=token.type()[-1].lower() in
VOWELS
Conditional Frequency
Distributions
• A condition specifies the context in which an
experiment is performed
• A conditional frequency distribution is a collection
of frequency distribtuions for the same
experiment, run under different conditions
• The individual frequency distributions are indexed
by the condition.
• NLTK ConditionalFreqDist class
• >>> cfdist = ConditionalFreqDist()
<ConditionalFreqDist with 0 conditions>
Conditional Frequency
Distributions (continued)
• To access the frequency distribution for a condition, use
the indexing operator :
>>> cfdist['a']
<FreqDist with 0 outcomes>
• # Record lengths of some words starting with 'a'
>>> for word in 'apple and arm'.split():
...
cfdist['a'].inc(len(word))
• # How many are 3 characters long?
>>> cfdist['a'].freq(3)
0.66667
• To list accessed conditions, use the conditions method:
>>> cfdist.conditions()
['a']
Example: Conditioning on a
Word’s Initial Letter
• >>> from nltk.token import WSTokenizer
>>> from nltk.probability import
ConditionalFreqDist
>>> from nltk.draw.plot import Plot
#
>>> corpus = open('corpus.txt').read()
>>> tokens =
WSTokenizer().tokenize(corpus)
>>>
cfdist = ConditionalFreqDist()
Example (continued)
• # How does initial letter affect word length?
>>> for token in tokens:
...
outcome = len(token.type())
...
condition = token.type()[0].lower()
...
cfdist[condition].inc(outcome)
• What are the condition and the outcome?
Example (continued)
• # How does initial letter affect word length?
>>> for token in tokens:
...
outcome = len(token.type())
...
condition = token.type()[0].lower()
...
cfdist[condition].inc(outcome)
• What are the condition and the outcome?
• Condition = the initial letter of the token
• Outcome = its word length
Prediction
• Prediction is the problem of deciding a likely
outcome for a given run of an experiment.
• To predict the outcome, we first examine a
training corpus.
• Training corpus
– The context and outcome for each run are known
– Given a new run, we choose the outcome that occurred
most frequently for the context
– Conditional frequency distribution finds the most
frequent occurrrence
Prediction Example: Outline
•
•
•
Record each outcome in the training corpus,
using the context that the experiment was
under as the condition
Access the frequency distribution for a given
context with the indexing operator
Use the max() method to find the most likely
outcome
Example: Predicting Words
• Predict word's type, based on preceding word
type
• >>> from nltk.token import WSTokenizer
>>> from nltk.probability import
ConditionalFreqDist
>>> corpus = open('corpus.txt').read()
>>> tokens =
WSTokenizer().tokenize(corpus) >>> cfdist =
ConditionalFreqDist() #empty
Example (continued)
• >>> context = None
# The type of the preceding word
>>> for token in tokens:
...
outcome = token.type()
...
cfdist[context].inc(outcome)
...
context = token.type()
Example (continued)
• >>> cfdist['prediction'].max()
'problems'
>>> cfdist['problems'].max()
'in'
>>> cfdist['in'].max()
'the‘
• What are we predicting here?
Example (continued)
We predict the most likely word for any context
Generation application:
>>> word = 'prediction'
>>> for i in range(15):
...
print word,
...
word = cfdist[word].max()
prediction problems in the frequency distribution of
the frequency distribution of the frequency
distribution of
For Next Time
•
•
•
HW3
To run NLTK from unixs.cis.pitt.edu, you
should add /afs/cs.pitt.edu/projects/nltk/bin to
your search path
Regular Expressions (J&M handout, NLTK
tutorial)