Transcript Getting your Text in good “mining” Shape.

Cleaning up your
Customer’s Dirty Language
for Text Mining
James A. Cox, Ph.D.
Software Development Manager, SAS Text Miner
Enterprise Miner Development
Copyright © 2005, SAS Institute Inc. All rights reserved.
Text Mining your customers: A window onto
their world
Copyright © 2005, SAS Institute Inc. All rights reserved.
2
But what if your window is not clean?
Copyright © 2005, SAS Institute Inc. All rights reserved.
3
Out of the mouth of … customers
*!@%$
From: Joe Shmo
To: whoever is interested
I LUV you’re new prodict!keep makeing work
beter!
--- No man is an island (by whoever it was that said
that)
Copyright © 2005, SAS Institute Inc. All rights reserved.
4
Text Parsing
money supply
+Clinton (Bill Clinton,…)
+Clinton (Clinton, NJ,…)
+bill (bills, billed)
+bill (bills)
regardless of
Copyright © 2005, SAS Institute Inc. All rights reserved.
Noun Group
Person
City
Verb
Noun
Prep
Keep
Keep
Keep
Keep
Keep
Drop
5
An organized view of your customers





Email, call centers, web surveys
Accessing info, merging it together
Eliminating run-on punctuation
Fixing misspellings
Discovering common themes using Text Miner!
Copyright © 2005, SAS Institute Inc. All rights reserved.
6
An example using Cell Phone data.






Access complaints from www.complaints.com
Eliminate extraneous text
Eliminate duplicates
Fix misspellings, abbreviations using %textsyn()
Analyze resulting data with SAS Text Miner
Compare to “uncleaned” text
Copyright © 2005, SAS Institute Inc. All rights reserved.
7
Copyright © 2005, SAS Institute Inc. All rights reserved.
8
Google interface
Copyright © 2005, SAS Institute Inc. All rights reserved.
9
KNOW what your customers are telling you
 Competitive advantage!!!
 If you don’t, your competitors might beat you to
it.
Copyright © 2005, SAS Institute Inc. All rights reserved.
10
Macros
 %textsyn and %puncexpand (renamed %tmpunct) will
be available in EA release this month
 These, along with other tips and techniques, and the
google interface can be downloaded from
ftp://ftp.sas.com/techsup/download/EMiner/M2004cd/
Copyright © 2005, SAS Institute Inc. All rights reserved.
11
Text Mining at SUGI 30
 Terry Woodfield, Patricia Cerrito, John Wallace -- whoops
 BOF session, Room 111 at 7 PM tomorrow night
 Manya Mayes --- predicting titles, tomorrow 3:30,
Room 112
Copyright © 2005, SAS Institute Inc. All rights reserved.
12
Copyright © 2005, SAS Institute Inc. All rights reserved.
13