Transcript Data Mining and Its Applications

Data Mining and Its
Applications
Data Mining Techniques – For Marketing, Sales, and Customer Support, by
Michael J.A. Berry and Gordon Linoff, John Wiley & Sons, Inc., 1997.
Discovering Data Mining from concept to implementation, by Cabena, Harjinian,
Stadler, Verhees and Zanasi, Prentice Hall, 1997.
Building Data Mining Applications for CRM, by Alex Berson, Stephen Smith and
Kurt Thearling, McGraw Hall, 1999.
Data Mining Cookbook – Modeling Data for Marketing, Risk, and Customer
Relationship Management, by Olivia Parr Rud, John Wiley & Sons, Inc, 2001.
Mastering Data Mining – The Art and Science of Customer Relationship
management, by Michael J.A. Berry and Gordon S. Linoff, John Wiley & Sons, Inc,
2000.
Machine Learning, by Tom M. Mitchell, McGraw-Hill, 1997.
Data Mining – Concepts and Techniques, by Jiawei Han and Micheline Kamber,
Morgan Kaufmann, 2001.
Introduction to Data Mining, by Pang-Ning Tan, Michael Steinbach, and Vipin Kumar,
Addison Wesley, 2005.
Data Mining and Its Applications
1
Why Mine Data?
 Lots of data is being collected
and warehoused



Web data, e-commerce
purchases at department/
grocery stores
Bank/Credit Card
transactions
 Computers have become cheaper and more powerful
 Competitive Pressure is Strong
 Provide better, customized services for an edge (e.g. in Customer
Relationship Management)
Data Mining and Its Applications
2
Mining Large Data Sets - Motivation
 There is often information “hidden” in the data that
is not readily evident
 Human analysts may take weeks to discover useful
information
 Much of the data is never analyzed at all
Data Mining and Its Applications
3
What is Data Mining?
Many Definitions


Non-trivial extraction of implicit, previously unknown and
potentially useful information from data
Exploration & analysis, by automatic or
semi-automatic means, of
large quantities of data
in order to discover
meaningful patterns
Data Mining and Its Applications
4
What is (not) Data Mining?
What is not Data
Mining?


What is Data Mining?
– Look up phone
number in phone
directory
– Certain names are more
prevalent in certain US
locations (O’Brien, O’Rurke,
O’Reilly… in Boston area)
– Query a Web
search engine for
information about
“Amazon”
– Group together similar
documents returned by
search engine according to
their context (e.g. Amazon
rainforest, Amazon.com,)
Data Mining and Its Applications
5
Data Mining Tasks
 Prediction Methods

Use some variables to predict unknown or
future values of other variables.
 Description Methods
 Find human-interpretable patterns that
describe the data.
Data Mining and Its Applications
6
Three Main Data Mining Tasks
 Classification
 Clustering
 Association Rule Discovery
 There are many other approaches. But most
of them can be categorized into one of the
three approaches.
Data Mining and Its Applications
7
Data Mining for Retail Industry
 Retail industry: huge amounts of data on sales,
customer shopping history, etc.
 Applications of retail data mining
Identify customer buying behaviors
 Discover customer shopping patterns and trends
 Improve the quality of customer service
 Achieve better customer retention and satisfaction
 Enhance goods consumption ratios
 Design more effective goods transportation and
distribution policies

April 1, 2016
Data Mining: Concepts and Techniques
8
Customer Profiling
what kinds of customers were
profitable in last year?
 Question:
 Data
Customer details such as Age, Gender,
Occupation, Salary Levels, Account, etc.,
 Earnings from customers in last year.
 Data Mining
 Divide customers into profitability categories
according to earnings such as highly profitable,
profitable, non-profitable, loss.
 Find rules using data mining techniques
 Analyze the rules and take actions

Data Mining and Its Applications
9
Customer Profiling: Rules
IF age > 30 and Age <=45 and
occupation is professional and
salary level is between 50,000 and 70,000
Then this user is profitable
The rules are with some statistic support such
as support and confidence.
Data Mining and Its Applications
10
Customer Segmentation
 Customer segmentation is a process to divide
customers into different groups or segments.
Customers in the same segment have similar needs
or behaviors so that similar marketing strategies
or service policies can be applied to them.
 Customer segments are required in several
business areas including






Marketing
Customer services
Products and service development
Sales promotion
Purchase recommendation
Customer retention
Data Mining and Its Applications
11
Customer Retention


In most industries the cost of retaining a customer,
subscriber or client is substantially less than the
initial cost of obtaining that customer.
Question:


Find out what kinds of customers tend to churn and build a
model which can predict the likely-to-churn customers.
Data mining solution:



Collect data about the customers who have churned.
Select a set of customers who have been loyal.
Merge the two data sets to form training, testing and
evaluation data sets.
Data Mining and Its Applications
12
Financial Products Recommendation
 Mellon Bank Corporation is a major financial services
company head-quarted in Pittsburgh.



Build an extendible loan secured by the values of a
client’s own property.
Achieve the highest possible Return On Investment.
Based on customers with DDA, build a model for HELOC.
Data Mining and Its Applications
13
Data Preparaton
 The primary data source was the approximately
40,000 Mellon customers who had (or once had)
HELCOCs and DDAs.
 Data




Demographic data sourced both internally and externally (age,
income, length of residence, and other indicators of economic
condition)
DDA data (history of loan balance over 3, 6, 9, 12, 18 months,
history of returned checks, history of interest rates.
Property data sourced externally (home purchase price, loan-tovalue ratio)
Other data related to credit worthiness
 Use 120 variables
Data Mining and Its Applications
14
Data Mining and Its Applications
15
Responders
Data Mining and Its Applications
16
Basket Analysis
Data Mining and Its Applications
17
Basket Analysis
A B C
A CD
Rule
AD
CA
AC
B&CD
B CD
ADE
B C E
Support
Confidence
2/5
2/5
2/5
1/5
2/3
2/4
2/3
1/3
Data Mining and Its Applications
18
The Impact of Fraud
 GAO (The United States General Accounting
Office) cited $19.1 billion in improper
government payments in 17 major programs
for fiscal year 1998.







Medicare $12.6 Billion
Supplemental Security Income $1.6 B
The Food Stamp Program $1.4 B
Old Age and Survival Insurance $1.2 B
Disability Insurance $941 Million
Housing Subsidies $847 Million
Veterans’ Benefits, Unemployment Insurance and
Others $514 Million
Data Mining and Its Applications
19
Background
 HIC (The Health Insurance Commission) in
Australia is a federal government agency.
 HIC pays insurance claims more than 20
million Australian dollars and pay out about
A$8 billion in funds every year.
 More than 300 million transactions are
processed and stored every year. 1.3TB in
five year.
Data Mining and Its Applications
20
Preventing Fraud and Abuse
 Business Objectives

The focus of the HIC project was on the
recent and steady 10% annual rise in the cost
of pathology claims for clinical tests.
 Approaches
To identify potential fraudulent claims or
claims arising from inappropriate practice, and
 To develop general profiles of the GP practices
in order to compare practice behaviors of
individual GPs.

Data Mining and Its Applications
21
Data Proprocessing
 Two databases
 Episode Database
• One Episode record records a patient visit.
• In total, 6.8 million records.
• There were 227 different pathology tests.

GP (doctor) database
• There are 17,000 records related to active GPs
 The behavior of 10,409 GPs was to be studied.
 A matrix of 10,409 by 227 elements.
 The elements were then scaled from 0 to 1 with respect
to the total number of tests of each kind.
Data Mining and Its Applications
22
Input to Segmentation
Data Mining and Its Applications
23
Overview
Data Mining and Its Applications
24
Data Mining
 They conducted association rule mining, when support =
0.25%，the team decided that the presence of some
tests in the input database was causing spurious rules
to be revealed (Pathology Episode Initiation (PEI)).
 PEI tests depend on who ordered them and where they
were ordered.
 When the PEI tests were removed, the number of rules
dropped significantly.
Data Mining and Its Applications
25
Result Analysis
 A request for a microscopic examination of
feces for parasites (OCP) was associated
with a cultural examination of feces (FCS)
in 0.85% of cases.
A
92.6% chance that if OCP tests were
requested, they would be done with FCS.
 A 0.61% of chance, OCP was associated with a
different more expensive test called MCS32,
which costs A$13.55 per test.
Data Mining and Its Applications
26
GP Profiles
Data Mining and Its Applications
27
Discussions
 Segment 13:
Represent the majority of traditional GPs who
are practicing conventionally. 5,450 GPs. Total
52% of GPs.
 Only 6.2% of the medical pathology tests

 Segment 4:
 54 GPs. Only 0.51% of GPs.
 2.7% of the medical pathology tests.
Data Mining and Its Applications
28
Financial Data Mining: News Sensitive
Stock Prediction
Data Mining and Its Applications
29
Advanced Topics
 Sequential Mining
 Time-Series Mining
 Spatial Mining
 Web Mining
 Social Network Mining
 Text Mining
 Data Streaming Mining
 Mining and Privacy
Data Mining and Its Applications
30
Examples of Data Mining Systems
 Mirosoft SQLServer
Integrate DB and OLAP with mining
 Support OLEDB for DM standard
SAS Enterprise Miner
 A variety of statistical analysis tools
 Data warehouse tools and data mining algorithms
IBM Intelligent Miner
 A wide range of data mining algorithms
 Scalable mining algorithms
 data preparation, and data visualization tools
 Tight integration with IBM's DB2 RDBMS
Clementine (SPSS)
 An integrated data mining development environment for endusers and developers
 Multiple data mining algorithms and visualization tools
Weka (http://cs.waikato.ac.nz/ml/weka)
 A free data mining tool.





Data Mining: Concepts and Techniques
31