Transcript The Process of Data Mining

Data Mining Methodology
1
Why have a Methodology

Don’t want to learn things that aren’t true
 May not represent any underlying reality
○ Spurious correlation
○ May not be statistically significant or may be
statistically significant but coincidental
○ Because data mining makes less assumptions
about the data and searches through a richer
hypothesis space, this is a big issue
 Model overfitting is an issue
2
Why have a Methodology II

Data may not reflect relevant population
 Data mining normally assumes training data
matches the test and score data

Quick overview of how data used for DM
○ Training set used to build the model
○ Validation set used to tune model or select amongst
alternative models
○ Test set used to evaluate model & report quality
 For prediction tasks, test set must have the “answer”
○ Model eventually applied to score set, which for
predictive tasks, does not have the answer
○ Evaluation must always occur on data not used to
build or tune or select the model
3
Why have a Methodology III

Do not want to learn things that are not
useful
 May be already known
 May not be actionable
4
Hypothesis Testing

Data Mining is not usually used for
hypothesis testing
 B&L does not really say this
 Typical assumption is data already collected
and you have little influence on the process
○ Data may be in a data warehouse
○ usually do not modify the scenarios for
collecting the data or the parameters
○ Experimental design not part of data mining
○ Active learning is related to this, where you
carefully select the data to learn from
5
The Methodology (Fayyad)

According to the article by Fayyad et. al,
the main steps are:
 Data Selection
 Preprocessing
 Transformation
 Data Mining
 Interpretation/Evaluation
6
The Methodology (B & L)

According to Berry & Linoff, the main steps are:










Translate business problem into a DM problem
Select Data
Get to know the data
Create a model set
Fix problems with the data (“preprocess”)
Transform the data
Build models (“Data mining”)
Assess models (“Interpret/Evaluate”)
Deploy Models
Assess Results then start over
7
Steps in the Process: Selection

Many of the steps are not very complex,
so her some selective comments:
 Selection:
○ DM usually tries to use all available data
○ May not be necessary, can generate learning
curves where see how performance varies
with increasing amounts of data
○ Data Mining is not afraid of using lots of
variables (unlike statistics). But some data
mining methods (especially statistical ones)
do have problems with many variables.
8
Steps in the Process: Know the Data

Getting to know the data:
 always useful and also helps make sure you
understand the problem
 Data visualization can help
 Data mining is not really like a black box
where the computer does all of the work
○ having or generating good features (variables)
is critical. Data visualization can help
9
Steps in the Process: Create Model Set

Creating a model (training) set
 Sometimes you may want to form the training
set other than by random sampling
○ It is often recommended to balance the classes if
they are highly unbalanced
 Not really a good idea or needed. Can use cost-sensitive
learning instead, but we will address later
 May want to focus on harder problems
- Active learning skews the training data, but the purpose
is to save effort in manually labeling the training data
10
Steps in the Process: Create Model Set
 Data sets relevant to Data Mining
○ Training set: used to build initial model
○ Validation set: used to either tune model (e.g.,
pruning) or select amongst multiple models
○ Test set: used to evaluate goodness of model
 For predictive tasks, must have class labels
○ Score set: Data that model ultimately build for
 For predictive tasks, class labels are not available
 Note that training, validation and test data come
from labeled data
 Cross validation can maximize size of labeled data
○ 10-fold cross validation uses 90% for training and
10% testing. It will entail 10 runs.
11
Steps in the Process: Fix Data
Many data mining methods don’t need as
much variable “fixing” as statistical methods
 Types of fixing

 Missing values: many ways to fix
 Too many categorical values: reduce
○ Binning, etc.
 Numerical values skewed
○ Take log etc

Data preprocessing (Fayyad) may just alter
the representation
12
Steps in the Process: Transform

Aggregate data to a higher level
 Time series data often must be converted into
examples for classification algorithms
○ Phone call data aggregated from call level to
describe activity associated with a phone #/user

Construction of new features is part of this
step. Feature construction can be critical.
 Area of plot more useful for estimating value of
home than length and width.
13
Steps in the Process: Assess Model

Predictive models are assessed based on the
correctness of their predictions
 Accuracy is the simplest measure, but often not very
useful since not all errors are equal
○ we will learn more about this later
○ Lift curves are discussed in B&L (p 81)
 Lift ratio = P(class|sample) / P(class|population)
 Life only makes sense when we can be selective, like in direct
marketing where we don’t have to judge every response

Descriptive models can be hard to evaluate since
their may not be objective criteria
 How do you tell if a clustering is meaningful?
○ More on assessment methods later
14
Steps in the Process: Deploy

Research models are fine, we run them off
line and when we want to
 In a business, must deal with real-world issues
○ In the WISDM project, we want to classify activities
in real time. This is also needed for many fraud
detection models. Must be able to execute the
model and do it quickly, possibly on different
hardware.
 Some tools allow you to export the model as code
○ Even in off-line evaluation, may need to handle
huge amounts of data
15
Steps in the Process: Assess Results

True assessment is not just of model,
but includes the business context
 Takes into account all costs and benefits
 This may include costs that are very hard to
quantify
○ How much does a false negative medical test
cost it causes the patient to die of a
preventable disease?
16
Steps in the Process: Iterate

Data Mining is an iterative process
 Iteration can occur between most of the steps
 Example: You don’t like overall results so you
add another feature. You then assess its impact
to see if you should keep it.
 Example: You realize that assessment of your
model does not make sense and is missing
some costs, so you then incorporate these costs
into the model
17