Transcript Data Preprocessing

Data Pre-processing
Lecture 3
Gonca Gulser
What is it?
Ideas????
Definition: Series of actions to improve the quality of data for
making it ready to any kind of analysis
Possible Problems...

Identifying INCOMPLETE data




Eliminate NOISE



Errors
Outliers (should we always get rid of them? Any special case?)
Identify INCONSISTANCY


Missing attribute
Lack of Attribute Values
Contain only aggregate data
A value can be code differently across whole DB
Too DISPERSE to analyse

Too many attributes for any algorithm.
Forms of Data Prepossessing
Forms of Data Pre-process

Data Cleaning

Data Integration

Data Transformation

Data Reduction
Data Cleaning

Missing Value Handling

Smooth out Noise

Correct inconsistencies
Missing Value Handling
(Data Cleaning)
Any ideas???
Missing Value Handling
(Data Cleaning)
Ignore Tuple
 not very effective especially if the tuple contains several missing.
 It is poor when the percentage of missing values per attribute varies considerably
2) Fill the missing Manually
• Time consuming especially in huge datasets
3) Use global constant to fill in the missing
• Replace with “unknown” or “missing”
• Not useful because may lead Data mining tool to produce interesting results for them
4) Use attribute mean/mode/median to fill the missing value
• What about categorical data?
• Why is mean dangerous?
• Skewed data
5) Use the attribute mean for all sample belonging to the same class
• Categorize the attributes & Use mean of each category to fill the missing
6) Use the most probable value to fill in the missing value
• Regression/inference-based tools/ decision tree
1)


Methods from 3 to 6 are biased. - The generated values might not be correct so it increase the algorithm's
error rate
6th method is the most popular one because it uses more past data to predict the current situation... You
must be sure that your past data is reliable...
Smooth out Noise
(Data Cleaning)
What is Noise? - Random error or variance in the measured data
Methods
1) Binning



Sort the data
Divide into equal chunks (bins)
Get the mean of each bin and replace|| Smoothing with boundaries
Sorted data for price = 4,8,15,21,21,24,25,28,34
Partition into Bins:
Bin1: 4,8,15
Bin2: 21,21,24
Bin3: 25,28,34
Smoothing By means
Bin1: 9,9,9
Bin2: 22,22,22
Bin3: 29,29,29
Smoothing By boundaries
Bin1: 4,5,15
Bin2: 21,21,24
Bin3: 25,25,34
Smooth out Noise
(Data Cleaning) cont...
2) Combined Human and Computer Power
By any given algorithm let computer produce an outlier or noise list called “surprise”
Then go over the list and remove the irrelevant data by hand...
It is easier and time saving than go through all data set
3) Regression
Linear
MultiLinear
Logarithmic
4) Other methods
Data reduction involving discretization (divide data into sub-categories like low\medium\high) such as
decision tree reduce the data step by step
Concept Hierarchies- a form of discretization also used for noise
Forms of Data Pre-process

Data Cleaning

Data Integration

Data Transformation

Data Reduction
Data Integration and Transformation

What is it?
INTEGRATION: Merge Data from multiple data sources
TRANSFORMATION: Transform data into an appropriate format for any given data
mining algorithm.
Data Integration

Schema Integration
Meta Data can solve the problem... ex: Cut_id and cust_number are same thing

Redundancy




An attribute is redundant if it can be derived from any given attribute in the
database ex: annual revenue
If result > 0, then A and B are positively correlated
Can be detected by correlation analysis
If result < 0, then A and B are negatively correlated
If result = 0, then A and B are not correlated
Remove one of the duplicate attributes



Detection and Resolution of Data Conflicts
Because of different metrics and different perceptions on data, multiple sources
have same data in totally different formats and logic.
Examples:
•
A unit may be hold in European metric system (kg) in one data source and in
British metric system (pounds) in another data source
•
A price of a room may be in different currencies and also may contain different
attributes (such as Hilton's room price may include breakfast but Sheraton’s may
not)
Data Transformation

Transform or consolidate data into appropriate forms for Data
Mining

Methods




Smoothing – Get rid of noise
Aggregation – Summary or aggregation operation. To use data to calculate new
measure. (calculated measure in OLAP cubes) e.x. Using daily sales to calculate
quarterly or annual sales.
Generalization – Transform into higher level concept e.x. Concept hierarchies or
divide age into young\medium\old
Normalization – fall the data into specific range usually -1 to 1



Useful for classification and clustering algorithms.
The classification algorithms like neural networks, needs data into the range between -1 to 1
Distance based clustering algorithms like k-means does not require data into range. However, we
usually need to normalize values in order not give over emphasize on naturally higher value
attributes. e.x. If we put age and salary as attribute we need to normalize both in order to get rid
of the effects of higher values of salary.
Data Transformation cont...
Normalization Algorithms

Min-Max normalization – performs linear transformation on the original data
𝑂𝑟𝑖𝑔𝑖𝑛𝑎𝑙 𝑉𝑎𝑙𝑢𝑒−min(𝑝𝑜𝑝𝑢𝑙𝑎𝑡𝑖𝑜𝑛)
max 𝑝𝑜𝑝𝑢𝑙𝑎𝑡𝑖𝑜𝑛 −min(𝑝𝑜𝑝𝑢𝑙𝑎𝑡𝑖𝑜𝑛)
e.x Suppose that the min and max values for the attribute income are
$12,000 and $98,000 we would like to map the income to the range
0.0, 1.0. By min-max normalization a value of $73,600 for income is
transformed to
(73,600-12,000)/(98,000-12,000)= 0.716
Data Transformation cont...

Z-score normalization – the values of an attribute is
normalized based on mean and the standard deviation of the
attribute.
𝑂𝑟𝑖𝑔𝑖𝑛𝑎𝑙 𝑉𝑎𝑙𝑢𝑒 −𝑚𝑒𝑎𝑛(𝑝𝑜𝑝𝑢𝑙𝑎𝑡𝑖𝑜𝑛)
𝑠𝑡𝑑𝑒𝑣(𝑝𝑜𝑝𝑢𝑙𝑎𝑡𝑖𝑜𝑛)
e.x Suppose that the mean and the standard deviation of income are $54.000
and $16,000 respectively. With z-score normalization, a value for $73,600 is
transformed to
(73,600-54,000)/16,000 = 1.225
Data Transformation cont...

Normalization by decimal scaling – normalizes by moving the
decimal points moved depends on the maximum absolute
value of the attribute
Vnormalize=
𝑣
10𝑗
where, j is the smallest integer that max(vnormalize)=1
e.x. Suppose that the value range for A is -986 – 917. The maximum absolute
value for A is 986. To normalize based on decimal scaling we need to divide
each value by 1000 (j=3) so that -986 normalizes to -0.986
Data Transformation cont...

Attribute Construction (feature construction) – new attributes
are constructed and added from the given set of attributes to
help the mining process
e.x adding attribute area to data set by using height and width
Forms of Data Pre-process

Data Cleaning

Data Integration

Data Transformation

Data Reduction
Data Reduction


Make the amount of data smaller
Be Careful!!!!




Reduced dataset should represent the original data set
Results of reduces dataset should be reflect the original sets data
Reduction should ease and fasten the data mining process
Data Reduction Strategies
 Data Cube Aggregation – aggregation should be applied to construct data cubes
 Dimension Reduction – irrelevant, weakly relevant or redundant attributes or dimensions
may be detected and removed
 Data Compression – encoding mechanisms are used to reduce the data set size
 Numerosity Reduction – data is replaced or estimated by using a smaller data
representation
 Discretization and concept hierarchy generation – data values for attributes are replaced
by ranges or higher conceptual levels.
Golden Rule – Reduction Time > Saved Time
No Reduction
Data Reduction
Data Cube Aggregation

Climbing up the upper level of concept hierarchy... OLAP
facility to summarize data
2008
2009
2010
Quarter
Sales
Sales
Sales
Q1
$224,000
$250,000
&249,000
Q2
$408,000
Q3
Q4
$350,000
$586,000
Year
Sales
2008
$1,586,000
2009
$2,345,677
2010
$3,594,000
Data Reduction
Dimension Reduction



Reduce the irrelevant or redundant attributes
Select the attribute subsets – attribute subset selection: find the minimum subset
of attributes to perform data mining action by not effecting the reliability and
robustness.
AWARE!!! All methods can only find local optimum... we just hope the local one is
also global optimum
METHODS:

Stepwise Forward Selection – start with empty set. Add one by one attributes. Stop if no more
information gained ...

Stepwise Backward Selection – start with full set of attribute. Eliminate one by one until
information gain changed significantly

Combination of Forward and Backward Selection – in each step algorithm selects the best attribute
and eliminates the worst attribute

Decision Tree Induction – When constructing a tree, algorithm starts with the best attribute and get
the second best and so on... Algorithm stops when there is not any significant information gain.
Data Reduction
Data Compression
Data encoding and transformations are applied to obtain a
reduced or compressed representation of the original data.
If the original data can be reconstructed from the compressed
version, the technique is called “lossless”
If only the approximation is gained after reconstructing, the
technique is called “lossy”
Two main techniques






Wavelet Transformation
Principal Component Analysis (PCA)
METHODS:

Principal Component Analysis – It searches the c (components) in the kdimensional orthogonal vectors that can be best represent the data where
c<=k


PCA can also be used as dimension reduction also
I t can not eliminate the attributes to form new attribute set. PCA construct totally
new attributes (components) that can explain the min %70 of all attributes.
Data Reduction
Numerosity Reduction



Gathering a smaller representation of original data. A way of
getting samples from original data.
AWARE!!! not to loose essence of data... Best representative
should be chosen.
Techniques


Regression & Log linear model – they can handle skewed data. They both are
sensitive to high dimensions (We will deal with them in clustering in detail)
Histograms: Use binning to approximate data distributions and are a popular
form of data reduction. A histogram for an attribute A partition the data
distribution of A into disjoint subsets or buckets The buckets are displayed on
horizontal axis, while the height (area) of a bucket typically reflects the average
frequency of the values represented by the bucket.
Data Reduction
Numerosity Reduction- Histograms


How are the buckets determined and the attribute values partitioned?
Partition Rules:
Equiwidth – the width of the bucket range is uniform.
Equidepth – the buckets are created so that, roughly, the frequency of
each bucket is constant (each bucket contains the sane number of
contiguous data samples)
V-optimal – Histogram with the least variance Histogram variance is a
weighted some of the original values that each bucket represents,
where bucket weight is equal to the number of values in the bucket. (if
data is one dimensional, V-optimal is K-means)
MaxDiff – The difference between each pair of adjacent values. A
bucket boundary is established between each pair for pairs having the
K-1 largest difference, where K is specified by user
Data Reduction
Numerosity Reduction- Histograms

Properties of Histograms






Highly effective at approximating both sparse and dense data
Effective at approximating skewed and uniform data
Histograms can be multidimensional
Multidimensional histograms can capture dependencies between attributes
Multidimensional histograms are good at handling data sets that have up to 5
dimensions.
They also are good to store outliers as well.
Data Reduction
Numerosity Reduction cont...
Other than histograms also the following used for numerosity
reduction
 Clustering
 Sampling




Simple Random Sampling
Simple Random Sampling with replacement
Cluster Sample
Stratified Sample
Data Reduction
Discretization and Concept Hierarchy Generation


Reduce the number of values for a given continues attribute
by dividing the range of the attribute into intervals.
Discretization and concept hierarchy generation for Numeric
Data





Binning
Histogram Analysis
Cluster Analysis
Entropy-Based Discretization – An info based measure called “entropy” can be
used to recursively partition the values of numeric attribute A, resulting in a
hierarchical discretization (we come back at decision trees)
Segmentation by natural partitioning – user defined partitioning
Data Reduction
Discretization and Concept Hierarchy Generation


Reduce the number of values for a given continues attribute
by dividing the range of the attribute into intervals.
Discretization and concept hierarchy generation for Numeric
Data






Binning
Histogram Analysis
Cluster Analysis
Entropy-Based Discretization – An info based measure called “entropy” can be
used to recursively partition the values of numeric attribute A, resulting in a
hierarchical discretization (we come back at decision trees)
Segmentation by natural partitioning – user defined partitioning
For categorical data


Basically user defined concept hierarchies and discretization
e.x. Geographical location, job category, colours and etc
Data Reduction
Discretization and Concept Hierarchy Generation cont...


For categorical data
Basically user defined concept hierarchies and discretization

e.x. Geographical location, job category, colours and etc
Thank You !!!
Q&A