Transcript This is the main title which is set in blue

What’s in your customer’s wallet?
C. Perlich, R. Lawrence, S. Rosset,
I. Khabibrakhmanov, S. Mahatma, S. Weiss
Predictive Modeling Group
Mathematical Sciences Department
IBM T.J. Watson Research Center
Media 6 Degrees
© Copyright IBM Corporation 2010
Predictive Modeling Group – Mathematical Sciences – IBM Research
Sales Alignment Task
Slide 2
© Copyright IBM Corporation 2010
Predictive Modeling Group – Mathematical Sciences – IBM Research
Market Alignment Project (MAP)
Objective: Allocate resources so as to


Focus the marketing and sales efforts on customers with high growth potential
Reduce risk of exposure to limited set of customers by identifying new valuable
customers
Wallet/Opportunity Definitions
1.
2.
3.
The total spending by a customer in a particular set of products in a given time
Total IBM attainable spending of a customer
Realistic IBM-attainable spending
Slide 3
© Copyright IBM Corporation 2010
Predictive Modeling Group – Mathematical Sciences – IBM Research
We cannot directly observe the Opportunity
Company Revenue (D&B)
We observe this
in the data
Wallet/Opportunity
IBM Sales to
this Company
But we do not
observe this
How can we make this
a data mining
problem?
Slide 4
© Copyright IBM Corporation 2010
Predictive Modeling Group – Mathematical Sciences – IBM Research
We formulate the problem as Quantile Estimation
 Imagine 1,000 customers with identical customer features
 Consider the distribution of the IBM Sales to these customers:
Best
Customers
IBM Sales
Opportunity is
High Quantile
Slide 5
© Copyright IBM Corporation 2010
Predictive Modeling Group – Mathematical Sciences – IBM Research
How do we estimate the Quantile?
 Linear regression estimates the conditional mean by minimizing sum
of squared error
 Quantile regression estimates the conditional quantile p by
minimizing asymmetric loss function
 p  ( y  yˆ )
L p ( y, yˆ )  
(1  p )  ( yˆ  y )
if y  yˆ
if yˆ  y
y  observed IBM Sales
yˆ  predicted Revenue Opportunit y
 Lp is optimized in expectation for Quantile p
 Quantile loss is used for estimation and evaluation


Implemented several variants (Linear, Decision trees, …)
KDD 2007 Paper
Slide 6
© Copyright IBM Corporation 2010
Predictive Modeling Group – Mathematical Sciences – IBM Research
Linear Quantile Regression (Koenker)
9
8
Opportunity for C 2
IBMRevenue
Revenue
IBM
7
6
Opportunity for C
1
Opportunity for C 1
5
4
C2
3
C
C1
2
1
10
Slide 7
20
30
40
50
60
Company
Firm Sales Sales
70
80
© Copyright IBM Corporation 2010
Predictive Modeling Group – Mathematical Sciences – IBM Research
Explanatory features are extracted from multiple sources
Dun & Bradstreet
(D&B) Data
IBM Client
Transactions
Entity Matching
Feature Extraction
D&B Features






Industry
Revenue (Rank)
Employees
State
D&B Structure Code
…
IBM Transactional Features
 Prior-year revenue in
other product brands
 Long-term revenue in
other product brands
…
 Train model against current year revenue based on previous year
 Apply model by rolling forward to current year and predicting
future opportunity
Slide 8
© Copyright IBM Corporation 2010
Predictive Modeling
Group –aMathematical
– IBM Research
The MAP process
provides
unique Sciences
integration
of OR and expert
insight
MAP Workshops
IBM Sales Team Interviews
MAP Web Interface
Model
Estimates
Expert
Feedback
Modeled
Opportunity
MAP Models
Integrated
Data
Data Model
Validated
Opportunity
Realign Sales Resources
Slide 9
© Copyright IBM Corporation 2010
Predictive Modeling Group – Mathematical Sciences – IBM Research
MAP Validation and Expert Feedback
Expert-validated
Opportunity (log)
Validates Opportunity
Expert
20
Experts accept
opportunity (45%)
18
16
Increase (17%)
14
12
Experts change
opportunity (40%)
10
Decrease (23%)
8
6
4
2
0
0
2
4
6
8
10
12
14
16
18
20
Experts reduced
opportunity to 0
(15%)
MODEL_OPPTY
ModelkNN
Opportunity
(log)
Opportunity
Slide 10
© Copyright IBM Corporation 2010
Predictive
Modeling
Group
– Mathematicaland
Sciences
– IBM Research
In 2008 MAP
covered
50+
countries
~100%
of IBM revenue and
opportunity
g
2005
2006
2007
,
2008
 Resources shifted to high growth Markets and Accounts
 Shifted resources performed >10 pts better
Slide 11
© Copyright IBM Corporation 2010
Predictive Modeling Group – Mathematical Sciences – IBM Research
Scope and some of the tedious details
• 3 Million customers
• 20 Brands (Product categories)
• 4 Markets
• Annual model refresh
• The Quantile is chosen for each brand and market separately
based on market insights on IBM market share
• Whitespace model for customers with no prior IBM revenue are
build using the same methodology but only D&B features
• Entity matching between IBM customer records and D&B
hierarchy is HARD
• Evaluation remains somewhat subjective and we collect feedback
Slide 12
© Copyright IBM Corporation 2010
Validated Revenue Opportunity
Modeling
Group –segmentation
Mathematical Sciencesand
– IBM Research
MAP outputPredictive
drives
account
resource allocation
decisions
Invest
High
growth
potential
Opportunistic
Small
Accounts
Core Growth
Modest growth
potential
Sellers shifted
Resource implications
 Shift resources to Core
Growth and Invest Accounts
 Reduce resource overlap
 8,000 sellers shifted
(2006 – 2009 )
Core Optimize
Flat or declining
Prior Year Actual Revenue
Slide 13
© Copyright IBM Corporation 2010
Predictive Modeling Group – Mathematical Sciences – IBM Research
Validated Revenue Opportunity
MAP drove significant revenue impact in 2008
Invest
Core Growth
$53B of Revenue
3,000 sellers shifted (2008)
30,000 sellers
Opportunistic
Core Optimize
$9B of Revenue
Prior Year Actual Revenue
[3,000 Sellers] x [$2M Revenue / Seller] x [10% Performance
Improvement]
= $600M (2008 Revenue Impact)
Slide 14
© Copyright IBM Corporation 2010
Predictive Modeling Group – Mathematical Sciences – IBM Research
MAP Take away
 Interesting predictive modeling task that calls for an
unorthodox loss function
 Combination of data mining AND expert feedback
 Integration into the annual sales management cycle
 Significant effort on data collection and preparation
 Many additional analytical tools were build on top of
MAP
 Territory definition and assignment
 Quota assignment
 Substantial impact on the bottom line
© Copyright IBM Corporation 2010