Transcript Taking Uncertainty Into Account: Bias Issues Arising from

Taking Uncertainty Into Account:
Bias Issues Arising from Uncertainty in Risk Models
John A. Major, ASA
Guy Carpenter & Company, Inc.
Example: Exponential Distribution



N=20 observations
T = sample mean; l=1 true mean
MLE EP curve:



ˆ ( x)  exp  x
Q

T
q-exceedance point (PML, VaR)
ˆ  T  ln( q)
 X
q

X.01 = 4.605 actual
Sampling Distribution of T
Estimated PDFs
Client Questions



What is the 1 in 100-yr PML (1% VaR)?
What is probability of exceeding 4.605?
Can you give me an EP curve to answer
these and similar questions?


Does sampling error affect the answer?
Can I get unbiased answers?
3 Kinds of Bias

 
“dollar” or X-bias: E Xˆ q vs X q


  
“probabilistic” or P-bias: E Q Xˆ q vs q


the average of PML dollar estimates
the average true exceedance probability of
estimated PML points
  
“exceedance” or Q-bias: E Qˆ X q vs q

the average estimated exceedance
probability
Exponential MLE is X-unbiased
ET   l
 
E Xˆ q  E T  ln( q)  l  ln( q)  X q
Exponential MLE is X-unbiased
Exponential MLE is P-biased



  
E Q Xˆ q  q for small q
Expected actual risk is greater than
nominal
Uncertainty increases risk!
Exponential MLE is P-biased
Correcting for P-bias

Predictive distribution


Mix randomness and uncertainty



“Prediction interval” in regression
integrate model pdf over parameter
distribution

Exponential model: Q( x)  exp  x
T
Predictive result:
n
x 

Q ( x )  1 

 T n 

Predictive vs. Model Density
Which to use?

MLE curve is X-unbiased



Predictive curve is P-unbiased



no uncertainty adjustment, but...
on average, gets right $ answer
“takes uncertainty into account” and...
on average, reflects true exceedance pr
But they disagree...

and it gets worse...
Exponential MLE is Q-biased





E Qˆ X q   q for small q
Expected estimated risk is greater than
the true risk (at the specified threshold)
Uncertainty now causes risk to be
overstated!
Exponential MLE is Q-biased
Correcting for Q-bias

Minimum Variance Unbiased Estimator


Rao-Blackwell Theorem



standard procedure in classical statistics
Expectation of unbiased estimator,
conditional on sufficient statistic

Exponential model: Q( x)  exp  x T
MVUE result:
n 1
x 

Q ( x )  1 

 T n 

MVUE vs. Model Density
Paradox



Say we get an estimated T=1 (correct)
MLE says X.01=4.605, Pr{X>4.605}=1%
Predictive: X.01=5.179 is p-unbiased


risk is greater than MLE answer because
impact of uncertainty
MVUE: Pr{X>4.605}=.69% is q-unbiased

risk is less because MLE tends to overstate
exceedance probability
How the Paradox Arises
Conclusions


Uncertainty induces bias in estimators
Biases operate in different directions


depends on the question being asked
There is no monolithic “fix” for taking
uncertainty into account


Predictive distribution fixes p-bias,
while making q-bias worse
Recommendations


First: Show modal estimates (MLE etc.)
Second: Show effect of uncertainty




Keep uncertainty distinct from randomness
Sensitivity testing w.r.t. parameters
Confidence intervals on estimators
Third: Adjust for bias only as necessary


Carefully attend to the question asked
Advise that bias adjustment is equivocal