Transcript Value at Risk: Where did it all go wrong?

Value at Risk: Where did it all go wrong?
Andrew D Smith
Yorkshire Actuarial Society. 09 May 2012.
© 2010 The Actuarial Profession  www.actuaries.org.uk
What is Value-at-Risk?
• Jorion (2007): “The worst loss over a target horizon
• Such that there is a low, pre-specified probability that the
actual loss will be larger”
• Legislative references: Insurance Solvency II Directive:
The Solvency Capital Requirement ... shall correspond to the Value-atRisk of the basic own funds of an insurance or reinsurance undertaking
subject to a confidence level of 99.5 % over a one-year period.
© 2010 The Actuarial Profession  www.actuaries.org.uk
1
Own funds’ probability
distribution in one year (ignoring
shareholder dividends or capital
raising)
mean own funds
Expected profit
Opening own funds
(t=0)
VaR
Basic Own Funds = Assets minus Technical Provisions
Calculating VaR Using Percentiles
0.5%-ile own funds
0.5% probability (red region)
t=0
“now”
© 2010 The Actuarial Profession  www.actuaries.org.uk
t=1
”VaR horizon”
2
Daily Value at Risk example: Barclays
Source: http://www.barclaysannualreports.com/
© 2010 The Actuarial Profession  www.actuaries.org.uk
3
Examples of Extreme Losses & Percentiles
(calculations based on Normal distributions)
Number of Standard Deviations
0
Fortis
AIG
5
€17.6bn
99.97%
10
€28.0bn
99.9999976%
15
20
VaR 31/12/2007
Loss 2008
$99.3bn
$19.5bn
99.999999999999999999999999999999
99.95%
99999999999999999999999999999974%
© 2010 The Actuarial Profession  www.actuaries.org.uk
4
Calculating VaR: A Worked Example
© 2010 The Actuarial
© 2010 The
Profession
Actuarial Profession
www.actuaries.org.uk
 www.actuaries.org.uk
5
VaR Calculations: Our Abstraction
Our thought experiment
• 100 years of clean annual
data
• Taken from a stationary
process
• Data is historic company
profits
• No evidence available
besides 100 data points
© 2010 The Actuarial Profession  www.actuaries.org.uk
More realism
• Between 10 and 100 years of
data, maybe high frequency
• Early data of questionable
accuracy and relevance
• Data describes “risk drivers”
(eg equity levels, interest
rates, credit defaults,
mortality assumptions etc)
• Profit is a complicated
function of risk drivers.
• Relevant knowledge from
related problems
6
Based on Excel Example
• Examine historic profit: annual and cumulative
• Method #1 (through the cycle)
– Fit a distribution to observed profits (normal, Gumbel)
– Check distribution fit
• Method #2 (pro-cyclical)
– Regress Profit(t) against Profit(t-1)
– Fit distribution of residual
– Estimate percentile profit as regression formula + percentile
residual
• Method #3 (conditional)
– Examine situations leading to largest residuals
– Estimate percentiles for Profit(t) conditional on Profit(t-1)
© 2010 The Actuarial Profession  www.actuaries.org.uk
7
Perfect knowledge: What if we knew the “true” model?
© 2010 The Actuarial
© 2010 The
Profession
Actuarial Profession
www.actuaries.org.uk
 www.actuaries.org.uk
8
Longitudinal and Cross-Sectional Analysis
Inside a Scenario File
Scenario #
Time period
Cross-sectional
Longitudinal
History
Same for each scenario
© 2010 The Actuarial Profession  www.actuaries.org.uk
Forecast
Varies by Scenario
9
How I Generated the Data
• Warning: this is an example of a “non-guessable” model
– Unless you have huge volumes of data
• I do not claim your profits follow such a process!
• Let T be the end of the projection
• Use the backwards autoregressive (order 1) model:
X T   ln  ln( U )
X t  ln  ln( U1 )1  cos U 2 
X t 1 
;t  T
2
• U, U1, U2 independent uniform(0,1), Excel Rand() function.
• Stationary distribution = Gumbel (proof next slide)
© 2010 The Actuarial Profession  www.actuaries.org.uk
10
The Maths
Joint CDF of Profit(t-1) and Profit(t)
F ( x, y )  Prob Profit (t  1)  x; Profit (t )  y

 2 exp  e  y

 exp  e
x
 e x y / 2 

 
2 



e x y / 2 
y/2

  2e

2 



e x y / 2 
y/2

  2e

2 

 
 exp e
x
• Can read off
– Stationary distribution (Gumbel),
– Conditional distribution of Y given X (Wald)
– Conditional distribution of X given Y (AR1 process on
previous slide)
© 2010 The Actuarial Profession  www.actuaries.org.uk
11
How I Generated the Numbers
Markov Process Conditional Percentiles
99.95%
Profit(t)
15
99.5%
10
95%
5
50%
0
-5
0
5
5%
15 0.5%
0.05%
10
Profit(t-1)
-5
© 2010 The Actuarial Profession  www.actuaries.org.uk
12
Normal cdf: μ = Euler’s constant; σ = π/√6
PP – Plot: Non-normality detectable for large
data sets so model risk reduces
1
0.8
0.6
0.4
normal-Gumbel
y=x
0.2
0
0
© 2010 The Actuarial Profession  www.actuaries.org.uk
0.2
0.4
0.6
0.8
Gumbel cdf
1
13
Validation and Back-Testing VaR
© 2010 The Actuarial
© 2010 The
Profession
Actuarial Profession
www.actuaries.org.uk
 www.actuaries.org.uk
14
Longitudinal Validation under Basel
How do you know your model is right?
• Bank regulation:10 day VaR at 99% Confidence
– Look back over last year (250 trading days, overlapping periods
each looking 10 days back) in which both VaR and profit are
updated
Amber zone
Green zone
0
5
unbiased (2.5 = 250 * 1%)
Red zone
10
15
Number of exceptions in a year
• What does this process test?
– The “back test” includes implicit tests of model and parameter error
as well as outcomes
– Although it won’t test risks that didn’t materialise in the last year
© 2010 The Actuarial Profession  www.actuaries.org.uk
15
Which of the Three VaR Methods is right?
• Method #1 (through the cycle aka unconditional aka
stationary aka longitudinal)
– Fit a distribution to observed profits (normal, Gumbel)
• Method #2 (pro-cyclical)
– Regress Profit(t) against Profit(t-1)
– Fit distribution of residual
– Estimate percentile profit as regression formula +
percentile residual
• Method #3 (conditional aka point in time aka crosssectional)
– Estimate percentiles of P(t) conditional on Profit(t-1)
Can you Defend your Model?
16
Monte Carlo Back-test validity
and Pro-cyclicality
Profit(t)
10
Each VaR curve separates
0.5% of the dots from the
other 99.5%
5
Profit(t-1)
0
-5
0
5
Conditional VaR
Stationary Var
-5
© 2010 The Actuarial Profession  www.actuaries.org.uk
17
Mean estimated percentile over the cycle
Average Percentile Estimates over the Cycle
Same exception rate / different mean VaR
10
5
99.95%
99.5%
95%
median
5%
0.5%
0.05%
0
-5
Unconditional
© 2010 The Actuarial Profession  www.actuaries.org.uk
Linear
Conditional
18
What is the best VaR Methodology?
Stakeholder
Example of possible concerns
Policyholder
Benefit security
Cost of insurance cover
Corporate manager
Reported return on capital
Management flexibility
Regulator
Market confidence
Financial stability
Shareholder
Share price growth
Dividends
General public
Amplitude of economic cycle
Bail-outs
Actuaries?
© 2010 The Actuarial Profession  www.actuaries.org.uk
19
Sources of Error in VaR Calculations
©Can
2010you
TheDefend
Actuarial
Profession
your
Model?  www.actuaries.org.uk
20
Potential sources of error in VaR Calculations
(the well-known examples)
Category
Example
Random
Draw from an experiment
whose distribution is not in
dispute. Textbook examples:
coin toss, drawing coloured
balls from an urn.
Parameter
error
Estimation of parameters
from finite samples
Portfolio optimisation finds
strategies where returns are
over-stated or risks understated
Model error
Chosen mathematical model
family does not contain the
process that generated the
data
Complexity bias (eg use normal
distribution instead of fat tails,
linear AR1 instead of non-linear
heterosecastic, dimension
reduction, commercial
pressure)
© 2010 The Actuarial Profession  www.actuaries.org.uk
Bias
21
Monte Carlo Calibration Test
Robust Testing of VaR Across Multiple Distributions
Fitted %-iles
Sim #1
Model #1
Sim #2
Sim #200
Model #2
Model #50
10 000 runs in total
Validator prepares
8 fitted percentiles
(eg 0.5%, 1%, 5%, 10%,
90%, 95%,99%, 99.5%)
10 000 times
Production team prepares
Yr26
Test Yr26 outcome against percentiles
Years 1-25
22
Less-discussed sources of error
Did these contribute to AIG/Fortis Exceptions?
Category
Example
Bias
Cyclical (point
in time
estimates)
Mis-identification of
hidden state variables,
excluding “irrelevant”
historic periods
Symmetric dampeners, judgements about
underlying investment value and correction
of distorted or illiquid markets
Data
Incomplete or inaccurate
Falsification or selective submission of data.
Underwriting bias such as winners curse.
Exaggerate benefit of lessons learns or
effectiveness of recently imposed controls.
Exposure
(proxy model)
Mis-statement of asset
and liability sensitivity to
combined moves in risk
drivers
Constructing hedges to minimise stated
VaR; devising “easy” stress test that are
known to pass. Lack of preparation for outof-test stresses.
Computation
Roundoff in floating point
calculations; differential
equation discretisation,
simulation sampling error
Debug effort focuses on commercially
unacceptable output.
© 2010 The Actuarial Profession  www.actuaries.org.uk
23
What does an auditor look for?
A “True and Fair View”
• Error tolerance expressed in terms of “materiality” limits.
• Checking controls and calculations, and identifying any bungles.
– Compare cumulative effect to materiality limits
• Are key management judgements reasonable.?
– Comparable to statistical quality standards
– Compare to common practice in the market.
– Range for reasonable judgements may be narrower than parameter
standard errors
– Board retains responsibility for judgements
• The scope of materiality limits is limited to human bungles
– Such as omitting a class of liabilities or calculating tax incorrectly.
– Differences in judgement do not contribute to the materiality limit.
• Therefore, it is entirely possible for two reasonable judgements to
produce results differing by many multiples of the stated materiality
limit.
Can you Defend your Model?
24
Does it Matter if the Model is Wrong?
© 2010 The Actuarial
© 2010 The
Profession
Actuarial Profession
www.actuaries.org.uk
 www.actuaries.org.uk
25
Robust Statistics
• Some firms’ VaR calculations rely on knowing the “right”
model, but can you ever attain that knowledge?
• Many models could describe the data and not be rejected
by statistical tests
• Robustness dictates that the methodology should work
(produce at most 0.5% exceptions) for a range of possible
models
• For example, through-the-cycle methods rely only on
stationarity; our spreadsheet example showed how these
tools can still work even if the “true” model is unknown.
© 2010 The Actuarial Profession 
www.actuaries.org.uk
26
Conclusions
• Many approaches to VaR can produce the right exception
frequency over the cycle. These differ in
– Average required capital over the cycle
– Pro-cyclicalility
– Robustness to mis-specification
• Errors in models, parameters, point-in-time, data, exposure and
computations have contributed to excess exceptions
• Addressing biases requires detailed business process
knowledge, not only statistics.
• Different stakeholders may not agree on what makes a “good”
VaR calculation.
© 2010 The Actuarial Profession  www.actuaries.org.uk
27
Questions or comments?
Further questions?
My contact details:
Andrew D Smith
Partner, Deloitte
Hill House, 1 Little New Street
London EC4A 3TR
[email protected]
Disclaimer
Any views or opinions in this
presentation are those of the author
alone and not of his employer or any
other body with which he associates.
© 2010 The Actuarial Profession  www.actuaries.org.uk
28