Transcript A FRAMEWORK FOR ASSESSING THE POTENTIAL VALUE OF A

A TWO LEVEL MONTE CARLO APPROACH TO CALCULATING EXPECTED
VALUE OF SAMPLE INFORMATION (EVSI):- HOW TO VALUE A RESEARCH DESIGN.
Alan Brennan, , Jim Chilcott, Samer Kharroubi, Tony O’Hagan, University of Sheffield, England. [email protected]
5) With “perfect” information (i.e totally accurate estimates of each parameter) we would
always choose the optimal strategy. Overall expected value of perfect information is
10, 000
 NetBenefitOfOptimalStrategy  NetBenefitOfBaseline Strategy
i
Part 2 – Two Level Algorithm for EVSI for Parameters (one parameter version)
6) Simulate a data collection exercise for your parameter of interest by:
• sampling the true underlying value 1 for parameter of interest
(e.g. the underlying true % response rate to a drug)
once from its prior range of uncertainty,
• then deciding on a sample size (n) for the parameter of interest, and then
• sampling a mean value for the simulated data obtained on the parameter of interest
(e.g. the response rate found in a trial of n=100, given 1).
(1st level)
7) combine the prior knowledge with the simulated data for each parameter using
Bayes formula to update the probability distribution for the parameter of interest
and produce a simulated posterior distribution for the parameter of interest
This has a revised mean and a smaller variance than the ‘prior’ because new
information has been obtained.
8) simulate the new combined data set allowing the parameters of interest to vary
across the uncertain range defined by their simulated posterior and the other
remaining unknown parameters to vary according to prior uncertainty (2nd level)
9) calculate the best strategy given this new knowledge on the parameter of interest
i.e. strategy which gives the most net benefit over all the simulations in (8)
10) calculate the mean net benefit of this new best strategy
11) Repeat steps 6 to 10, say 1,000 times, and calculate the average of the net benefits
obtained in (10)
12) The average value of gaining the specified sample information on the parameter of
interest is then
d

Step 7: Expected Value of Sample Information on i
(6) – (1)
Partial EVSI =
E X max E NB(d,  ) |  i  max E NB(d,  )

d

i

This same algorithm can be used for sets of parameters, each parameter can have a different
sample size, giving the ability to reflect complex research designs.
(6)
Total Cost
£
10%
0.1000
0.5
10%
0.0500
1.0
88%
0.3119
3.1
70%
0.2120 4.1
-18%
0.0998
1.0
25%
-5%
%
Side effects 20%
-0.10
0.00
Change in utility -0.10
if side effect
0.50
Duration of side 0.50
effect (years) -
10%
0.02
0.20
5%
0.02
0.20
31%
-0.06
0.70
24%
-0.14
0.65 -
-7%
-0.08
0.05
1,208
£
Total Cost
1,695
0.6175
Total QALY
0.7100
£
0.0925
1,956
£ QALY 2,388
Cost per
£5,267
(7)
0= prior uncertainty in the mean (standard deviation)
Cost per QALY
£
Net Benefit of T1 versus
T0
Net Benefit of T1 versus
T0 £ 5,405 £437.80
£ 4,967
= precision of the prior mean (B1)
X = sample mean (further data collection from more patients / clinical trial study entrants).
= precision of the sample mean (B2).
= sample variance
= implied posterior mean (the Bayesian update of the mean
 0 0   s 
following the sample information)
0   s
2


 /n  2
 12   2 pop 2
 = implied posterior standard deviation (the Bayesian update of
    / n  0 the mean following the sample information)
.
pop
 0

1 
£230
£223
£0
-1.4
-1.2
-1
-0.8
-0.6
-0.4
-0.2
£
629
-0.2446
0
0.2
0.4
0.6
0.8
1
1.2
1.4
£
1,558
1.6
£
3,256
-£2,570
£4,011
-£3,075
-£1,000
-£1,500
-£2,000
£7,086
Inc QALY
Option
£172
£150
£50
£10
25
2 level EVPI
All (trial, util + durations)
Durations
Trial + Utility
Utility only
Trial only
£194
50
100
200
Results 2: Sets of Parameters
(Proposed research designs)
If the sample size is very large, the numerator of the adjustment term tends to zero, the
denominator tends to the prior variance and so, posterior variance tends towards zero.
Conclusions
£
£
£
£
£
EVSI n = 50
1,351
887
916
768
359
£
£
£
£
£
1,319
867
878
716
330
EVI for Parameters: research options
P erfect
If the sample size is very small then ,in formula 5, the adjustment term will almost be equal to
1 and posterior variance is almost identical to the prior variance.
Convert to normal variables i.e. exp(lognormal),
then do Bayesian update as in Normal (Case 1 above),
then convert back to lognormal
1,937
0.5948
EVI per patient
£222
£215
£200
a) Trial Only – measure response rates T0,T1
Trial of n = 50 has EVSI = £330
b) Utility – measure utility improvement for
responders to T0, T1
Utility study of n = 50 has EVSI = £716
Note implied posterior variance will always be smaller than the prior variance because the
denominator of the adjustment term is always larger than the numerator.
Suppose prior for mean number of side effects per person is ~ Gamma (a,b)
If we obtain a further n samples, (y1, y2, … yn) from a Poisson distribution then
Posterior for mean number of side effects per person ~ Gamma (a+ yi, b+n)
£
E V SI fo r % res p o n d er t o T 0
£250
£100
How to Do it: Case 3 = Gamma / Poisson
(e.g. no. of side effects a patient experiences in a year )
1,308
0.8394
£1,000
Results 2 (cont’d)
c) Long term observational data on the
duration of effective response
Observational data on n=50 has EVSI = £867
d) Collect data on all variables
Data on all variables n=50 has EVSI = £1319
Result 1: Individual Variables
EVSI can be done for individual variables
e.g. response rate to drug T0. The value of the
knowledge increases as sample size increases
but reaches a plateau at the perfect information
ceiling.
2pop = patient level uncertainty from a sample ( needed for Bayesian update formula)
Suppose prior for % of responders is ~ Beta (a,b)
If we obtain a further n cases, of which y are successful responders then
Posterior ~ Beta (a+y,b+n-y)
£
£1,500
£500
d
How to Do it: Case 2 = Beta / Binomial (e.g. % responders)
487
-£500
0= prior mean for the parameter
I s  1 /  2
 2   2pop / n
Increment
T1 (T1 over T0)
1,499 £
499
11%
0%
7.05
2.05
589 £
-
T0
1,000 £
10%
5.01
589 £
£2,000
Total QALY
How to Do it: Case 1 = Normal
I 0  1 /  02
Sampled Values
T1
1 £
2%
1.00
200 £
10%
-
This is a 2 level simulation due to 2 expectations
How to Do it: Case 4 = Lognormal (e.g. cost of hospitlaisation)
EVSI (n)= average net benefit given sample information on the parameter (11) average net benefit given no further information i.e. NB of baseline decision (3)

T0
1
2%
1.00
200
£1,400
£1,200
£1,000
£800
£600
£400
£200
£EVSI n=50
Trial
4) For each sample set of parameters, work out the optimal strategy given that particular
“possible future world”, record the net benefit

Increment
T0
T1 (T1 over T0)
Mean Model
1,000
500
Cost of£drug 1,500 £
10%
8%
-2%
%
admissions
5.20
0.90
Days in Hospital6.10
400 per
£ Day
400 £
Cost
70%
%
Responding 80%
0.3000
Utility change0.3000
if respond
3.0
3.0 (years)
Duration
of response
% Side effects
Change in utility if side effect
Duration of side effect (years)
e
Utility only
3) Work out the baseline decision – i.e. strategy giving the highest net benefit
(on average over the 1,000 simulations).
i 1
i = the parameters of interest for partial EVPI
 -i = the other parameters (those not of interest, i.e. remaining uncertainty)
£
% Responding
Utility change if respond
Duration of response (years)
d
Uncertainty in Parameter Means
Standard Deviations
Durations
2) Simulate say 1,000 sample sets of uncertain parameter values (monte carlo).
(1)
£
b
c
Illustrative
Model
Trial + Utility
1) Characterise uncertain parameters with probability distributions e.g. normal(μ,σ2),
beta(a,b), gamma (a,b), triangular(a,b,c) … etc (prior knowledge)
Cost of drug
% admissions
Days in Hospital
Cost per Day
d
E X max E i NB(d, ) |  i
a
1,000
Number of Patients in the UK
£10,000
Threshold cost per QALY
Inc Cost
0) Set up a decision model comparing different strategies and set up a decision rule e.g.
“select strategy if marginal cost per QALY is < $50,000”. This is equivalent to “chose
strategy with highest net benefit” where, Net benefit = $50,000 * QALY – Cost
Illustrative Model
Number of Patients in the UK
Threshold cost per QALY
Mean Model
Step 1: no further information (the value of the baseline decision)
Given current information chose decision giving maximum expected net benefit.
Expected net benefit (no further info) =
max E NB(d,  )
Step 6: Sample Information on i
Expected Net benefit, sample on i =
The Algorithm: Part 1 – baseline decision and EVPI
EVPI  PopulationToBenefit 
Mathematical Formulation: EVSI for Parameters

= the parameters for the model (uncertain currently).
d
= set of possible decisions or strategies.
NB(d, ) = the net benefit for decision d, and parameters 
All (trial, util +
durations)
Purpose: describe a generalised two level Monte Carlo simulation for
calculating partial EVSI for parameters in a health economic decision model.
Background: Claxton has promoted the EVSI concept as a measure of the
societal value of research designs to help identify optimal sample sizes for
primary data collection.1,2 The method (using unit normal loss integral formula to calculate
EVSI for sample size n), only applies if the net benefit function is normally distributed, and if
the proposed sample exercise measures all the model parameters.
•Such conditions are not universal or necessarily common.
•A generalised method is needed for partial EVSI.
.
EVSI n=50
2 level EVPI
1) The mathematical expression for partial EVSI shows the 2 level monte carlo simulation needed.
2) Given sample data, Bayesian updating requires an analytic expression to revise parameters of the
distribution (given here for normal, beta, gamma and lognormal distributions).
3) More general distributions may require other approximations or MCMC methods.
4) The algorithm applies equally for sets of parameters as for a single one and provides a significant
step towards a generalised method for partial EVSI
6) Further work is needed on computational issues- how many samples? Non-linear approximations?
- see Matt Stevenson, Jeremy Oakley posters
1Claxton and Posenettand Posnett J. "An economic approach to clinical trial design and research priority
setting", Health Economics, 1996(5); 6: 513-524.
2 Claxton et al. Bayesian Value of Information Analysis: An application to a Policy Model of Alzheimer's Disease,
International Journal of Technology Assessment in Health Care, Volume 17 Issue 1, 2001
Acknowledgements:
Co-authors: Jim, Samer and Tony (for help in thinking, working through maths and
encouragement to get it done). I am a member of CHEBS: The Centre for Bayesian
Statistics in Health Economics. Particular thanks to Karl Claxton and Tony Ades who
helped finalise our thinking at a CHEBS “focus fortnight” event in April 2002.