Transcript External validity

External Validity of Trials
Background


External or ecological validity refers to
whether the results of the trial can be
generalised to the general clinical
population.
Randomisation per se does NOT make a
study externally valid.
Validity Issues



Are the ‘correct’ patients being included
in the trials?
Are the ‘correct’ practitioners taking
part in the study?
Are the ‘correct’ facilities being used?
Trial Problems


Many if not most trials are undertaken
in circumstances that are far removed
from ‘usual’ clinical practice.
Usually clinicians running trials are
content experts yet the results are to be
applied by non-experts.
Osteoporosis Trials


Most of the large osteoporosis trials
with fracture as an outcome recruited
participants from ‘bone’ clinics.
Patients under care of clinical experts
but results are expected to be applied
by clinicians who are not experts.
Participants



Usually participants who go into trials
are different from those who do not.
These ‘storm troopers’ of patients
usually comply with instructions better
than average.
Combined with different clinicians and
patients results can be different from
the ‘real’ world.
Trial participants v non
% (n) with
Took part in
trial
n = 1 409
Did not take
part in trial,
n = 1 814
1 risk factor
42.4 (597)
62.9 (1 141)
.000
2 risk factors
40.9 (576)
29.4 (533)
.000
3 risk factors
13 (183)
6.9 (126)
.000
4 risk factors
3.8 (53)
0.8 (14)
.000
Crude risk of fracture
Type of
fracture
Took part in
trial
n = 1 409
Did not take
part in trial,
n = 1 814
P value
Overall
6.5%
10.4%
.000
Hip
1.4%
1.8%
.405
Wrist
2.5%
3.8%
.044
Risk of fracture (adjusted)
Type of
fracture
OR
(adjusted)
95% CI
lower
upper
P value
Overall
0.42
0.31
0.55
.000
Hip
0.50
0.28
0.90
.021
Wrist
0.52
0.34
0.79
.002
Alendronate


Alendronate is a bisphosphonate for
osteoporosis treatment. Large trials
conducted in expert clinical centres
showed it reduced fractures with few or
no side-effects.
Things were different when it was
licensed and ‘real’ clinicians used the
drug.
Alendronate problems



To take a bisphosphonate one needs to
take it on an empty stomach with a
large glass of water and remain upright
for at least 2 hours.
Otherwise the drug can get affect the
oesophagus and cause ulceration.
In real life this is what happened and
led to a warning. Trial data gave to
evidence this was a problem.
The Wrong Solution


Often once the phase III trial has been
finished companies finance a ‘real’ world
non-randomised study.
This is usually nothing more than a
marketing exercise and produces worthless
data on ‘real’ world experience of the
problem.
A correct solution



Involves some form of randomisation to
eliminate confounding.
For example one might undertake a
cluster trial of giving the drug nonspecialist physicians to use of their
patients BEFORE the drug is widely
available.
This will produce more robust data.
Alternatively



We should try and design our trials to
be as pragmatic as possible so that the
results are widely generalisable.
This means recruiting non-expert
clinicians and their patients.
Using a pragmatic design rather than an
explanatory design.
Back Pain Trials


An early RCT of chiropractic
manipulation for back pain was
criticised by physiotherapists that the
chiropractic was undertaken in private
practice.
May have been a ‘Bach’ effect.
MRC back pain trial


To address the issue of nice premises
the MRC back pain trial included an arm
where patients were randomised to be
treated on private premises or NHS
premises.
Increased the costs of the trial hugely.
Surgical Trials


Often surgical trials have poor external
validity because the new technique is
usually developed in a teaching hospital
among ‘expert’ surgeons (e.g.
laparoscopic methods).
It is unlikely the results of surgery
undertaken in ‘top’ hospitals would be
the same as a ‘bog-standard’ DGH.
Big and Simple


Try and make the trial large, as it allows
for various PRE-SPECIFIED sub-group
analyses (specialists vs generalists).
Simple which means one can get as
many in as possible.
Summary


Most trials will NEVER be able to recruit
exactly the same types of participants
as those who receive treatment in
clinical practice.
PRAGMATIC trials have the strongest
external validity and usually one of
these should be done before
implementation.