Transcript HIV Genotyping Assay Validation

HIV Drug Resistance Training
Module 7:
HIV Genotyping Assay
Validation
1
Topics



Getting Ready to Validate
Validation Concepts
Validation Procedures
2
Objectives
Describe principles behind validation procedures.
Identify the procedures needed to gain
confidence that the results are accurate and
reproducible.
 Given specific laboratory, customize procedures.


3
getting ready to validate
Why is it important to validate assays?
What should you do to prepare for validation?
When do you need to validate?
4
Validation of Genotyping Assay
Performance
Helps ensure quality results
Easier to compare results from labs using
different methods
 To establish performance characteristics used to
compare assays
 Required prior to testing any samples for WHO
 Validation is assay-specific


– For instance, plasma vs. DBS sensitivity could be
different, and extractions steps are not shared, so
these methods should be validated separately.
5
Minimum Requirements

Minimal requirements for an assay validation to
establish/confirm analytical performance
characteristics:
–
–
–
–
–
Sensitivity
Specificity
Accuracy
Precision
Reproducibility
6
Preparation: Procedures

Standard Operating Procedures (SOPs)
– For what?
– How?

Quality control (QC) SOPs
– For what?

Personnel?
7
Preparation: Validation Protocol

Written validation protocol finalized before results
are analyzed
– No changes to protocol allowed once validation
experiments started
Determine acceptance criteria for each section in
advance
 Validation performed under the same conditions
(facility, equipment, reagents, personnel) that will
be used for real samples

8
When to Repeat Validation Procedures

If there is failure to pass any criteria:
– Document corrective action and re-validate
– Do NOT simply repeat experiments with same assay

If procedure changes:
– Minor changes  equivalency testing
– Major changes  re-validation for in-house assays:
• Primer modifications and replacement
• Change RT and/or PCR enzymes
– See module 11, QA/QC
9
Discussion



Why is it important to validate assays?
What should you do to prepare for validation?
When do you need to validate?
10
validation concepts
What criteria are we measuring against?
11
Components of Genotyping Assay Validation





Accuracy
Precision/Reproducibility
Sensitivity for RT-PCR amplification
Sensitivity for detection of minority variants
Specificity
12
Accuracy Definition

Are “known” mutations detected?
13
Accuracy: How close do we get to the
expected (target) value or result.
Accurate
Inaccurate
14
Accuracy Discussion

Best tested using:
– clonal virus or lab constructs (e.g. site-directed
mutants)
– very well-characterized, non-clonal sample (e.g.
artificial 50/50 mixtures of clones)

Test for:
– “Resistance-associated” mutations
– Likely to be frequently encountered


Pro: Usually the easy to measure
Con: Difficult to define criteria ahead of time for
very new assays
15
Precision Definition

Ability to generate the same result on multiple
aliquots of the same sample within a test run
(intra-assay variability)
– Fewer samples (e.g. 3 to 5), more replicates (e.g. 5 to
10)
16
Precision: How close are all the results to
each other?
Precise
Imprecise
(although inaccurate!)
(although sometimes accurate)
17
Accuracy and Precision
The degree of fluctuation in the measurements is
indicative of the “precision” of the assay.
 The closeness of measurements to the true value
is indicative of the “accuracy” of the assay.
 Quality Control is used to monitor both the
precision and the accuracy of the assay in order
to provide reliable results.

18
Accuracy and Precision
Precise: Yes
Accurate: No
Precise: Yes
Accurate: Yes
Precise: No
Accurate: No
19
Reproducibility

Ability to generate the same result on multiple
aliquots of the same sample in different test runs
(inter-assay variability)
–
–
–
–
More samples (e.g. up to 40), fewer replicates (2 or 3)
Over time (e.g. 2 weeks)
Between operators
Among lots of critical reagents
20
Reproducibility: When conditions change,
how does that affect the results?
21
Reproducibility and Precision Criteria

Based on nucleotide sequence identity in:
– Pairwise comparisons or
– Comparison to a consensus sequence

Mismatches may be considered a difference (if
“compatible” e.g. A and R, or R and D, but not A
and Y, or M and K)
– Acceptance criteria may depend on complexity of
samples tested

Based on amino acid sequence identity
– “resistance-associated” mutations can be analyzed
separately
22
Critical Reagent Lot-to-lot Variability is a
Component of Assay Reproducibility
Reagent vendors test products (e.g. PCR
enzymes) using procedures different from HIVDR
genotyping
 Cannot assume that performance is the same,
between lots
 Incorporate at least 2 lots of critical reagents in
validation experiments
 Perform lot-release testing as part of routine QC
(see module 11)

23
Sensitivity: Amplification
Definition: What is the minimum viral load
required to generate an accurate result?
 Considerations:

– Importance placed on multiple subtypes varies with
intended application
– Overlaps with linearity if samples also genotyped and
sequences compared
– May or may not involve probing multiple variant
detection (mixtures) at low VL
24
Sensitivity: Detection of Minority Variants


In samples with mixtures
What % of the pool must minor variant be in
order to be reliably detected?
– Can separate out post-amplification steps (e.g. by
mixing plasmids and sequencing vs. mixing viruses and
amplifying then sequencing)
– In reality, affected by viral load, though this is not
often highlighted
25
Dependence of Sensitivity of Detection of Minor
Variants and Input Viral Load

Assume that 200 µl plasma used for RNA
extraction, 25% used for RT-PCR; RT successful
for 20% of RNA molecules; minority variant
present at 10% of total
Viral load
(copies/ml)
RNA copies in
RT rxn
Amplifiable
genomes
Copy no. (minor
variant)
100,000
5,000
1000
100
10,000
500
100
10
1000
50
10
1
26
Discussion

What criteria are we measuring against?
27
validation procedures
How can we set up validation procedures for our lab?
28
Validation Criteria (Example)

Accuracy
– 100% of known mutations must be detected

Precision/Reproducibility
– ≥ 90% of pairwise comparisons must be at least 98%
identical, mixtures counted as difference

Sensitivity for amplification
– ≥ 95% of samples with viral loads between 500 and
1000 copies/ml must be amplified and successfully
genotyped (n ≥ 10)

Sensitivity for detection of minority variants
– defined as the lowest % at which a mutation is
detected in > 50% of replicate tests
29
Specimen Selection
Specimens used in the validation experiments
should be chosen to test the specific aspect of
assay performance (accuracy, reproducibility,
etc.)
 Especially for precision and reproducibility, chose
specimens that are similar to those that will be
tested routinely (i.e. specimen type, genetic
subtype, viral load range)
 Specimens should be well-characterized in prevalidation experiments before starting the actual
validation

30
Validation Procedure Customization

Details of the variables to be considered are likely
to be different in each lab
– Physical set-up
– Personnel (number of qualified analysts)
– Degree of automation

Validation protocols can be reviewed in
consultation with other accredited labs or with
WHO staff
31
Discussion

How can we set up validation procedures for our
lab?
32
Reflection

What do we need to do to make sure we are
correctly validating our assays?
33
Summary



Getting Ready to Validate
Validation Concepts
Validation Procedures
34