Transcript Child and Adolescent Psychopathology

Chapter 2:
Strategies for Evidence-Based
Assessment of Children and
Adolescents
Erica A. Youngstrom
Thomas A. Frazier
“Three P’s” of Assessment
 Predict
 Does the assessment method predict important criteria?
 Prescribe
 Does the assessment method prescribe specific
treatments?
 Process
 Does the assessment method inform our understanding
of processes in developmental psychopathology?
Factors Contributing to
Stagnation Within the Field
 Lack of a clear linkage between measurement tools
and clinical practice
 Erosion of service reimbursement and the challenge to
demonstrate cost effectiveness of psychological
assessments
 Current assessment devices are selected primarily
based on convention rather than on any clear sense
that they are the best tools for specific purposes
 Nosological systems were initially developed for adults
and then just adapted for use with adolescents and
children
Prediction
 Which assessment tools should be used to
measure the development of psychopathology?
 How can users compare the different tests that are
available and make an informed choice about
them?
 Prediction can mean demonstrating concurrent
correlations, as well as showing associations with
criteria that are separated by time.
Concurrent Criterion Validity
 What does it mean?
 Correlating assessment tools with other meaningful criteria
 An iterative process involving elaboration and consolidation
 Elaboration  involves expanding the network of correlations to
include not just validators, but also correlates in terms of typical
treatment response, outcome, quality of life, and functioning
 Crucial to contextualizing the construct
 Consolidation  involves clarifying when different measures are
assessing the same underlying latent variables, as well as
creating models about relationships among these underlying
processes
Statistical Methods: Correlations
 Correlations
 A measure of effect size that can be compared across
studies and across measures within a field
 Has several drawbacks:
• As an index of prognosis, is difficult to apply to individual cases
• Can consider only one correlate at a time
• Can also be unduly influenced by extreme scores, especially when
you have a small sample size
 May be more useful to directly compare validity
correlations for different measures
Statistical Methods: Regression
 Several advantages over bivariate correlational analyses:
 Preserve the actual units of measurement in the unstandardized
regression weights
 Provide formal tests of whether combinations of predictors provide
incremental improvement over a single predictor
 Make predictions about an individual’s score on the criterion variable
 Create a framework where it is possible to test statistical mediation or
moderation of relationships
 Some limitations:
 Lack of sufficient published statistics
 Computational burden on the clinician
 Regression weights are dependent on the specific combination of
instruments used
 One possible solution:
 Publish more complete details of regression results
Additional Statistical Methods
 Raw Scores
 Working in raw units would indicate the actual scores that
would be predicted on the dependent variable, which
brings the focus back to how clinicians would actually use
the assessment
 Incremental Validity
 An additional variable provides a statistically significant
improvement in prediction of the criterion
 Comparing Observed Versus Predicted
Performance
 Most commonly used to compare cognitive ability and
academic function
Statistical Methods for
Consolidation of Scores
 Three methods for determining the number of
underlying dimensions underlying a battery
1. Partial Analysis
2. Scree Test
3. Minimum Average Partials
 All three methods converge on more parsimonious
factor structures, which has several advantages:
 Fewer scales to interpret  reduces Type I error risk
 Larger number of items per factor  greater internal
consistency, smaller standard errors of measurement,
and more accurate description of the individual
Grouping Individuals: “QMethods”
 Factor Analysis
 Aggregate people with similar profiles of scores across
variables, rather than the more widespread approach of
grouping variables with similar scores across people
 Cluster Analysis
 Latent Class Analysis
 Cluster cases on the basis of observed categorical
indicator variables
 Mixture Modeling
 Takes into account residual relationships among
indicators
Advantages of Q-Methods
 Allow the data to dictate how many core profiles
emerge, and what the relative prevalence is of
each profile
 Provide a parsimonious response to the problem of
comorbidity because most core profiles involve
elevations on multiple scales
 Multivariate profiles can incorporate information
from multiple informants
Event History Methods and
Individual Trajectories
 Event History Analysis
 Methods for looking at the length of time until a particular event
of interest happens
 Survival Analysis and Cox Regression
 Uniquely situated to evaluate onset, cessation, relapse, and
recovery
 Individual Trajectories
 Hierarchical Linear Models
• Repeated measures are “nested” within the individual participant
 Growth Curve Models
• Observed variables are treated as indicators of change for indirectly
observed latent variables
 Growth Mixture Models or Latent Class Growth Models
• Unique trajectories of indicators are empirically defined
Improving Predictive Value
of Assessments
 Publish studies that consolidate existing measures
into more parsimonious dimensions
 Conduct studies that elaborate the connections
between measures and constructs across
development
 Directly compare the predictive value of multiple
tests under the same conditions
 Supplement or supplant correlational analyses with
multivariate regression, reported in enough detail
to allow application to individual cases
Prescription
 Sometimes referred to as “treatment matching”
between a diagnosis and an intervention strategy
 Clinical diagnosis is “imperfectly reliable”
 We can never be 100% certain that any child has a given
disorder
 Once our estimate of the probability exceeds a certain
point, we can be confident to proceed with treatment
Primary, Secondary, and Tertiary
Intervention
 Primary Intervention
 “Universal prevention” that treats everyone, regardless of risk
 Most rational approach when treatment costs and risks are low, and
benefits clearly outweigh risk and cost
 Example: fluoridating drinking water to prevent dental problems
 Secondary Intervention
 Focus on those who are at risk of developing a condition, but have
not yet developed the condition
 Appropriate for cases where an individual falls between the
assessment and treatment thresholds
 Tertiary Intervention
 Most expensive and may also have the most serious risk of sideeffects
 Assessment question is not if there is a problem, but rather what the
specific nature of the problem is
Using Assessment to Aid in
Diagnosis
 Sensitivity: The percentage of cases with the diagnosis
that would be classified correctly by the test
 Specificity: Quantifies the percentage of cases without
the diagnosis that would be classified correctly by the
test
 Receiver Operating Characteristics (ROC) analysis
 Method for quantifying the relationship between the sensitivity
and specificity of an assessment tool for a particular diagnosis
 Sensitivity is plotted as a function of specificity
Nomogram
 Used to simplify the process of determining a posterior probability
(probability of a diagnosis combined with test results estimates a new
probability)
 Advantages
 Makes it possible to apply test results to a specific individual, directly estimating
the posterior probability
 Substantial improvements in the accuracy and precision of test interpretations
 Reducing the influence of cognitive heuristics on the interpretation of test results
 Flexibility in choice of starting point
 Elimination of computation
 Facilitation of discussion between families and practitioners
 Limitations
 Unfamiliar to most researchers
 Few researchers are publishing the likelihood ratios directly in their research
reports
Assessments as Aid in
Treatment Selection
 Absolute Risk Reduction (ARR)
 Subtracts the rate of a negative categorical event in the treatment
group from the rate in the comparison group
 Number Needed to Treat (NNT)
 1 divided by the ARR
 Number Needed to Harm (NNH)
 1 divided by the rate of adverse events in the group receiving the
treatment versus the rate of the same adverse events in the
comparison group
 Likelihood of Help Versus Harm (LHH)
 The ratio of the NNT and NNH, with both being adjusted for the
clinician’s judgment of the patient’s risk compared to the risk of the
average patient in studies providing the values of NNT and NNH
 Limitation of these parameters  all assume dichotomous
outcomes
Process
 Clinical Outcomes
 Quantifies the degree of change occurring as the result of
treatment
 Encompasses measurement of variables that are
informative about the mechanisms for growth and
change
 Assessment can also be used to detect variables
that will change the prognosis or response to
treatment
Treatment Outcome Evaluation
 Gives clinicians access to feedback about change
during the course of treatment, which in turn:
 Produces lower rates of dropout
 Leads to more efficient resource allocation across
caseloads
 Results in better outcomes
Measuring Treatment Outcome
 Repeated, Brief Measures
 Example: Longitudinal Interview Follow-up Evaluation
(LIFE): Ratings on a fixed scale that are repeated each
week
 Advantages: Low burden on the respondent, sensitivity to
trends of change over treatment
 Disadvantage: Resulting data are difficult to analyze
using conventional statistical methods
 Behavior Checklists and Questionnaires
 Example: Child Depression Inventory
 Advantage: Issue of converting outcomes to effect sizes
Clinically Significant Change
 Measured in two ways:
1. Reliable change
• Is the amount of change shown by an individual case large enough
to reflect “true” change, in the classical theory sense rather than
being attributable to the unreliability of the measure?
• Reliable Change Index (RCI) is expressed as a z-score with the
standard error of the difference acting as the denominator
2. Change that moves the patient’s score below a
predefined normative threshold
Mediators and Moderators
 Mediators
 The intermediate variables that provide mechanisms of
change
 Measurement error, potential suppression effects, and
low statistical power complicate the assessment of
mediational models
 Moderators
 Also known as an “interaction effect” because moderators
change the relationship between another pair of variables
 Diathesis-Stress Model is an example
Adherence and Fidelity
 Both influence the effectiveness of an intervention
 Measuring adherence
 Rate of completion of homework assignments
 Attendance of scheduled sessions
 Measuring fidelity
 Therapist progress notes, checklists, or rating scales
completed by the therapist
Reappraisal of the “Three Ps” –
Evaluating Assessments
 Practitioners should critically read the literature and
look for the best measures suited to their evaluation
 Conduct studies that directly compare assessment
methods on these three dimensions
 Conduct more meta-analytic studies
 If new measures are developed, adaptive testing using
Item Response Theory to quantify item properties will
make it possible to obtain increased precision and a
better range of scores without increasing participant
burden