Transcript Statistical Thinking and Analysis

Statistical Thinking and Analysis
Deming – Theory of Profound Knowledge
 Systems thinking –
 System is more than a sum of its parts
 Understanding that the parts interact to produce the end product
 Coordination and collaboration of parts increase productivity of system
 Interconnected subsystems and processes affect each other
 Process variation –
 All processes have variation that can either be inherent in the process or due to external influences
 Variation is a major source of nonconforming output leading to reduced quality and higher cost
 Identifying and reducing sources of variation is a major undertaking for performance improvement
initiatives
 Theory of knowledge –
 Information must be tempered by experience and theory to become knowledge
 Effective managers combine experience and theory to create organizational knowledge
 Psychology –
 Understanding the variation in people is as important as understanding the variation in processes
 Successful managers use human psychology to effectively coordinate, collaborate, and motivate workers to optimize
system outcomes
Statistical thinking
 Ron Snee (1986):
“… statistical thinking is used to describe the thought processes that acknowledge
the ubiquitous nature of variation and that its identification, characterization,
quantification, control, and reduction provide a unique opportunity for
improvement. ”
“ ….Every enterprise is made up of a collection of interconnected processes
whose input, control variable, and output are subject to variation. This leads
to the conclusion that statistical thinking must be used routinely at all levels of
the organization.”
.
Shewhart’s concept of process variation
 Common cause
 Variation is based on the process
materials and procedures;
 Variation is predictable using
mathematics related to
probability and chance;
 Variation is irregular, i.e. shows
no particular pattern; and
 High and low values within the
measurements are statistically
indistinguishable
 Special (attributable) cause
 Variation due to new, unanticipated,
emergent, or previously unknown
factors within the system;
 Variation that is entirely unpredictable,
even using statistical probability
techniques;
 Variation that is outside historical
trends; and
 Variation that indicates an underlying
change in the system or some previously
unidentified factor.
Translated into statistical thinking:
 Common Cause variation
 Data points from process fall inside control limits
 Data points are statistically indistinguishable
 Special Cause variation
 Data points from process fall outside control limits (3 standard
deviations)
 < 0.3% (3 chances per 1000) probability of occurrence
 Tampering
 Deming’s concept of treating common cause variation like a
special cause
Why is this stuff important?
Let’s Review Some Data
Collection Rules…
Data collection principles
 Understand need for information collected
 Collect everything you think might be needed
 Least invasive methods of collection
 Operational definition of each data element
 Appropriate format for analysis
 Before starting, review the study to ensure correctness
Common data sources in health care
Source
Claims
Pro
Con
•Data used to pay claims
•Analyzed for errors by edits in payer
computer system
•Data entry errors – insurer, provider
•Paucity of information (limited clinical info)
•Inconsistent payments for same services
•Upcoding
•Capitation effects
Medicaid
•Consistent coding systems within state
•Population fairly uniform
•Same as above
•Varying types of plans around the US
•Tendency to upcode more pronounced
Medicare
•Relatively consistent data set
•Edits tend to reduce coding errors
•Upcoding still a problem
•Payment schedules vary by region, more than by
specialty
Provider
Billing
Systems
•Source data from point of care
•Usually consistent within a practice
•Broad variation in coding between practices
•Coding variation also for same services
•Variety of formats
•Original source data from point of care
•Complete record of clinical encounter
•Expensive to review
•Variation in recording
•Handwriting
•Variety of recording conventions
•Measures customer opinions directly
•Often can be done simply
•Lack of scientific approach, leading to bias
•Selection bias
•Validation
Patient Charts
Surveys
Statistical process control (SPC) – a
method to understand variation
 Shows trends in the process mean over time
 Evaluates process variability at each point in
time
 Provides graphic evidence that process is in
control (or not) at each point
Two primary types of data
 Attributes
 Counts of individual items
 Examples?
 Continuous (variables)
 Variables along a measurement scale
 Real numbers, no “gaps” between measures
Types of Control Charts
 Attribute data charts
 p and np charts
 c and u charts
 Continuous data charts
 IX-MR charts
 X-bar and R charts
 X-bar and s charts
Commonly used control charts
Control Charts for Attributes
Data
Attribute chart selection
 p-charts
 Proportions of nonconformities
 Example: C-section rates
 np-charts
 Numbers of nonconformities
 Example: maternal deaths
 c-chart
 Nonconformities per inspection unit, constant number of inspection
units
 Examples: housekeeping errors per room; missed appointments per day
 u-chart
 Nonconformities per inspection unit, like c, BUT…
 Used when the number of inspection units varies
Attributes data limb
of decision tree…
Example c-chart
20
15
Mean
LCL
10
UCL
Nonconformities
5
Day of Study
10
7
6
5
4
3
2
0
1
Number of Nonconformities
c-chart for XYZ Clinic
u-charts
u-chart for St. Elsewhere Food Service
Note “wavy” control
limit line – why?
0.0700
0.0600
UCL
0.0400
LCL
0.0300
u
0.0200
Mean u
0.0100
Day of Data Collection
7
-0.0100
4
0.0000
1
u-value
0.0500
Commonly used control charts
Control Charts for
Continuous Variables
Continuous
(variables) data limb
of decision tree…
IX-MR chart creation
 Example: ALOS for a hospital
 Data obtained from a hospital over 24 months
 Calculate mean of all samples, plot as center line
 Calculate MR, average moving range
 Control limits = + D4 * MR-bar (D4 is the
“correction factor”, see Table 5.7, p 191 in the
text)
 Plot on graph
Remember: software does this work for you…
IX-MR chart
40
IX Chart
30
24.7170
Average
20
10
7.3636
0
-10
-9.9897
-20
Range
Date/Time/Period
30
25
20
15
10
5
0
MR Chart
Note out of control MR
chart point!
21.3133
6.5238
Date/Time/Period/Number
IX chart has sample size of 1
Moving range is the difference between successive points and is surrogate for
standard deviation (with correction factor)
IX-MR chart
 What’s important?
 MR Chart – what does it mean if the MR is out of
control?
 IX Chart – what does it mean if an IX value is out of
control?
 What other analyses could we do?
Common control charts depend on…
 “Reasonable” conformity of the data set to a Gaussian (normal, bell
shaped) distribution
 Most analysis programs will provide a histogram of the data to
determine if data are normally distributed
IX-MR Histogram
12
10
0.08
24.71
-9.99
0.07
0.06
8
Number
0.05
6
0.04
0.03
4
0.02
2
0.01
0
-9.986920881
0
-3.046697983
3.893524915
10.83374781
17.77397071
24.71419361
Note bimodal distribution of data, indicating reason for MR chart in previous slide to be out of
control; thus, IX-MR may not be appropriate for this data set
What if the MR is out of control?
 Determine special cause using root cause analysis and eliminate
 Re-run the analysis with special cause eliminated
 Track data through more cycles to ensure that attributable cause
was correctly identified
 Other options:
 Data transformation, e.g. natural log of each point
 Usually better to identify special cause
Other types of continuous variable
charts
X-bar and Range Chart
 Similar to IX-MR chart, except:
 Subgroup size = 2 – 9
 Measure of variation is range
 Procedure:
 Mean of each subgroup plotted
 Mean of those means is centerline
 Range of each subgroup plotted
 Mean of those ranges is centerline
 D4 is used to adjust ranges to control limits
 A2 is used to create X-bar control limits
X-bar-R chart
50
Phlebotomist Time - Notify to Draw
Average Time (X-bar)
40
30
20
10
Day of Study
60
Subgroup Range
4.5943
0
R Chart - Phlebotomist Time
40
20
0
0.0000
Day of Study
Note the R chart is in control
The histogram is “reasonably”
normally distributed…
Histogram
40
35
-22.82
59.54
30
Number
25
20
15
10
5
0
© 2011 Jones
and Bartlett
Publishers,
LLC
-22.8158868-14.57998216
-6.344077532
1.8918271
10.12773173 18.36363636
26.599541 34.83544563 43.07135026 51.30725489 59.54315952 67.77906415
The last commonly used
continuous variable chart…
X-bar and s chart
 Similar to others, except
 Subgroup size >9
 Measure of variation = sample standard deviation
 Procedure
 Mean of each subgroup plotted
 Mean of those means is centerline
 s of each subgroup calculated and plotted
 Mean of those s-values is centerline
 B4 and B3 are used to adjust s to control limits
 A3 is used to create X-bar control limits
Airflow Example
 Airflow measurements on a clinical unit
 Ten measurements a day, spaced throughout the day
 Subgroup size = 10
 Subgroup time period = 1 day
 Measurements then plotted on x-bar s chart
The Airflow Example
37.43
UCL
33.66
CL
29.89
LCL
6.62731
UCL
3.86207
CL
Xbar
SD
1.09683
LCL
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
Day
Note similarity to X-bar-R chart
20
21
22
23
24
25
Data conversion…
 Used when raw data are not normally distributed
 Used when raw data sample sizes are not uniform
 Types of conversions
 Lognormal
 Arcsin
 z-score
 How do we calculate z-scores?
Example Z-score plot
Z-Score Chart
4.000
3.000
UCL
2.000
1.000
0.000
Mean
-1.000
-2.000
-3.000
-4.000
Z scores are x-values divided by the standard deviation
LCL
In summary…
 Data types - attribute vs. continuous variables determine type of




control chart
Control charts have center line (average of control chart means) and
upper and lower control limits (+3s)
For attribute charts, data points are nonconformity values or rates
For continuous variable charts, data points are sample values or
averages of sample values
Measures of variation for control charts are corrected using bias
correction tables
Other useful analyses
ANOM
ANOVA
Regression
Rankings are used in health care
 Concept of rankings
 How are they used?
 Are they valid?
 What about control limits?
 Measures falling within control limits are common cause - statistically
indistinguishable
 Can’t be ranked!
 Time factor - most rankings are for specific period of time
 Physician or provider profiles – experiences?
Ranking – some approaches to
validation
 95% Confidence Intervals
 Not time series based, usually single point in time
 Help establish the level of variation in the
measurement used for the ranking (higher
variation, less predictive ability from ranks)
 Still difficult to identify outliers
Percentiles
 Often used for comparisons
 Examples
 Percent mortality post-op
 Nosocomial infection rates
 Error rate for claims entry
 Others?
 Problems with percentages
 Denominator size may vary, making comparisons potentially
invalid
 Case mix adjustment not often done to adjust for sampling bias
Now for something a little
different… Analysis of Means!
 Not time series data
 Used for attribute (count) data with unequal subgroup
sizes
 Rate of particular measure of count data
 Examples?




C-section rates
Antibiotic utilization rates
Infection rates post-op
Others?
 Does provide adjustment for issues like case mix, if done
correctly
ANOM example: C-section rates
ANOM Chart - Comparison of Proportion Data
0.600
0.500
Proportion
0.400
Proportions
0.300
Lower Common Cause Limits
Upper Common Cause Limits
0.200
0.100
0.000
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
Subject Number
C-section rates among providers doing deliveries
UCCL = upper control limits for each provider
LCCL = lower control limits for each provider
Control limits adjusted for opportunities, i.e. cases, that provider treats
22
23
24
25
26
27
28
29
30
Analysis of variance (ANOVA)
 Test hypotheses about differences between two or more means
 Used in DOE to determine if changes in mean in one
intervention subgroup statistically differ from other intervention
subgroups
 See Example 5.6 (p 220)
Regression analysis
 Test hypotheses of relationships between a response variable (Y)
and one or more predictor variables (X)
 Determination of statistical significance of relationships (r value)
 Sign of coefficient (b) for predictor variable determines if effect
is positive or negative
 R2 value provides predictive level of model (i.e. how much of the
variation in Y is due to the selected predictor variables)
Types of regression
 Simple linear regression – relates one x-variable to one
dependent y-variable
Linear Model
120
100
80
y = 4.007x + 8.663
60
R² = 0.9793
40
20
0
0
5
10
15
20
25
30
Types of regression
 Multiple regression
 One dependent variable with multiple predictor variables
 Graphic output is a multidimensional surface, so usually not
provided
 Output includes:
 Coefficients (b) and levels of significance (p-value) for each x-value
 r value
 R2 value
Design of Experiments
A scientific approach to improvement
DOE – when evidence is needed
 Method for validating processes and determining
which factors are most important
 Just like in science class – multiple runs, varying
“factors” (predictor variables) at different “levels”
 Statistically valid approach to identify “main effects”
(primary effect of each factor) and “interaction
effects” (effects caused by combinations of factors)
 Optimization of experiment is desirable to ensure
identification of salient factors