Transcript DESIGN OF EXPERIMENTS

DESIGN OF EXPERIMENTS
by
R. C. Baker
How to gain 20 years of experience in
one short week!
1
Role of DOE in Process Improvement
• DOE is a formal mathematical method for
systematically planning and conducting
scientific studies that change experimental
variables together in order to determine
their effect of a given response.
• DOE makes controlled changes to input
variables in order to gain maximum
amounts of information on cause and effect
relationships with a minimum sample size. 2
Role of DOE in Process Improvement
• DOE is more efficient that a standard
approach of changing “one variable at a
time” in order to observe the variable’s
impact on a given response.
• DOE generates information on the effect
various factors have on a response variable
and in some cases may be able to determine
optimal settings for those factors.
3
Role of DOE in Process Improvement
• DOE encourages “brainstorming” activities
associated with discussing key factors that
may affect a given response and allows the
experimenter to identify the “key” factors
for future studies.
• DOE is readily supported by numerous
statistical software packages available on
the market.
4
BASIC STEPS IN DOE
•
•
•
•
•
Four elements associated with DOE:
1. The design of the experiment,
2. The collection of the data,
3. The statistical analysis of the data, and
4. The conclusions reached and
recommendations made as a result of the
experiment.
5
TERMINOLOGY
• Replication – repetition of a basic
experiment without changing any factor
settings, allows the experimenter to estimate
the experimental error (noise) in the system
used to determine whether observed
differences in the data are “real” or “just
noise”, allows the experimenter to obtain
more statistical power (ability to identify
small effects)
6
TERMINOLOGY
• .Randomization – a statistical tool used to
minimize potential uncontrollable biases in
the experiment by randomly assigning
material, people, order that experimental
trials are conducted, or any other factor not
under the control of the experimenter.
Results in “averaging out” the effects of the
extraneous factors that may be present in
order to minimize the risk of these factors
affecting the experimental results.
7
TERMINOLOGY
• Blocking – technique used to increase the
precision of an experiment by breaking the
experiment into homogeneous segments
(blocks) in order to control any potential
block to block variability (multiple lots of
raw material, several shifts, several
machines, several inspectors). Any effects
on the experimental results as a result of the
blocking factor will be identified and
minimized.
8
TERMINOLOGY
• Confounding - A concept that basically
means that multiple effects are tied together
into one parent effect and cannot be
separated. For example,
• 1. Two people flipping two different coins
would result in the effect of the person and
the effect of the coin to be confounded
• 2. As experiments get large, higher order
interactions (discussed later) are
confounded with lower order interactions or
main effect.
9
TERMINOLOGY
• Factors – experimental factors or
independent variables (continuous or
discrete) an investigator manipulates to
capture any changes in the output of the
process. Other factors of concern are those
that are uncontrollable and those which are
controllable but held constant during the
experimental runs.
10
TERMINOLOGY
• Responses – dependent variable measured
to describe the output of the process.
• Treatment Combinations (run) –
experimental trial where all factors are set at
a specified level.
11
TERMINOLOGY
•
•
Fixed Effects Model - If the treatment
levels are specifically chosen by the
experimenter, then conclusions reached
will only apply to those levels.
Random Effects Model – If the treatment
levels are randomly chosen from a
population of many possible treatment
levels, then conclusions reached can be
extended to all treatment levels in the
population.
12
PLANNING A DOE
• Everyone involved in the experiment should
have a clear idea in advance of exactly what
is to be studied, the objectives of the
experiment, the questions one hopes to
answer and the results anticipated
13
PLANNING A DOE
• Select a response/dependent variable
(variables) that will provide information
about the problem under study and the
proposed measurement method for this
response variable, including an
understanding of the measurement system
variability
14
PLANNING A DOE
• Select the independent variables/factors
(quantitative or qualitative) to be
investigated in the experiment, the number
of levels for each factor, and the levels of
each factor chosen either specifically (fixed
effects model) or randomly (random effects
model).
15
PLANNING A DOE
• Choose an appropriate experimental design
(relatively simple design and analysis methods
are almost always best) that will allow your
experimental questions to be answered once the
data is collected and analyzed, keeping in mind
tradeoffs between statistical power and economic
efficiency. At this point in time it is generally
useful to simulate the study by generating and
analyzing artificial data to insure that
experimental questions can be answered as a
result of conducting your experiment
16
PLANNING A DOE
• Perform the experiment (collect data)
paying particular attention such things as
randomization and measurement system
accuracy, while maintaining as uniform an
experimental environment as possible.
How the data are to be collected is a critical
stage in DOE
17
PLANNING A DOE
• Analyze the data using the appropriate
statistical model insuring that attention is
paid to checking the model accuracy by
validating underlying assumptions
associated with the model. Be liberal in the
utilization of all tools, including graphical
techniques, available in the statistical
software package to insure that a maximum
amount of information is generated
18
PLANNING A DOE
• Based on the results of the analysis, draw
conclusions/inferences about the results,
interpret the physical meaning of these
results, determine the practical significance
of the findings, and make recommendations
for a course of action including further
experiments
19
SIMPLE COMPARATIVE EXPERIMENTS
• Single Mean Hypothesis Test
• Difference in Means Hypothesis Test with
Equal Variances
• Difference in Means Hypothesis Test with
Unequal Variances
• Difference in Variances Hypothesis Test
• Paired Difference in Mean Hypothesis Test
• One Way Analysis of Variance
20
CRITICAL ISSUES ASSOCIATED WITH
SIMPLE COMPARATIVE EXPERIMENTS
• How Large a Sample Should We Take?
• Why Does the Sample Size Matter
Anyway?
• What Kind of Protection Do We Have
Associated with Rejecting “Good” Stuff?
• What Kind of Protection Do We Have
Associated with Accepting “Bad” Stuff?
21
Single Mean Hypothesis Test
• After a production run of 12 oz. bottles,
concern is expressed about the possibility
that the average fill is too low.
• Ho: m = 12
• Ha: m <> 12
• level of significance = a = .05
• sample size = 9
• SPEC FOR THE MEAN: 12 + .1
22
Single Mean Hypothesis Test
•
•
•
•
•
•
Sample mean = 11.9
Sample standard deviation = 0.15
Sample size = 9
Computed t statistic = -2.0
P-Value = 0.0805162
CONCLUSION: Since P-Value > .05, you
fail to reject hypothesis and ship product.
23
Single Mean Hypothesis Test Power Curve
Power Curve
alpha = 0.05, sigma = 0.15
1
Power
0.8
0.6
0.4
0.2
0
11.8
11.9
12
12.1
12.2
True Mean
24
Single Mean Hypothesis Test Power
Curve - Different Sample Sizes
25
DIFFERENCE IN MEANS - EQUAL
VARIANCES
•
•
•
•
•
Ho: m1 = m2
Ha: m1 <> m2
level of significance = a = .05
sample sizes both = 15
Assumption: s1 = s2
*****************************************************
• Sample means = 11.8 and 12.1
• Sample standard deviations = 0.1 and 0.2
• Sample sizes = 15 and 15
26
DIFFERENCE IN MEANS - EQUAL VARIANCES
Can you detect this difference?
27
DIFFERENCE IN MEANS - EQUAL
VARIANCES
28
DIFFERENCE IN MEANS - unEQUAL
VARIANCES
• Same as the “Equal Variance” case except
the variances are not assumed equal.
• How do you know if it is reasonable to
assume that variances are equal OR
unequal?
29
DIFFERENCE IN VARIANCE
HYPOTHESIS TEST
•
•
•
•
•
•
•
•
•
Same example as Difference in Mean:
Sample standard deviations = 0.1 and 0.2
Sample sizes = 15 and 15
**********************************
Null Hypothesis: ratio of variances = 1.0
Alternative: not equal
Computed F statistic = 0.25
P-Value = 0.0140071
Reject the null hypothesis for alpha = 0.05. 30
DIFFERENCE IN VARIANCE
HYPOTHESIS TEST
Can you detect this difference?
31
DIFFERENCE IN VARIANCE
HYPOTHESIS TEST -POWER CURVE
32
PAIRED DIFFERENCE IN MEANS
HYPOTHESIS TEST
• Two different inspectors each measure 10
parts on the same piece of test equipment.
• Null hypothesis: DIFFERENCE IN
MEANS = 0.0
• Alternative: not equal
• Computed t statistic = -1.22702
• P-Value = 0.250944
• Do not reject the null hypothesis for alpha =
0.05.
33
PAIRED DIFFERENCE IN MEANS
HYPOTHESIS TEST - POWER CURVE
Power Curve
alpha = 0.05, sigma = 3.866
1
Power
0.8
0.6
0.4
0.2
0
-5
-4
-3
-2
-1
0
1
2
3
4
5
Difference in Means
34
ONE WAY ANALYSIS OF VARIANCE
• Used to test hypothesis that the means of
several populations are equal.
• Example: Production line has 7 fill needles
and you wish to assess whether or not the
average fill is the same for all 7 needles.
• Experiment: sample 20 fills from each of
the 9 needles and test at 5% level of sign.
• Ho: m1 = m2 = m3= m4 = m5 = m6 = m7
35
RESULTS: ANALYSIS OF VARIANCE
TABLE
Analysis of Variance
---------------------------------------------------------------------------Source
Sum of Squares
Df Mean Square
F-Ratio
P-Valu
---------------------------------------------------------------------------Between groups
1.10019
6
0.183364
18.66
0.000
Within groups
1.30717
133
0.00982837
---------------------------------------------------------------------------Total (Corr.)
2.40736
139
36
SINCE NEEDLE MEANS ARE NOT ALL
EQUAL, WHICH ONES ARE DIFFERENT?
• Multiple Range Tests for 7 Needles
Method: 95.0 percent LSD
Col_2
Count
Mean
Homogeneous Groups
-------------------------------------------------------------------------------N7
20
11.786
X
N2
20
11.9811
X
N1
20
11.9827
X
N6
20
11.9873
X
N3
20
11.9951
X
N5
20
11.9953
X
N4
20
12.11
X
37
VISUAL COMPARISON OF 7
NEEDLES
Box-and-Whisker Plot
N1
Col_2
N2
N3
N4
N5
N6
N7
11.5
11.7
11.9
12.1
12.3
Col_1
38
FACTORIAL (2k) DESIGNS
• Experiments involving several factors ( k =
# of factors) where it is necessary to study
the joint effect of these factors on a specific
response.
• Each of the factors are set at two levels (a
“low” level and a “high” level) which may
be qualitative (machine A/machine B, fan
on/fan off) or quantitative (temperature
800/temperature 900, line speed 4000 per
hour/line speed 5000 per hour).
39
FACTORIAL (2k) DESIGNS
• Factors are assumed to be fixed (fixed
effects model)
• Designs are completely randomized
(experimental trials are run in a random
order, etc.)
• The usual normality assumptions are
satisfied.
40
FACTORIAL (2k) DESIGNS
• Particularly useful in the early stages of
experimental work when you are likely to
have many factors being investigated and
you want to minimize the number of
treatment combinations (sample size) but, at
the same time, study all k factors in a
complete factorial arrangement (the
experiment collects data at all possible
combinations of factor levels).
41
FACTORIAL (2k) DESIGNS
• As k gets large, the sample size will
increase exponentially. If experiment is
replicated, the # runs again increases.
k
2
3
4
5
6
7
8
9
10
# of runs
4
8
16
32
64
128
256
512
1024
42
FACTORIAL (2k) DESIGNS (k = 2)
• Two factors set at two levels (normally
referred to as low and high) would result in
the following design where each level of
factor A is paired with each level of factor
B.
Generalized Settings
RUN Factor A Factor B RESPONSE
Orthogonal Settings
RUN Factor A Factor B RESPONSE
1
low
low
y1
1
-1
-1
y1
2
high
low
y2
2
+1
-1
y2
3
low
high
y3
3
-1
+1
y3
4
high
high
y4
4
+1
+1
y4
43
FACTORIAL (2k) DESIGNS (k = 2)
• Estimating main effects associated with
changing the level of each factor from low
to high. This is the estimated effect on the
response variable associated with changing
factor A or B from their low to high values.
( y 2  y 4 ) ( y1  y3 )
Factor A Effect =

2
2
( y3  y4 ) ( y1  y2 )
Factor B Effect =

2
2
44
FACTORIAL (2k) DESIGNS (k = 2):
GRAPHICAL OUTPUT
• Neither factor A nor Factor B have an effect
on the response variable.
45
FACTORIAL (2k) DESIGNS (k = 2):
GRAPHICAL OUTPUT
• Factor A has an effect on the response
variable, but Factor B does not.
46
FACTORIAL (2k) DESIGNS (k = 2):
GRAPHICAL OUTPUT
• Factor A and Factor B have an effect on the
response variable.
47
FACTORIAL (2k) DESIGNS (k = 2):
GRAPHICAL OUTPUT
• Factor B has an effect on the response variable, but only if
factor A is set at the “High” level. This is called
interaction and it basically means that the effect one factor
has on a response is dependent on the level you set other
factors at. Interactions can be major problems in a DOE if
you fail to account for the interaction when designing your
experiment.
48
EXAMPLE:
FACTORIAL (2k) DESIGNS (k = 2)
• A microbiologist is interested in the effect
of two different culture mediums [medium 1
(low) and medium 2 (high)] and two
different times [10 hours (low) and 20 hours
(high)] on the growth rate of a particular
CFU [Bugs].
49
EXAMPLE:
FACTORIAL (2k) DESIGNS (k = 2)
• Since two factors are of interest, k =2, and
we would need the following four runs
resulting in
Generalized Settings
RUN Medium
Time
Growth Rate
1
low
low
17
2
high
low
15
3
low
high
38
4
high
high
39
50
EXAMPLE:
FACTORIAL (2k) DESIGNS (k = 2)
• Estimates for the medium and time
effects are
• Medium effect = [(15+39)/2] – [(17 + 38)/2]
= -0.5
• Time effect = [(38+39)/2] – [(17 + 15)/2] =
22.5
51
EXAMPLE:
FACTORIAL (2k) DESIGNS (k = 2)
52
EXAMPLE:
FACTORIAL (2k) DESIGNS (k = 2)
• A statistical analysis using the appropriate
statistical model would result in the
following information. Factor A (medium)
and Factor B (time)
Type III Su ms of S quares
---- ------- ------- ------- ------ ------- ------- ------- ------- ------- ------- ------ ---Sour ce
Sum of Squa res
Df
M ean Squ are
F-Ratio
P- Valu
---- ------- ------- ------- ------ ------- ------- ------- ------- ------- ------- ------ ---FACT OR A
0 .25
1
0 .25
0.11
0 .795
FACT OR B
506 .25
1
506 .25
225.00
0 .042
Resi dual
2 .25
1
2 .25
---- ------- ------- ------- ------ ------- ------- ------- ------- ------- ------- ------ ---Tota l (corr ected)
508 .75
3
All F-ratio s are b ased on the r esidual mean s quare e rror.
53
EXAMPLE:
CONCLUSIONS
• In statistical language, one would conclude
that factor A (medium) is not statistically
significant at a 5% level of significance
since the p-value is greater than 5% (0.05),
but factor B (time) is statistically significant
at a 5 % level of significance since this pvalue is less than 5%.
54
EXAMPLE:
CONCLUSIONS
• In layman terms, this means that we have no
evidence that would allow us to conclude
that the medium used has an effect on the
growth rate, although it may well have an
effect (our conclusion was incorrect).
55
EXAMPLE:
CONCLUSIONS
• Additionally, we have evidence that would
allow us to conclude that time does have an
effect on the growth rate, although it may
well not have an effect (our conclusion was
incorrect).
56
EXAMPLE:
CONCLUSIONS
• In general we control the likelihood of
reaching these incorrect conclusions by the
selection of the level of significance for the
test and the amount of data collected
(sample size).
57
2k DESIGNS (k > 2)
• As the number of factors increase, the
number of runs needed to complete a
complete factorial experiment will increase
dramatically. The following 2k design
layout depict the number of runs needed for
values of k from 2 to 5. For example, when
k = 5, it will take 25 = 32 experimental runs
for the complete factorial experiment.
58
Interactions for 2k Designs (k = 3)
•
Interactions between various factors can
be estimated for different designs above
by multiplying the appropriate columns
together and then subtracting the average
response for the lows from the average
response for the highs.
59
Interactions for 2k Designs (k = 3)
a
-1
+1
-1
+1
-1
+1
-1
+1
b
-1
-1
+1
+1
-1
-1
+1
+1
c
-1
-1
-1
-1
+!
+1
+1
+1
ab
1
-1
-1
1
1
-1
-1
1
ac
1
-1
1
-1
-1
1
-1
1
bc
1
1
-1
-1
-1
-1
1
1
abc
-1
1
1
-1
1
-1
-1
1
60
2k DESIGNS (k > 2)
• Once the effect for all factors and
interactions are determined, you are able to
develop a prediction model to estimate the
response for specific values of the factors.
In general, we will do this with statistical
software, but for these designs, you can do
it by hand calculations if you wish.
61
2k DESIGNS (k > 2)
• For example, if there are no significant interactions
present, you can estimate a response by the
following formula. (for quantitative factors only)
Y = (average of all responses) +
=Y (

factorEFFECT
[(
) * ( factorLEVEL)]
2
A

)* A ( B )*B
2
2
62
ONE FACTOR EXAMPLE
Plot of Fitted Model
GRADE
95
85
75
65
55
10
12
14
16
#HRS STUDY
18
20
63
ONE FACTOR EXAMPLE
• The output shows the results of fitting a
general linear model to describe the
relationship between GRADE and #HRS
STUDY. The equation of the fitted general
model is
• GRADE = 29.3 + 3.1* (#HRS STUDY)
• The fitted orthogonal model is
• GRADE = 75 + 15 * (SCALED # HRS)
64
Two Level Screening Designs
• Suppose that your brainstorming session
resulted in 7 factors that various people
think “might” have an effect on a response.
A full factorial design would require 27 =
128 experimental runs without replication.
The purpose of screening designs is to
reduce (identify) the number of factors
down to the “major” role players with a
minimal number of experimental runs. One
way to do this is to use the 23 full factorial
design and use interaction columns for
factors.
65
Note that
* Any factor d effect is now confounded with the a*b
interaction
* Any factor e effect is now confounded with the a*c
interaction
* etc.
* What is the d*e interaction confounded with????????
a
-1
+1
-1
+1
-1
+1
-1
+1
b
-1
-1
+1
+1
-1
-1
+1
+1
c
-1
-1
-1
-1
+!
+1
+1
+1
d = ab
1
-1
-1
1
1
-1
-1
1
e = ac
1
-1
1
-1
-1
1
-1
1
f = bc
1
1
-1
-1
-1
-1
1
1
g = abc
-1
1
1
-1
1
-1
-1
1
66
Problems that Interactions Cause!
•
Interactions – If interactions exist and you fail to
account for this, you may reach erroneous
conclusions. Suppose that you plan an
experiment with four runs and three factors
resulting in the following data:
Run
1
2
3
4
Factor A
+1
+1
-1
-1
Factor B
+1
-1
+1
-1
Results
10
5
5
10
67
Problems that Interactions Cause!
• Factor A Effect = 0
• Factor B Effect = 0
• In this example, if you were assuming that
“smaller is better” then it appears to make
no difference where you set factors A and B.
If you were to set factor A at the low value
and factor B at the low value, your response
variable would be larger than desired. In
this case there is a factor A interaction with
factor B.
68
Problems that Interactions Cause!
Interaction Plot
RESPONSE
10
FACT OR B
-1
1
9
8
7
6
5
-1
1
FACTOR A
69
Resolution of a Design
•
•
•
Resolution III Designs – No main effects are
aliased with any other main effect BUT some (or
all) main effects are aliased with two way
interactions
Resolution IV Designs – No main effects are
aliased with any other main effect OR two factor
interaction, BUT two factor interactions may be
aliased with other two factor interactions
Resolution V Designs – No main effect OR two
factor interaction is aliased with any other main
effect or two factor interaction, BUT two factor
interactions are aliased with three factor
interactions.
70
Common Screening Designs
•
Fractional Factorial Designs – the total
number of experimental runs must be a
power of 2 (4, 8, 16, 32, 64, …). If you
believe first order interactions are small
compared to main effects, then you could
choose a resolution III design. Just
remember that if you have major
interactions, it can mess up your screening
experiment.
71
Common Screening Designs
•
Plackett-Burman Designs – Two level,
resolution III designs used to study up to
n-1 factors in n experimental runs, where
n is a multiple of 4 ( # of runs will be 4, 8,
12, 16, …). Since n may be quite large,
you can study a large number of factors
with moderately small sample sizes. (n =
100 means you can study 99 factors with
100 runs)
72
Other Design Issues
• May want to collect data at center points to
estimate non-linear responses
• More than two levels of a factor – no
problem (multi-level factorial)
• What do you do if you want to build a nonlinear model to “optimize” the response. (hit
a target, maximize, or minimize) – called
response surface modeling
73
Response Surface Designs – Box-Behnken
RUN
F1
F2
F3
Y100
1
10
45
60
11825
2
30
45
40
8781
3
20
30
40
8413
4
10
30
50
9216
5
20
45
50
9288
6
30
60
50
8261
7
20
45
50
9329
8
30
45
60
10855
9
20
45
50
9205
10
20
60
40
8538
11
10
45
40
9718
12
30
30
50
11308
13
20
60
60
10316
14
10
60
50
12056
15
20
30
60
10378
74
Response Surface Designs – Box-Behnken
Regression coeffs. for Var_3
---------------------------------------------------------------------constant
= 2312.5
A:Factor_A = 36.575
B:Factor_B = 200.067
C:Factor_C = 3.85
AA
= 9.09875
AB
= -9.81167
AC
= -0.0825
BB
= 0.117222
BC
= -0.311667
CC
= 1.10875
75
Response Surface Designs – Box-Behnken
Contours of Estimated Response Surface
Factor_C=60.0
60
Factor_B
55
50
45
40
35
30
10
14
18
22
Factor_A
26
30
Var_3
9300.0
9500.0
9700.0
9900.0
10100.0
10300.0
10500.0
10700.0
10900.0
11100.0
11300.0
11500.0
11700.0
76
CLASSROOM EXERCISE
•
•
•
•
STUDENT IN-CLASS EXPERIMENT:
Collect data for experiment to determine
factor settings (two factors) to hit a target
response (spot on wall).
Factor A – height of shaker (low and high)
Factor B – location of shaker (close to
hand and close to wall)
Design experiment – would suggest
several replications
77
CLASSROOM EXERCISE
• Conduct Experiment – student holds 3 foot
“pin the tail on the donkey” stick and
attempts to hit the target. An observer will
assist to mark the hit on the target.
• Collect data – students take data home for
week and come back with what you would
recommend AND why.
• YOU TELL THE CLASS HOW TO PLAY
THE GAME TO “WIN”.
78
CLASSROOM EXERCISE
79
CLASSROOM EXERCISE
MARKER VERTICAL
1ST OBS 2ND OBS 3RD OBS 4TH OBS
STICK
POLE
MEAN
STANDARD
DEVIATION
L
L
-2.750
-4.500
-4.750
-5.000
-4.250
1.021
H
L
-12.500
-6.750
-4.625
-4.000
-6.969
3.871
L
H
3.000
3.250
3.875
6.250
4.094
1.484
H
H
4.625
11.250
12.625
14.000
10.625
4.155
MARKER
STICK
L = VERTICAL POLE WAS CLOSE TO WALL (MARKER END OF STICK
H=VERTICAL POLE WAS CLOSE TO HAND
VERTICAL
POLE
L=SHAKING DEVICE LOCATED LOW ON VERTICAL POLE
H=SHAKING DEVICE LOCATED HIGH ON VERTICAL POLE
80
Contour Plots for Mean and Std. Dev.
81