Transcript Part One - National Institute of Aerospace

Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
Overview of Uncertainty in
Aerospace Design
Dr. Douglas Stanley
Georgia Institute of Technology
National Institute of Aerospace
757.325.6811
[email protected]
Dr. Alan Wilhite
Georgia Institute of Technology
National Institute of Aerospace
757.864.6810
[email protected]
With Special Thanks to: Dr. Dimitri Mavris and Dr. Michelle Kirby
of Georgia Tech
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
Outline
• Definitions
• Brief Review of Probability
–
–
–
–
Random Events
Probability Distributions
Sampling
Functions of Random Variables
• Overview of Risk and Continuous Risk Management
–
–
–
–
–
Risk Identification
Risk Analysis
Risk Planning
Risk Tracking and Control
Expert Elicitation in Risk Management
• Uncertainty and Margin in Design
–
–
–
–
Weight/Performance Margins and Uncertainty
Cost Margins and Uncertainty
Schedule Margins and Uncertainty
Technology Risk Mitigation
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
•
•
•
•
•
•
Outline
Risk Leveling
Probabilistic Risk Assessment
Response Surface Methods
Probabilistic Design
Example of Probabilistic Design Under Uncertainty
Decision Making Process
–
–
–
–
Characteristics
Common Biases
Figures of Merit
Multi-Attribute Utility Theory
• Making Design Decisions Under Uncertainty
• Summary
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
•
Definitions
Uncertainty
– The state of being uncertain; Doubt
– The estimated amount by which a calculated value may differ from the
true value
•
Uncertain
– Not known or established; Not determined, Not having sure knowledge
•
Risk
– The possibility of suffering harm or loss; Danger; Hazard
– The chance of loss; The degree of probability of loss
– Probability of a non-desirable event
•
Probability
– The relative possibility that an event will occur; Likelihood
– The relative frequency with which an event is likely to occur
– Ratio of number of occurrences to number of possible occurrences
•
We wish to use probability to measure uncertainty, in order to reduce
uncertainty to mitigate risk or to make designs robust to uncertainty
Dictionary.reference.com
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
Summary
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
Summary of Risk and Uncertainty
• Risk comes from uncertainty in the design, development, production and
operations processes
• If no uncertainty there is no risk…reducing uncertainty reduces risk
• Very high level of uncertainty in exploration systems due to lack of
development and operations experience base
• Sources of uncertainty include:
– Uncertainty in performance, safety, cost, and schedule models
– Uncertainty/changes in customer requirements
– Uncertainty in integration effects on performance
– Uncertainty in manufacturing variation/tolerances
– Uncertainty in test results
– Uncertain operating environments (temperature, pressure, acoustics, etc.)
– Uncertain responses to operating environments/failure modes
– Uncertain component/subsystem/system life
– Potential human errors in design, development, production or operation
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
Summary of Margin and Risk
• Judicious use of MARGIN is the most important risk mitigation strategy!
– Includes cost, schedule, and performance/weight margins
– Use of margin is necessary but not sufficient for risk management
– Still need to identify and mitigate root causes of risk
• Increasing margin decreases risk, but at the expense of other FOMs
– May decrease performance/payload or increase cost at some point
– Eventually reaches diminishing returns in customer value proposition
– Finding the right “balance” between margin/risk and other FOMs is, once
again, a multi-attribute decision problem
• How do I know how much increasing margin decreases risk?
– Through probabilistic analysis using historical data regression or expert
elicitation coupled with Monte Carlo simulation
• How do I know how much increasing margin affects other FOMs?
– Through integrated systems analysis capability or expert elicitation
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
•
•
•
•
•
•
•
•
•
•
Summary of Design Under Uncertainty
The level of uncertainty modeled in the design process depends on
the nature of problem, available resources, and criticality of decision
Risk identification, analysis, planning, tracking and control require
many decisions that require formal methods such as expert elicitation
Judicious use of performance, cost and schedule margin is the most
important risk mitigation strategy in dealing with uncertainty
Risk leveling prevents resources from being focused on risks that do
not have a significant relative effect on the system
A set of complete, independent, and well-defined Figures of Merit are
essential for good design decisions
Multi-Attribute Utility Theory provides the most analytically sound and
comprehensive process for design decision making
An integrated systems analysis capability is essential to good design
Sensitivity analysis enables better decisions by testing assumptions
Probabilistic analysis enables better designs by quantifying risk
Monte Carlo Analysis and Response Surface Methods are key tools
that enable efficient methods for probabilistic design
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
Probability
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
Probability and Random Events
• A basic underlying assumption of probability theory is that
it deals with random events
• A random event is one in which the conditions are such
that each member of the population, N, has an equal
chance of being chosen
• A special and precise system of language and notation is
used in probability theory
• Two events, A and B, are said to be independent if the
occurrence of either one has no effect on the occurrence
of the other
• Two events that have no elements in common are said to
be mutually exclusive events
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
Errors and Samples
• The act of making any type of experimental observation
involves two types of errors:
– Systematic errors (which exert a nonrandom basis)
– Experimental, or random, errors
• When a large number of observations are made from a
random sample, a method is needed to characterize the
data
– Histograms
– Frequency Distribution
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
•
•
Probability Distribution
A probability frequency distribution is a characterization of the possible
values that a random variable may assume along with the probability of
assuming these values.
The probability function has the following characteristics
0  f(xi)  1; S f(xi) = 1
•
A probability density function, f(x), is characterized by the probabilities of
various outcomes of continuous random variables. Probabilities are defined
over intervals computed as the area under the density function between x1
and x2.
f(x)
P(X > 7.3) = .33
6
•
7
8
x
A cumulative distribution function specifies the probability that a random
variable X will assume a value less than or equal to a specified value, x
denoted as P(X  1).
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
Measures of Central Tendency and
Dispersion
• A probability frequency distribution can be described with
numbers that indicate the central location of the
distribution and how the observations are spread out from
the central location (dispersion)
– Arithmetic mean, or average
– Median
– Mode
– Variance
– Standard Deviation
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
Types of Distributions –
Normal Distribution
Normal Distribution
Standard Normal (m = 0, s = 1)
f(x) = (1/(s(2p)0.5))exp(-0.5((x-m)/s)2)
•
Normal Distributions
– Very important in sampling because of the Central Limit Theorem
– Many physical measurements follow the symmetrical, bell-shaped curve
of the normal, or Gaussian, frequency distribution
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
Types of Distributions –
Weibull Distribution
Weibull Distribution for q = 1
and various values of m
m = Shape Parameter
q = Scale Parameter
f(x) = ((m/q)/(x/q)m-1)exp(-(x/q)m), x > 0
•
Weibull Distribution
– Widely used for many engineering problems because of its versatility,
since many random variables follow a bounded, nonsymmetrical
distribution, such as fatigue life of components
– Used to include “infant mortality” in component life modeling (“bathtub”)
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
Types of Distributions –
Gamma Distribution
Gamma Distribution for h = 3
and various values of l
h = Shape Parameter
l = Scale Parameter
∞
f(x) = (lhxh-1e-lx)/(0 xh-1e-xdx), x > 0, l > 0, h > 0
•
Gamma Distribution
– Used to describe random variables that are bounded at one end
– Measures time required for total of h independent events to take place if
events occur at a constant rate of l
– Used to model failures and in queuing theory
– Chi-square and exponential distributions are special cases
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
Types of Distributions –
Exponential Distribution
Exponential Distribution for
l = 1/q, where l = Failure
Rate
1/q
f(x) = (1/q)e-x/q, x > 0
•
Exponential Distribution
– Measures time required for first event to take place if events occur at a
constant rate of l (widely used to measure time to failure)
– Special case of the gamma distribution for η = 1
– Special case of the Weibull distribution for m =1 and x0 = 0
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
Engineering Statistics –
Sampling Distributions
• The central problem in statistics is relating the population
and the samples that are drawn from it
• This problem is viewed from two perspectives:
– What does the population tell us about the behavior of
the samples?
– What does a sample or series of samples tell us about
the population form which the sample came?
• Central Limit Theorem tells us that:
– If a sample size is sufficiently large, the mean of a
random sample from a population has a sampling
distribution that is approximately normal, regardless of
the shape of the relative frequency distribution of the
target population
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
Probability Sampling Analysis
Example
Sample Times (in Seconds) to Inspect Test Devices for Calibration
12.8 15.6 13.5 15.7 15.3 15.2 20.1 14.2 12.9 14.0
16.9 14.3 15.5 14.6 13.0 14.7 19.0 13.0 11.3 14.2
14.5 14.8 14.2 13.0 13.1 12.5 16.1 19.1 16.7 13.2
15.0 12.7 13.6 13.3 13.2 14.7 12.9 13.1 17.3 15.4
17.9 13.0 14.3 14.2 15.7 15.6 13.0 13.9 14.2 16.0
12.9 13.1 13.3 12.3 13.1 13.6 13.2 18.5 13.2 13.7
12.6 14.4 14.5 13.9 17.0 13.7 12.7 16.8 13.3 14.7
14.2 13.0 14.6 14.0 12.9 14.7 12.8 12.0 14.2 12.8
13.7 15.2 14.8 13.0 11.7 12.2 13.3 13.8 14.2 14.3
14.7 12.6 18.9 14.3 14.4 15.5 16.8 17.0 13.2 12.9
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
Probability Sampling Analysis
Example
Histogram for the frequency distribution
of inspection times.
Frequency polygon for the
frequency distribution of
inspection times.
Suggested shape of smoothed frequency
curve for the entire population of
inspection times.
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
Probability Sampling Analysis Example
Probability
Distribution
Function
90%
50%
Cumulative
Distribution
Function
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
Sampling Distributions and
Statistical Intervals
•
Distribution of Sample Means (t Distribution)
•
Distribution of Sample Variances (c2 and F Distributions)
•
Determination of Confidence Intervals
– Confidence Interval containing m with probability 1-a:
– Where 1-a is given by:
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
•
Statistical Tests of Hypotheses
The statistical decision-making process can be put on a rational,
systematic basis by considering various statistically based
hypotheses
– Null hypothesis Ho: μ = μo
– Alternative hypothesis H1: μ < μo
•
Example: Use to test if mean of sample meets minimum
acceptable value
– Type I error, it was acceptable but we concluded it was not
– Type II error, it was not acceptable but we concluded it was (oops!)
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
Functions of Random Variables
• Functions can be as simple or as complicated as desired
• Random variables can be independent or correlated.
Some closed form solutions exist for addition and
subtraction of random variables.
• For closed form solutions, information on the input
random variables distribution is used to describe the
distribution of the output random variable given certain
conditions.
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
•
•
•
•
Addition of 2 RV’s
Let’s start with a simple case.
Uniform Distribution
Y1 = X1 + X2
Assumptions:
– X1 ~ U(0,1)
– X2 ~ U(0,1)
• Can you speculate on the distribution of Y1?
– Shape
– Mean
– Upper and lower bounds?
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
Addition of 2 RV’s
• 10,000 Monte Carlo runs
• Theory
– Bounds: [0.0, 2.0]
– Mean: 1.0
• Empirical data
– Bounds: [0.01 , 1.9]
– Mean: 1.0
• Looks like a triangular distribution. Do you understand
mathematically why it makes sense that it is?
• Monte Carlo randomly selects points from the distribution
and operates on them
• Named after casino by Los Alamos scientists in 1947
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
Addition of 3 RV’s
• Y2 = X1 + X2 + X3
• Assumptions:
– X1 ~ U(0,1)
– X2 ~ U(0,1)
– X3 ~ U(0,1)
• Can you speculate on the distribution of Y2?
– Shape
– Mean
– Upper and lower bounds?
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
Addition of 3 RV’s
• 10,000 Monte Carlo runs
• Theory
– Bounds: [0.0, 3.0]
– Mean: 1.5
• Empirical data
– Bounds: [0.12 , 2.88]
– Mean: 1.49
• No longer looks like a triangular distribution, but its not
quite a normal distribution either. Its simply bell-shaped.
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
Addition of 5 RV’s
• Y3 = X1 + X2 + X3 + X4 + X5
• Assumptions:
– X1 , X2 , X3 , X4 , X5 ~ U(0,1)
• Can you speculate on the distribution of Y2?
– Shape
– Mean
– Upper and lower bounds?
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
Addition of 5 RV’s
• 10,000 Monte Carlo runs
• Theory
– Bounds: [0.0, 5.0]
– Mean: 2.5
• Empirical data
– Bounds: [0.38 , 4.62]
– Mean: 2.49
• Looks much more like a normal.
• What is happening to the bounds of the empirical data
with respect to the theoretical data?
• What would happen with infinity Xi ~ U( 0,1)
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
Observations
• This was a simple case:
– Uniform distribution is the simplest one.
– Distributions are symmetric
– All distributions were equal
– Distribution limits (0, 1) make theoretical estimation
easier.
– Addition of RV’s is very intuitive
• Things to vary:
– Number of Monte Carlo runs
– Number AND type of RV’s
– Ranges, means and other parameters of the RV’s
– The actual function of the RV’s
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
The Normal Distribution
• De Moivre developed the normal distribution as an
approximation to the binomial distribution
• Used by Laplace in 1783 to study measurement errors
• Used by Gauss in 1809 in the analysis of astronomical
data
• Normal distributions have many convenient properties, so
random variables with unknown distributions are often
assumed to be normal
• Normal distribution is often a good approximation due to
a result known as the Central Limit Theorem
• Many common attributes such as test scores, height, etc.,
follow roughly normal distributions, with few members at
the high and low ends and many in the middle
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
• X1~N(0,1)
• Y1 = X1 + X2
m = -0.01
s = 1.43
s2 = 2.04
Adding 2 Normal RV’s
X2~N(0,1)
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
• X1~N(0,1)
• Y2 = X1 +X3
m = 3.01
s = 2.26
s2 = 5.12
Adding 2 Normal RV’s
X3~N(3,2)
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
• X3~N(3,2)
• Y3 = X3 +X4
m = 0.00
s = 2.25
s2 = 5.04
Adding 2 Normal RV’s
X4~N(-3,1)
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
• X1~N(0,1)
• Y2 = X1 - X3
m = -2.96
s = 2.23
s2 = 4.96
Subtracting 2 Normal RV’s
X3~N(3,2)
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
More Complex Functions
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
More Complex Functions
• Y1 = X1 + X2 + X3 + X4 + X5
• Entire range is from -1.22 to 15.21
• Mean = 6.26 Std. Dev.= 2.12 Kurt. = 3.13
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
Continuous Risk Management
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
Continuous Risk Management Process
Make Decisions Under Uncertainty at Every Step in Process!
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
Continuous Risk Management Process
Make Decisions Under Uncertainty at Every Step in Process!
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
Risk Identification Decisions:
Knowing the Unknown
• “…As we know, there are known knowns; there are things we know we
know. We also know there are known unknowns; that is to say we
know there are some things we do not know. But there are also
unknown unknowns -- the ones we don't know we don't know."
-- Don Rumsfeld
Ref: EAI-632
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
Risk Identification Decisions:
Identify Risk Early
Early Risk Identification Enables Good Design Decisions.
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
Risk Identification Decisions:
How Do I Identify Risks?
• Study historic sources of performance, safety, cost, and schedule risks
– NASA/DoD/Industry “lessons learned” databases (e.g., LLIS, REDSTAR)
– CAIB Report and other event investigations
• Systematically parse project/system WBS looking for risk drivers:
– New or adapted technology/designs
– Significant design challenges due to complexity or high level of integration
– Harsh or new operational environments
– Optimistic design assumptions and inadequate design margins
– Inadequate testing
– Historic root sources of unreliability for your mission/system (RoSA)
• Systematically examine Risk Breakdown Structure from ESMD Risk
Management Plan, Section 4.2.5, for issues, or other “check list”
• Systematically examine mission timeline and major events (e.g., EDL,
rendezvous, deployments) for potential failures (perform PRA).
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
Risk Identification Decisions:
How Do I Identify Risks?
• Systematically examine project schedule, comparing allocations to
historic norms (especially software and integrated test and evaluation.
• Systematically examine project cost estimates, comparing allocations
to historic norms.
• Systematically examine project staffing plan and labor estimates,
comparing allocations to historic norms
• Systematically examine all margins (e.g., mass, power, Isp), factors of
safety, flight performance reserves, manufacturing tolerances, etc. and
compare to historic norms.
• Have manufacturing and operations personnel participate in and
systematically examine all aspects of the system design and concept
of operations to identify potential risks.
• Perform concurrent design, create environment that fosters open
communication, and listen to all team members.
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
Continuous Risk Management Process
Early Risk Identification Enables Good Design Decisions.
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
Risk Analysis Decisions:
How Do I Assess and Prioritize Risks?
Risk Exposure = Probability x Impact
High
– Impact against what?
> FOMs!
Medium
– This is multi-attribute
decision problem
– Level of analysis depends on
resources and importance
> From pros/cons to MAUT
> From expert judgment to
quantitative analysis
– Need integrated systems
analysis capability
Low
Probability of
Occurrence
• How do you decide level of
impacts for prioritization?
Low
Medium
Impact if Occurs
High
• How do you decide level of
probability for prioritization?
– Experience/Databases
– Expert Elicitation
– QRA/PRA
Risk Analysis Decisions:
Probability Assessment
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
Probability
Rating
Ordinal
Value
Description
Very Low
1
Qualitative: Very unlikely to occur, management not required in most cases. Strong controls in place.
Quantitative: P< 10-5 (for risks with primary impact on Safety) or P<5% (for risks with primary
impact on Cost, Schedule, or Performance)
Low
2
Qualitative: Not likely to occur, management not required in all cases. Controls have minor
limitations/uncertainties.
Quantitative: 10-5 <P< 10-4 (for risks with primary impact on Safety) or 5%<P<10% (for risks with
primary impact on Cost, Schedule, or Performance)
Moderate
3
Qualitative: May occur, management required in some cases. Controls exist with some uncertainties.
Quantitative: 10-4 <P< 10-3 (for risks with primary impact on Safety) or 10%<P<33% (for risks with
primary impact on Cost, Schedule, or Performance)
High
4
Qualitative: Highly likely to occur, most cases require management attention. Controls have
significant uncertainties.
Quantitative: 10-3 <P< 10-2 (for risks with primary impact on Safety) or 33%<P<50% (for risks with
primary impact on Cost, Schedule, or Performance)
Very High
5
Qualitative: Nearly certain to occur, requires immediate management attention. Controls have little
or no effect.
Quantitative: 10-2 <P< 10-1 (for risks with primary impact on Safety) or 50%<P<100% (for risks with
primary impact on Cost, Schedule, or Performance)
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
Risk Analysis Decisions:
Impact Assessment
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
Risk Analysis Decisions:
Impact Assessment
Daniel Guggenheim
School of
Aerospace Engineering
Example: ELP Top Project Risks
Georgia Institute
Of Technology
August 10, 2006
Likelihood
Performance
Schedule
Cost
Safety
1
1154
Launch Vehicle Operability (J. Reuter)
Perf
1128
3,
1113
5,
6
1151
1158
7
2
1118
Ability for CLV to Meet Performance
Requirements (J. Reuter)
Perf
1114
11
4
9
1156
3
1128
J2X Development Schedule (J. Snoddy)
Sch
1116
12
1152
10
4
1113
Requirements Maturation (J. Reuter)
Sch
5
1155
Enhanced Flight Termination System (R. Burt)
6
1151
Human Space Flight Development Summary (A.
Priskos)
Sch
7
1158
Fault Tolerance Requirements (J. Reuter)
Sch
8
1156
Vehicle Controllability (J. Reuter)
Perf
1159
Inability to meet Earth Departure Stage (EDS)
loiter time requirements (P. Sumrall)
Perf
10
1152
Ability of Heritage Hardware to Meet New CLV
Requirements (R. Burt)
Cost/S
ch
11
1114
Transition Between CLV and SSP (D. Dumbacher)
Cost/S
ch
1116
Engineering Tools, Models and Processes (N.
Otte)
1154
1,2
1118
4
1159
2
Risk Title (Owner)
Risk
Type
1155
5
3
Rank
IRMA
ID No.
8
1
1
2
3
4
Consequences
Cost/S
ch
5
p – Top Directorate Risk (TDR)
r – Proposed Top Director Risk (P-TDR)
■ – Top Program Risk (TPR)
o – Proposed Top Program Risk (P-TPR)
– Top Project Risk (TProjR)
9
– Proposed Top Project Risk (P-TProjR)
q – Top Element Risk (TER)
s – Proposed Top Element Risk (P-TER)
12
Perf
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
Example: CX IRMA Risk: 1118 Summary Report
Open Date: 7/10/2006
Status as of 10/30/2006
ECD: 1/1/2007
Risk Title: Ability for CLV to Meet Performance Requirements
Escalation Level: TPR
Risk Rank: 2
Owning WBS Element:
ARES_I_VEH_INT
Risk Owner: James Reuter
Risk Statement:
Given the history of vehicle and payload growth; there is a possibility that the inability to
maintain the performance and margins needed to meet performance requirements.
Children - ARM # 1596, 1563, 1358, 1099, 1606
5 - Likelihood
Context:
(Imported from ARM Risk 1006) The CLV may not be able to meet mass and
performance requirements. These requirements are not yet well defined, other technical
requirements may impact this further.
Flights Affected:
Status:
10/4/2006 The Performance Enhancement Team (PET) has began a trades option
analysis study in order to mitigate this risk. The study results will provide the best
mitigation strategy to buy down this risk.
WBS Element Affected:
CLV
Consequence(s)
0 - Safety
4 - Performance
0 - Schedule
3 - Cost
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
Continuous Risk Management Process
Make Decisions Under Uncertainty at Every Step in Process!
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
Risk Planning Decisions: How Do I
Decide Best Approach to Mitigate Risks?
• Examine approaches to reduce BOTH the probability and impact of
identified risk
• Determine potential impact on any FOMs if risk occurs (e.g. payload)
• Systematically look at mitigation options for offsetting effects on FOMs
– Systematically examine all other design variables/assumptions, requirements,
or other control parameters that affect FOM (e.g., influence diagram)
– Must also look at effects of employing options on OTHER FOMs
– Select option(s) that offset impact on desired FOM with minimal impact on
other FOMs – including risk
– This is a multi-attribute decision problem!
• Also examine options to reduce probability that risk will occur
– Systematically examine all events in schedule/mission timeline or other
assumptions that affect probability (e.g., PRA, IMS, influence diagram)
– Again, must also look at effects of employing options on ALL FOMs
• Scope level of effort/methods to resources and criticality
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
– Given the history of vehicle and payload growth, the concern is the
inability to maintain the performance and margins needed to meet
payload requirements and affordability goals.
•
Ares I configuration evolved from 4-Segment, SSME to 5-Segment
Booster, J2-X (January 2006)
– Received challenge by Cx to provide additional payload capability
– Developed weight allocation challenges to elements to meet
performance requirements while retaining performance margins
•
L
I
ELO #2 Risk: Ares I Ability to Meet Performance Requirements
Likelihood
•
Risk Planning Decisions:
ELO Performance Risk Mitigation
Consequences
L = Lunar
I = ISS
“DAC-1” conducted for SRD requirements feasibility and design maturation
– Weight allocations not yet met
– Payload performance met only by use of performance margin
•
Performance Enhancement Team (PET) established to reduce or mitigate Ares I
Performance Risk
– Tasked to identify and evaluate candidate design refinements geared to meet or
exceed payload requirements without significant impact to system safety, cost,
schedule, operability
– Provide recommendations to the Project to support the DAC-2 configuration decision
– Identify threats of successfully meeting technical and programmatic requirements with
the reference configuration
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
Risk Planning Decisions:
ELO Performance Risk Mitigation
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
Continuous Risk Management Process
Make Decisions Under Uncertainty at Every Step in Process!
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
Risk Tracking and Control Decisions:
How Do I Track and Control Risks?
• Continue risk identification, assessment/prioritization, and mitigation
planning process – Continuous Risk Management
• Keep and update prioritized list and assess progress at reducing top
risks on regular basis in database (e.g., ARM)
• Decide to reduce or increase risk exposure score as necessary using
processes discussed above
• Continually track progress at meeting requirements through allocated/
decomposed TPMs and integrated/validated systems analysis tools
• Continually track readiness of critical technologies through TPMs
• Use systems engineering database (e.g. Cradle) to link risks to TPMs
and requirements
• Use logic-linked integrated master schedule
• Use earned value management system to evaluate cost vs. budget
• Use expert elicitation approaches to evaluate progress (see below)
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
Risk Tracking and Control Decisions:
Sample Technical Performance Measures
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
Risk Tracking and Control Decisions:
Sample TPM Tracking Approaches
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
Risk Tracking and Control Decisions:
Expert Elicitation Process (UAH/SAIC)
Technical Approach Overview for Technology Project
Tech Assessment
Guidelines
From the
Program
Form Independent
Expert Risk
Assessment
Teams
Define
Risk Assessment
Process and
Provide SW Tool
Technology
Project Data
Released to
Expert Team
Establish Risk
Assessment
Criteria and
Collect Data
Technology
Project Personnel
Input to
Team Discussions
Expert Team
Reaffirms
TPMs &
Reviews/Discusses
Available Data
Teleconferencing
System
Expert Team Provides
Collaborative
Risk Assessment
Using ITRACS©
• Process Expert Input
• Expert Team Reviews
Data
ITRACS©
Internet Accessible
Software
Report
Results
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
Risk Tracking and Control Decisions:
Expert Elicitation Process (UAH/SAIC)
Mechanics of the Process
• Select Technology to be assessed
• Select Technical or Programmatic Risk Metric to be assessed
Assessing the selected TPM or Programmatic Risk Metric:
Given the available information and data on the Technology Development, and considering all
the risk assessment criteria, what numerical interval is most likely to contain the outcome to
be achieved for this metric? And what is the relative likelihood of the other potential outcome
intervals compared to the most likely interval?
Metric Interval
20 to 25 units
Most Likely
Relative Likelihood
5% as likely as 35 to 40
25 to 30
25% as likely as 35 to 40
30 to 35
75% as likely as 35 to 40
35 to 40
100% (most likely interval)
40 to 45
10% as likely as 35 to 40
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
Risk Tracking and Control Decisions:
Expert Elicitation Process (UAH/SAIC)
Mechanics of the Process
ITRACS combines all the individual evaluators’ inputs to produce a
normalized collaborative probability distribution:
Metric Interval
20 to 25 units
25 to 30
30 to 35
35 to 40
40 to 45
Probability Distribution
.024
.095
.357
.476
.048
1.000
The collaborative probability distribution coupled with the metric goal, is
used to calculate the estimated risk of not achieving the development goal:
Metric Goal
< 38
.6
.4
Risk Area (24%)
.2
0
20
25
30
35
Metric Value
40
45
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
Risk Tracking and Control Decisions:
Expert Elicitation Process (UAH/SAIC)
• Perform initial assessment at project start (or during prioritization)
• Use process to perform project audits at scheduled review periods
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
Risk Tracking and Control Decisions:
Expert Elicitation Process (UAH/SAIC)
Probability of Success
Assumption: The Low to High range contains
100% of the possible values of the metric.
Expected Value – Mean or
average value of the
estimated probability
distribution. It is the value
of the metric expected by
the evaluators
Expected Value Deviation –
Deviation of the EV from the
goal, calculated as follows:
Absolute Value: EV – Goal
Goal
A minus sign in front of the
calculated value indicates that
the EV is worse than the goal.
SAIC
ITRACS
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
Steps in Expert Elicitation Processes
(General)
EPA Expert Elicitation Task Force White Paper – January 2009
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
Expert Elicitation Processes
(Cooke Approach)
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
Uncertainty and Margin in
Design
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
Margin and Risk
• Judicious use of MARGIN is the most important risk mitigation strategy!
– Includes cost, schedule, and performance/weight margins
– Use of margin is necessary but not sufficient for risk management
– Still need to identify and mitigate root causes of risk
• Increasing margin decreases risk, but at the expense of other FOMs
– May decrease performance/payload or increase cost at some point
– Eventually reaches diminishing returns in customer value proposition
– Finding the right “balance” between margin/risk and other FOMs is, once
again, a multi-attribute decision problem
• How do I know how much increasing margin decreases risk?
– Through probabilistic analysis using historical data regression or expert
elicitation coupled with Monte Carlo simulation
• How do I know how much increasing margin affects other FOMs?
– Through integrated systems analysis capability or expert elicitation
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
Weight/Performance Margins and
Uncertainty
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
Uncertainty, Risk, and Weight
• Weight is a key control variable during system design/development
– Key performance driver (e.g., payload, range)
– Can be traded for other FOMs (e.g., higher factors of safety, more
redundancy for reliability, cheaper but heavier materials for cost)
– Finding the right “balance” between weight margin/risk and other FOMs
is, once again, a multi-attribute decision problem
• Why don’t things weigh what I predicted?
– Inadequate model fidelity and human errors (“I forgots”)
– Weight growth due to integration effects during development
> Brackets, welds, joints, integrated acoustics/vibration/thermal loads
–
–
–
–
–
Manufacturing tolerances and constraints on manufacturability
Ground operations requirements not modeled (access panels, space)
Uncertainty/inadequate modeling of operational environment
Modifications to balance other FOMs (cost, safety, risk)
Changing requirements
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
Determining Weight Margins
• How do I mitigate risk of weight growth?
– Gather as much relevant historical data as possible to improve models
– Use high-fidelity models to capture integrated weights and loads
– Include manufacturing and operations personnel as an integral part of the
design team and LISTEN to them
– Gather as much data as possible on operational environment
– Find the right “balance” between weight margin/risk and other FOMs up
front through integrated systems analysis
– Spend adequate time up front defining requirements and DON’T change
– THEN, provide adequate weight margins!
• How do I decide on adequate weight margins?
– Depends on level of analysis/modeling used to derive weight prediction
– Gather as much relevant historical weight and growth data as possible
– Include margin allocations for contingency for non-modeled items, weight
growth during development, uncertainty in operating environments/loads,
manufacturing/operations tolerances, life and factors of safety
– Use probabilistic methods to capture historic knowledge
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
Spacecraft Weight Definition
SPEC MIL-M-38310A
MAXIMUM
Limit
• Verified Uncertainty
• Manufacturing Variation
• Allowance for adverse
conditions
• Criteria Changes
Weight
Increment
Nominal
• Growth
Target
• Contingency
• Estimates
• Parametric Studies
Based on Assumed
Design Criteria
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
Mass Margin Definition (JPL)
Payload allocation
Available fro Launch Vehicle
Launch Vehicle Margin (may be zero)
Spacecraft
Wet (Gross) Mass
allocation
Propellant(s) –
Sized for Spacecraft
Dry mass allocation
Spacecraft
Mass
(normalized)
Spacecraft
Dry Mass Allocation
Spacecraft Mass Margin
Spacecraft Dry Mass
Current Best Estimate
Dry mass
Current Best Estimate
taking into account
everything known
Daniel Guggenheim
School of
Aerospace Engineering
Mass Properties Control (JPL)*
Georgia Institute *Design, Verification/Validation and Operations Principles for Flight Systems (D-17868), Rev. 2
Of Technology
Neil Yarnell, Mar 03, 2003
Mass Growth
Mass
(kg)
Mass Allocation
2%
98% Mass Allocation
10%
20%
Mass Current Best Estimate
30%
Mass Margin Requirement
- Accommodates
mass growth for
knowns and unknowns
Preliminary Missions
and Systems Review
Preliminary
Design
Review
Critical
Design
Review
Launch
“Ample margins enable risk management
- balanced risk management is necessary to enable success.
- prudent to have ample mass and power resources to account for and accommodate
uncertainties and expected growth.
- ample mass and power resources in conjunction with ample funding resources provide flexibility
to resolve developmental and operational issues, and enable timely,
balanced risk management decisions without having to perform time-consuming trade studies
to micro-manage every kg. and watt.”
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
Mass Properties Control (AIAA)*
Structure
Thermal Control
Propulsion
Batteries
Wire Harness
Mechanisms
Instrumentation
Electrical Components
Percent Mass Growth Allowance
18
18
18
20
50
18
50
15
12
12
12
15
30
12
30
15
8
8
8
10
25
8
25
10
4
4
4
5
5
4
5
5
2
2
2
3
3
2
3
3
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
Design Maturity
Estimated
(preliminary sketches)
Layout
(or major modification of
existing hardware)
Pre-Release Drawings
(or minor modification of
existing hardware)
Released Drawings
(calculated value)
Existing Hardware
(acutal mass from another
program)
Actual Mass
(measured flight hardware)
Customer Supplied
Equipment
*Recommended
Practice for Mass Properties Control for Satellites, Missiles,
and Launch Vehicles, AIAA/ANSI R-020A-1999
(AIAA Mass Properties Control for Space Systems, 2006)
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
NASA Spacecraft
Weight Growth History
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
Status Weight/Original Weight
1.5
X-20
1.4
Mercury
X-15
Apollo LM
1.3
1.2
Apollo CSM
1.1
Pre-Phase A
Phase A
Phase B
Phase C
Gemini
1.0
0.9
0
1
2
3
Time, years
4
NASA Design Margins
for Spacecraft
5
6
25-35%
25-35%
20-30%
15-25%
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
Aerospace Vehicle
Weight Growth History
1) Apollo CM
2) Apollo LM
3) ASSET
4) Atlas III
5) Atlas V
6) B-9U
7) Classified Program A
8) Classified Program B
9) Classified Program C
10) Classified Program D
11) Classified Program E
12) Classified Program F
13) Classified Program G
14) Classified Program H
15) Clemintine
16) Gemini
17) H-33
18) L-1011
19) Mercury
Lowest Weight Growth = 7%
Average Weight Growth = 27.5%
Highest Weight Growth = 55%
20) MGS
21) Peacekeeper
22) PILOT
23) PRIME
24) Shuttle ET
25) Shuttle Orbiter
26) Skylab
27) Titan I
28) Titan II SLV
29) Titan III B
30) Titan III C
31) Titan III D/E
32) Titan 34 D
33) Titan IV
34) Viking
35) X-15A-2
36) X-33
37) X-34
38) XB-70A
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
Aerospace Vehicle
Weight Growth History By Category
60
50
Apollo LM
X-33
XB-70
40
Skylab
30
F-102
Concorde
STS Orbiter
Mercury
Apollo CSM
F-111
20
Saturn I S-I
10
0
DC-9
DC-8
DC-10
C-131
Commercial
Aircraft
X-20
Saturn V S-IV
Saturn V S-II
X-37
Gemini
F-101
F-106
Fighters
Launch
Vehicles
Human
In-Space
X-Vehicles
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
Aerospace Vehicle Mass Growth
Cumulative Probability Distribution
80% Probability or less -> 39%
What is your risk tolerance?
Triang(3.6058,
58.138)
70% Probability23.000,
or less -> 34%
X <= 0.00
0.0%
X <= 27.19
60% Probability or less
50.0%-> 30%
1
50% Probability or less -> 28%
Triang(3.6058, 23.000, 58.138)
X <= 10.88
5.0%
6
0.6
X <= 48.35
95.0%
5
Values x 10^-2
Probability
0.8
0.4
4
3
2
1
0
0
10
20
30
40
50
60
Triangular Distribution
(best fit)
0.2
0
0
10
20
30
Mass Growth, Percent
40
50
60
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
Aerospace Vehicle
Weight Growth History
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
Space Shuttle Growth
Phase C/D (1972-1983)
Percent
Wing
Tail
LH Tank
LOX Tank
Body
Gear
0.27
0.14
0.13
0.13
0.03
0.06
TPS
Propulsion
Subsystems
Isp, sec
0.01
0.12
0.50
-2.5
Historical Weight Estimating
Relationship (Wing)
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
100000
y = 3079.7x0.5544
+20%
Wing Weight =
3079a0.554
(1+.20
- )
Weight,
lbs
.17
-17%
Shuttle
H-33, Phase B Shuttle
NAR, Phase B Shuttle
747
C-5
L-1011
737
727-200
707
DC-8
10000
1
10
100
1000
a
(
Design Weight*Maneuver Load*Safety Factor*Structural Span
Root Chord
)
Daniel Guggenheim
School of
Aerospace Engineering
Weight Uncertainty Models
Georgia Institute
Of Technology
Uniform(-0.17000, 0.20000)
X <= -0.15150
5.0%
3
X <= 0.18150
95.0%
2.5
2
1.5
1
0.5
0
-0.2
-0.15
-0.1
-0.05
4
0.05
0.1
0.15
0.2
0.25
Uniform
Normal(0, .113) Trunc(-.4,+inf) Shift=+.015
X <= -0.17066
5.0%
0
X <= 0.20088
95.0%
Triang(-.17, 0, .2)
6
3.5
X <= -0.11392
5.0%
X <= 0.13917
95.0%
5
3
4
2.5
3
2
1.5
2
1
1
0.5
0
-0.5
-0.4
-0.3
-0.2
-0.1
0
0.1
Normal
0.2
0.3
0.4
0
-0.2
-0.15
-0.1
-0.05
0
0.05
0.1
Triangular
0.15
0.2
0.25
Weight Uncertainty Model –
Triangular Distribution
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
Triang(-.17, 0, .2)
6
X <= -0.11392
5.0%
X <= 0.13917
95.0%
5
4
3
2
1
0
-0.2
-0.15
-0.1
-0.05
0
0.05
0.1
0.15
0.2
0.25
Daniel Guggenheim
School of
Aerospace Engineering
Monte Carlo Simulation
Georgia Institute
Of Technology
Apply Uncertainty
Wing Weight = 3079a0.554 (1+d)
.
.
.
Triang(-.17, 0, .2)
6
Engine
X <= -0.11392
5.0%
X <= 0.13917
95.0%
5
4
3
Triang(-.17, 0, .2)
LOX Tankl
X <= -0.11392
5.0%
2
6
X <= 0.13917
95.0%
1
5
0
-0.2
-0.15
-0.14
-0.05
0
0.05
0.1
0.15
0.2
0.25
3
Triang(-.17, 0, .2)
LH Tankl
X <= -0.11392
5.0%
2
6
X <= 0.13917
95.0%
1
5
0
-0.2
-0.15
-0.14
-0.05
0
0.05
0.1
0.15
0.2
0.25
3
Triang(-.17, 0, .2)
X <= -0.11392
5.0%
2
6
1
Conduct Experiment
X <= 0.13917
95.0%
Tail
5
0
-0.2
-0.15
-0.14
-0.05
0
0.05
0.1
0.15
0.2
0.25
3
Wing
Triang(-.17, 0, .2)
X <= -0.11392
5.0%
2
6
X <= 0.13917
95.0%
1
5
0
-0.2
-0.15
-0.14
-0.05
0
0.05
0.1
0.15
0.2
0.25
3
Output
Distribution
2
1
0
-0.2
-0.15
-0.1
-0.05
0
0.05
0.1
0.15
0.2
0.25
No
Randomly Pick
Weight Uncertainty
N iterations?
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
Weight Probability Distribution
Dry Weight = 339Klbs ± 25% with 90% Confidence
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
Weight Uncertainty Impacts
100%
80%
Cumulative
Probability 60%
Mean = 340Klbs
40%
95% = 426Klbs
20%
0%
250000
300000
350000
Dry Weight, lbs
400000
450000
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
Probabilistic Weight Tracking –
High Speed Research Program
Relative MTOW Weight
1.4
Assessment Baseline
1.2
5-95th Percentile
Uncertainty Band
Benchmark
1.0
0.8
96
97
98
99
Year
00
01
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
Performance Margins Other Than Weight
Flight Performance Reserves
Specific Impulse Margin
Mixture Ratio Bias
Maximum Temperature Margin
Acoustic/Vibration Margins
Maximum Pressure (Yield and Burst) Margin
Maximum Loiter Time
Launch Window/Availability Margin
Flight Control Margins
Power Margins
Delta-V Margins
Payload Margin
Tank Ullage Margin
Boundary Layer Transition Margins
Etc.
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
Cost Margins and Uncertainty
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
Methods of Developing Cost Estimates
• “Bottoms-Up” Detailed Engineering Build-up
– The separate elements are identified in great detail and summed
into the total cost.
– Very complex for new systems since costs of development and
production are unknown
• “Top-Down” Parametric or Statistical
– Regression analysis is used to establish relations between cost
and initial parameters of the system, e.g. weight, size, speed,
power, SLOC, etc.
2
Cost component = c x1Exponent1 xExponent
2
where xi are the parameters, c and the exponents are determined
by regression of historical data.
– Used in conceptual design
• Analogy
– Future costs of a new project are based on costs of old projects
with allowances for cost escalation and complexity differences
based on simple multiplication factors.
SCost, new = S(x * Cost, old)
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
NASA/Air Force Cost Model (NAFCOM)
• Parametric cost model based on
122 NASA and Air Force space
flight hardware projects
– Launch Vehicles
– Robotic Satellites
– Human-Rated Spacecraft
– Space Shuttle
• Recent updates based on
benchmarking activity with
contractors, internal assessment
• NAFCOM customers
– MSFC, NASA HQ, IPAO, other NASA
centers
– NAFCOM is used by over 800 civil
servants and government contractors
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
NAFCOM CER Complexity Modeling
• Complexity Generator CER’s
– Multi-variable equations based on sophisticated statistical
analysis of the NAFCOM data base
– Identified 73 key technical and programmatic cost drivers,
such as
> Funding availability
> Risk management
> Integration complexity
> Pre-Development study
> New design
> Weight
> Structural efficiency
> Output Power
> Number of Transmitters
> Stabilization type
> Etc.
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
Modeling Cost Risk With CERs
Combined Cost
Modeling and Technical
Uncertainty
Cost = a + bXc
Cost Modeling
Uncertainty
Cost
Estimate
Historical data point
$
Cost estimating relationship
Technical Uncertainty
Standard percent error bounds
Cost Driver (Weight)
Input
variable
Daniel Guggenheim
School of
Aerospace Engineering
Cost Cumulative Distribution
Function (CDF)
Georgia Institute
Of Technology
95th percentile
$184M
100%
80th percentile
$146M
90%
Cumulative Probability
80%
70%
50th percentile
$115M
60%
Mean
$120M
50%
40%
30%
20%
10%
0%
55
65
75
85
95
105
115
125
135
145
155
165
175
185
195
50% probability of cost coming in at or below $115M
45% probability of cost coming in between $115M and $184M
20% probability of cost exceeding $146M
5% probability of cost exceeding $184M
205
215
225
$M
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
Operations Cost Risk – Hidden Costs
• “Direct” (Most Visible) Work Drives Massive
(and Least Visible) Technical &
Administrative Support Infrastructure
• Example: Direct Unplanned Repair Activity
Drives Ops Support Infra, Logistics,
Sustaining Engineering, SR&QA and Flight
Certification
STS Budget "Pyramid"
(FY 1994 Access to Space Study)
Generic
Operations Function
Elem. Receipt & Accept.
Direct (Visible) Work
“Tip of the Iceberg”
+
Indirect (Hidden)
+
Support (Hidden)
Recurring Ops $$s
Total
$M
FY94
Total
(%)
1.4
0.04%
Landing/Recovery
19.6
0.58%
Veh Assy & Integ
27.1
0.81%
Launch
56.8
1.69%
Offline Payload/Crew
75.9
2.26%
Turnaround
107.3
3.19%
Vehicle Depot Maint.
139.0
4.14%
Traffic/Flight Control
199.4
5.93%
Operations Support Infra
360.5
10.73%
Concept-Uniq Logistics
886.4
26.38%
Trans Sys Ops Plan'g & Mgmnt
1487.0
44.25%
Total ($M FY94)
3360.4
100.00%
<10%
~20%
~70%
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
Life Cycle Cost Risk
Life Cycle Cost Gets Locked In Early
using only Systems Engineering Decomposition
Daniel Guggenheim
School of
Aerospace Engineering
Requirements Cost Risk
Georgia Institute
Of Technology
Target Cost Overrun, Percent
200
GRO78
OMV
160
120
80
40
0
IRAS
Gali
PAY NOW
OR
PAY LATER
TDRSS
HST
TETH
GOES I-M
LAND76
CEN
EDO (recent start)
MARS
ACTS
ERB77
COBE
STS
CHA.REC
LAND78
GRO82
ERB80
SEASAT
UARS
VOYAGER
EUVE/EP
DE
Ulysses
SMM PIONVEN
IUE
0
5
10
15
HEAO
ISEE
20
Requirements Cost/Program Cost, percent
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
Requirements
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
Schedule Margins and Uncertainty
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
Uncertainty, Risk, and Schedule
• Why is it never on time?
–
–
–
–
–
–
“I forgots”
Unaccounted-for interdependencies and temporal linkages
Test failures
Hardware/software integration
Requirements changes
Programmatic, organizational, and funding issues
• How do I reduce schedule uncertainty and risk?
– Gather as much relevant historical relevant schedule data as possible
and use to anchor bottoms-up predictions
– Include integration, test, manufacturing and operations personnel in
schedule development and LISTEN to them
– Use logic-linked, integrated master schedule software (e.g., Primavera)
– Focus on critical path and top events that could get on critical path
– Perform probabilistic analysis using historical data or expert elicitation
– Provide adequate schedule margin based on probabilistic data
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
NASA Program Schedule Durations
From Red Star Database
Spacelab - Manned
Shuttle Orbiter - Manned
External Tank
S-II - Launch Vehicle
Skylab Airlock - Manned
Mars Observer - Unmanned
Apollo CSM - Manned
S-IVB - Launch Vehicle
PDR
PDR
CDR
CDR
DDTE
First Flight
Apollo LM - Manned
Viking Orbiter - Unmanned
Magellan - Planetary
Viking Lander - Planetary
Voyager - Unmanned
Centaur-G' - Launch Vehicle
Skylab Workshop - Manned
Gemini - Manned
Mars Exploration Rover
0
20
40
60
80
Months
100
120
140
All Programs
Mean = 6.3 years
InvGauss(5.4174, 18.7886) Shift=+0.88261
0.25
0.20
0.15
0.10
0.05
<
90.0%
2.98
5.0%
11.91
14
12
10
8
6
0.00
4
Georgia Institute
Of Technology
NASA Programs/Projects Duration
Probability Distribution
2
Daniel Guggenheim
School of
Aerospace Engineering
>
NASA Programs/Projects Duration
Probability Distributions
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
Manned Programs
Mean = 9.1
years
ExtValue(8.1224,
1.7630)
Unmanned Programs
Mean =1.5663)
5.3Shift=+2.3128
years
Gamma(1.9244,
0.35
0.30
0.30
0.25
0.25
0.20
0.20
0.15
0.15
0.10
0.10
0.05
0.05
0.00
< 5.0%
6.188
90.0%
>
13.359
90.0%
2.82
5.0%
9.55
14
12
10
8
6
4
2
13
12
11
10
9
8
7
6
5
0.00
>
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
Perform Probabilistic Critical Path
Analysis on Logic-Linked IMS
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
Cost and Schedule Interactions
• Program decision makers need understanding of how uncertainties in
costs and schedule interact
• Might choose a high risk schedule to meet a hard cost target
• Might be willing to have higher costs to ensure meeting a launch date
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
Capturing Cost and
Schedule Uncertainties
• Difference between conditional median cost of ($107.8M) given a
schedule of 53 months and conditional median cost ($87.4M) given a
“high-risk” schedule of 43 months is over $20M
• This could be very significant to a decision maker who wished to
trade cost, schedule, and risk
• Use joint probability models to analyze cost-schedule interactions
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
Technology Risk Mitigation
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
Technology Risk Mitigation Approach
• Select Technologies that are Evolutionary -- Not Revolutionary
– No Major Breakthroughs Required
• Define Low-Risk Back-Ups for Each Technology Project
– Includes Fall-Back and “Fall-Up” Positions (e.g., RLV Composite Tanks)
• Mature Key Technologies to TRL Levels 6 or 7 Before ATP Decision
• Define TPMs for Each Technology Task and Use to Track Progress
• Conduct Risk/Progress Evaluations at Major Technology Development
Milestone Reviews
– Track Technology Development Progress Through Changes to TPMs
– Evaluate Impact of Technology Progress on System Requirements
– Evaluate Technology Development Risk and Take Corrective Actions
> Develop Detailed Risk Mitigation Plans
> Introduce Back-up Technologies/Approaches as Needed
> Reallocate Funding as Required
• Tools Exist to Facilitate the Process (e.g., Active Risk Manager)
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
Risk Leveling
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
Risk Leveling
• How much should we try to reduce a given risk? How safe is safe
enough? Is a human life priceless?
• In design, one requirement/constraint should not be a significantly
larger driver than others
– Question requirement…show cost/benefit of relaxing requirement to
Decision Maker
– Good design has multiple simultaneous driving requirements/constraints
– Process of “requirements leveling”
• Same is true in risk analysis…must perform “risk leveling”
– Don’t let one or two risk sources dominate or go unaddressed
– Don’t spend scarce resources trying to reduce one risk to a lower level or
order-of-magnitude than others
– Make it “safe enough” and no safer
• How do we know when the risks are “leveled”
– Must have integrated systems analysis capability to model and asses risks
– Can perform probabilistic risk analysis (PRA)
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
ESAS Mission Mode
Loss of Crew FOM Results
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
Sources of Loss of Crew Risk for
ESAS Lunar Mission
Radiation Shielding Design Approach
Prior to ESAS
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
Organ Dose 4X’s 1972-SPE
1
4 X Aug 72
4 X Sep 89
0.75
Current LEO Limit
Gy-Eq (BFO)
Based on
Aluminum
Vehicle
0.5
0
0
5
10
15
Added Poly Shield Amount, g/cm^2
LSAM + Poly 5
g/cm2
Skin (Gy-Eq)
5.49
5.78
4.05
0.86
0.91
0.67
Eye (Gy-Eq)
4.79
5.05
3.56
0.83
0.88
0.65
BFO (Gy-Eq)
0.86
0.91
0.67
0.24
0.26
0.20
Effective Dose (Sv)
1.08
1.14
0.84
0.28
0.29
0.23
Organ Dose 4X’s 1989-SPE
0.25
Aluminum LSAM
Aluminum LSAM
LSAM + Poly 5
g/cm2
Skin (Gy-Eq)
0.91
0.96
0.72
0.29
0.31
0.26
Eye (Gy-Eq)
0.82
0.86
0.66
0.29
0.31
0.26
BFO (Gy-Eq)
0.29
0.30
0.26
0.17
0.18
0.16
Effective Dose (Sv)
0.30
0.32
0.26
0.17
0.18
0.16
1
Composite LSAM
Skin (Gy-Eq)
4.23
4.45
3.09
0.74
0.78
0.56
Eye (Gy-Eq)
3.81
4.01
2.80
0.72
0.76
0.55
BFO (Gy-Eq)
0.73
0.77
0.56
0.21
0.23
0.17
Effective Dose (Sv)
0.90
0.95
0.69
0.25
0.26
0.20
Organ Dose 4X’s 1989-SPE
Composite LSAM
4 X Aug 72
LSAM + Poly 5
g/cm2
LSAM + Poly 5 g/cm2
Skin (Gy-Eq)
0.73
0.77
0.58
0.27
0.28
0.24
Eye (Gy-Eq)
0.68
0.72
0.55
0.27
0.28
0.24
BFO (Gy-Eq)
0.27
0.28
0.23
0.16
0.17
0.15
Effective Dose (Sv)
0.27
0.29
0.24
0.16
0.17
0.15
4 X Sep 89
Current LEO Limit
Based on
Graphite (60%)Epoxy (40%)
Vehicle of same
mass
0.75
Gy-Eq (BFO)
Organ Dose 4X’s 1972-SPE
0.5
0.25
0
0
5
10
Added Poly Shield Amount, g/cm^2
15
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
Solar particle event design environments
basis and considerations:
•
•
Radiation limits outside LEO do not currently
exist - being developed by NCRP and the
CHMO
LEO Career Limit
–
•
99% event for mortality risks (acute or
chronic risks)
• The August ’72 event is generally
accepted as the benchmark solar
particle event in measurable history.
• One’s confidence of not exceeding the
’72 event fluence level above 30 MeV
on a one year mission near the solar
maximum is about 97%.
• To achieve 99.5% confidence level
above 30 MeV one must assume a
fluence of 4 times the August ’72
event.
Probability of 3% additional risk of lifetime
lethal cancer within a 95% confidence interval
LEO Blood-Forming Organs (BFO) Shortterm Limits
–
–
30-day limit - 25 cGy-Eq
Annual limit – 50 cGy-Eq
%Risk of Fatal Cancer
–
20
4x1972 Event for EX-CEV Design
Female 45-yr
Risk Limit
16
12
8
4
0
0
2
4
6
8
Polyethylene Augmentation Shield,
10
g/cm2
Females 45-yr (no prior missions)
20
%Risk of Fatal Cancer
•
ESAS Analysis Cycle 2
Radiation Risk Assessment
16
EX CEV baseline
CEV with 5 g/cm2 poly shield
Risk Limit
12
8
4
0
2
N x 1972 Event
4
ESAS Analysis Cycle 2
CEV Acute and Late Risks
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
•
•
•
Estimated probability of an SPE that could cause debilitation+ (1.5X Aug 1972
event)
Estimated probability of catastrophic event (4X Aug 1972 event)
Recommend maximum of 2 g/cm2 CEV shielding based upon risk leveling for a 16
day maximum mission (0.04 year), 0.005 P (exceeding ‘72 levels), and risk
probabilities given in table below
(99.5% confidence of not exceeding the ’72 event fluence level
above 30 MeV on a one year mission near the solar maximum)
Aluminum Vehicle, 4X 1972 SPE
HDPE Depth (g/cm2) % Acute Death*
% Sickness
% REID**
0
9.5
54.0
9.1 [3.2,17.3]
2
(0.02)
(2.9)
3.8 [1.3,10.5]
5
0
0
1.5 [0.5,4.3]
(99.5% confidence of not exceeding the ’72 event fluence level
Graphite-Epoxy Vehicle, 4X 1972 SPE above 30 MeV on a one year mission near the solar maximum)
HDPE Depth (g/cm2)
% Acute Death*
% Sickness
% REID**
0
3.0
34.4
7.6 [2.7,16.7]
2
(0.01)
(1.9)
3.4 [1.2,9.6]
5
0
0
1.4 [0.4,3.9]
• Death at 60-days with minimal medical treatment
** Risk of Cancer death for 45-yr Females
+Debilitating event identified as dose that would cause vomiting within 2 days in 50% of total population
ESAS Analysis Cycle 3 SPE Risks versus
Probability of SPE Occurrence in a 9-day Mission
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
CEV with 0 g/cm2 HDPE
Nx1972
Event
F(>30 MeV)
10
%Probability for
9 day mission
Acute Death
Acute Sickness
Career Limit
Violation
30-Day Limit
Violation
4X
2x10
0.02
<1 %
<5%
Yes
Yes
3X
1.5x1010
0.04
0
<1%
Yes
Yes
0.07
0
0
No (95% yes)
Yes
0.17
0
0
No (95% yes)
No
1.00
0
0
No (95% yes)
No
2X
1X
1% Event
10
1x10
5x109
1.5x10
9
CEV with 1 g/cm2 HDPE
Nx1972
Event
F(>30 MeV)
4X
2x1010
10
%Probability for
9 day mission
Acute Death
Acute Sickness
Career Limit
Violation
30-Day Limit
Violation
0.02
0
0
Yes
Yes
3X
1.5x10
0.04
0
0
No (95% yes)
Yes
2X
10
0.07
0
0
No (95% yes)
Yes
9
0.17
0
0
No
No
1.00
0
0
No
No
30-Day Limit
Violation
Yes
Yes
No
No
No
1X
1% Event
1x10
5x10
1.5x109
CEV with 2 g/cm2 HDPE
Nx1972
Event
4X
3X
2X
1X
1% Event
F(>30 MeV)
10
2x10
10
1.5x10
1x1010
9
5x10
9
1.5x10
%Probability for
9 day mission
Acute Death
Acute Sickness
Career Limit
Violation
0.02
0.04
0.07
0.17
1.00
0
0
0
0
0
0
0
0
0
0
No (95% yes)
No (95% yes)
No (95% yes)
No
No
Better than 1
in 600
Probability
of NO SPE
Radiation
effects for
a 9-day
mission
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
ESAS Analysis Cycle 3: 1.5 Launch Solution
Sensitivity to CEV Radiation Protection
72
30
28
27
Injected Mass
68
26
RECOMMENDATION:
Eliminate supplemental
radiation shielding
66.5
66
25
24
CEV Mass
64
23
23.4
22
62
21
21.0
60
20
0
1
2
3
4
CEV Supplemental Radiation Protection (g/cm2 )
5
CEV Mass (t)
TLI Injected Mass (t)
70
29
70.2
Injected Mass Sensitivity:
~740 kg per g/cm2
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
Probabilistic Risk
Assessment
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
Probabilistic Risk Assessment
The human mind cannot grasp the causes of phenomena in
the aggregate. But the need to find these causes is inherent in
man’s soul. And the human intellect, without investigating the
multiplicity and complexity of the conditions of phenomena,
any one of which taken separately may seem to be the cause,
snatches at the first, the most intelligible approximation to a
cause, and says: “This is the cause!”
Leo Tolstoy,
War and Peace
Scenario Development is used in risk analysis to facilitate the
systematic search for the causes of risk.
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
Probabilistic Risk Assessment (PRA)
• PRA provides thorough, quantitative scenario-based approach to
assessing the probability that a risk will occur and its consequences
• Used on Space Shuttle (Fragola) and ISS (Futron)
• Flow between process steps
– Master Logic Diagram
> Identifies how hazards are controlled
– Functional Event Sequence Diagram
> Shows how the system responds to off normal events
– Event Trees
> Inductively models that represent the way pivotal events can combine in
response to specific initiating events
– Fault Trees
> Deductive models that generate logical combinations of failures that can
cause a specific high level pivotal event
• Each technique addresses a part of the risk assessment problem
• In combination, allow analyst to accurately and completely represents
majority of risk
Joseph Fragola – NIA Risk-Based Design Short Course
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
Probabilistic Risk Assessment
Joseph Fragola – NIA Risk-Based Design Short Course
Daniel Guggenheim
School of
Aerospace Engineering
Probabilistic Risk Assessment
Georgia Institute
Of Technology
Master Logic Diagram
Event Tree
Loss-of -Vehicle
LOV due to
Orbiter Failure
LOV due to
Solid Rocket
Booster Failure
LOV due to
Main Engine
Failure
Loss of
Propulsion
Loss of
Pressure in the
MCC
High Mixture
Ratio in the
Fuel Preburner
Loss of
Containment
Loss of
Hydrogen Flow
High Mixture Ratio
Not Detected
Loss of Gross
Hydrogen
Flow
Yes
Lower flowrate
triggers active
computer control
sequence
Controller Increases
Oxidizer Flow to Fuel
Preburner
Yes
High Mixture Ratio in
the Fuel Preburner
No
Failure in Channel A
High Mixture Ratio in
the Both Preburners
High Mixture Ratio
Detected
Yes
No
Failure in Channel B
S/D
Erroneous Signal in
Channel A
Logic Control Failure
in Channel A
Erroneous Signal in
Channel B
Logic Control Failure
in Channel B
LOV
Harness Failure in
Channel A
Functional Event Sequence Diagram
Joseph Fragola – NIA Risk-Based Design Short Course
Harness Failure in
Channel B
Fault Tree
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
Probabilistic Risk Assessment
• Steps in PRA Process
Joseph Fragola – NIA Risk-Based Design Short Course
Daniel Guggenheim
School of
Aerospace Engineering
Georgia Institute
Of Technology
Probabilistic Risk Assessment
• Where does data come from?
Joseph Fragola – NIA Risk-Based Design Short Course