Transcript Operations Analysis Course Introduction

Operations Research and the Role of
Probability and Statistics in Military Analysis
21 February 2014
Dr. Rafael E. Matos
WBB, Inc.
(703) 448-6081 x108
Terminology: OR and OA
• Operations Research (OR)
– The academic discipline comprising a range of tools for analyzing
operations for purposes of improving or optimizing various business or
functional processes
• Operations Analysis (OA)
– The discipline of applying OR tools to specific subsets of functional
processes, in your case military operations
2
How OA Can Help You
• Provide analytic tools in helping to assess budget priorities
• Oversee Directed Studies
– Keeping tabs on outsourced studies
• Manage your own study
– How to run your own study
• Review studies from other sources
– Knowing strengths, weaknesses, good studies and bad ones
• Synthesize an analytic position based on other studies
– Putting the pieces together
3
Operations Analysis
Operations Analysis “classical” definition: “A scientific method of providing
executive departments with a quantitative basis for decisions regarding the
operations under their control”
•
Methods of Operations Research, Morse & Kimball, 1951
Decision
Objective Inputs
Array of facts: Quantitative
comparison of the meaningful
elements of the problem
Subjective Inputs
Largely provided by the decisionmaker: judgment and experience
factors
“The OA Domain”
Accurately defining the Problem is critical
4
The Military focus has changed
9-11-2001
PRE – 911
POST – 911
Major Regional Conflict/
Major Theater War
2 MRC – 2 MTW
CP
PK
PE
HA/
DR
C-Piracy
CT
Peace–keeping/
Enforcement
Humanitarian
Assistance/
Disaster Relief
C-Terrorism
Strategy and capabilities required for Spectrum of
Conflict were subsets of 2MRC/MTW force structure
5
Strategy and capabilities required for post-9/11
environment are not subsets
of MCO force structure
When you change the way you look at things,
the things you look at change…
6
An Example of Data
Anscombe's Quartet comprises 4 data sets of 11 points each:
I
II
III
x
y
x
y
x
y
x
y
10
8
13
9
11
14
6
4
12
7
5
8.04
6.95
7.58
8.81
8.33
9.96
7.24
4.26
10.84
4.82
5.68
10
8
13
9
11
14
6
4
12
7
5
9.14
8.14
8.74
8.77
9.26
8.10
6.13
3.10
9.13
7.26
4.74
10
8
13
9
11
14
6
4
12
7
5
7.46
6.77
12.74
7.11
7.81
8.84
6.08
5.39
8.15
6.42
5.73
8
8
8
8
8
8
8
19
8
8
8
6.58
5.76
7.71
8.84
8.47
7.04
5.25
12.50
5.56
7.91
6.89
For all four:
• Mean of the x values = 9.0
• Mean of the y values = 7.5
• Equation of the least-squared regression line is: y = 0.5x + 3
• Sums of squared errors (about the mean) = 110.0
• Regression sums of squared errors
(variance accounted for by x) = 27.5
• Residual sums of squared errors (about the regression line)
= 3.75
• Correlation coefficient = 0.82
• Coefficient of determination = 0.67
(F.J. Anscombe, "Graphs in Statistical Analysis," American Statistician, 27 [February 1973], 17-21)
7
IV
What does this mean ?
• Analysis of the data show that it
is similar
• But is it ?
• Let’s look at the data when it
is plotted
Completely Different Data Sets
Using line plots reveals the differences among the data sets
I
III
y = 0.50x + 3.00
R² = 0.67
16
16
12
12
8
8
4
4
0
0
0
5
10
15
20
0
5
II
16
10
IV
y = 0.50x + 3.00
R² = 0.67
16
12
12
8
8
4
4
0
15
20
y = 0.50x + 3.00
R² = 0.67
0
0
5
10
15
20
0
5
10
This quartet is used as an example of the importance of looking at your data before analyzing it in
Edward Tufte's book, The Visual Display of Quantitative Information.
8
y = 0.50x + 3.00
R² = 0.67
15
20
History of OA
•
•
•
Tanker torpedoed off US coast - 1942
First wide use of OA methods was during the
Battle of the Atlantic in World War II
Anti-Submarine Warfare Operations Research
Group (ASWORG) established as part of US
Atlantic Fleet Headquarters to evaluate: “How to
protect a large number of cargo ships against
enemy submarines with limited escorts”
Study of the ASW problem provided crucial
insights regarding:
– Convoy size - number of ships in each
– Search techniques - how to best employ sensors
to maximize the probability of detecting
submarines
– Screening - where to position escorts to put them
in the best position to detect and counter
submarine attacks
Atlantic convoy at sea
Convoy Protection Problem
•
Assumption: each ship in a convoy
takes up a certain area that has to be
defended by the escorts (1 mi2)
•
•
5
•
For a 25-ship convoy, vulnerable
perimeter is 5X4=20 miles
If an escort can protect 2.5 miles, this
convoy requires 8 escorts
2 similar convoys would require 16
escorts
•
•
A 49-ship convoy has a vulnerable
perimeter of 7X4=28 miles and
requires 11 escorts (11.2)
96% increase in convoy size, but only
40% increase in vulnerable
perimeter: requires 33% fewer
escorts than same number of ships
divided into 2 convoys
7
Application of relatively simple mathematics to a complex warfare problem
10
OA Today
• The use of OA has expanded greatly beyond ASW and
Naval Warfare to other services and the commercial
sector
– Department of Defense - uses OA to address the problem
of which programs to fund in the face of a changing
world with a limited amount of resources
– Transportation Industry - OR tools and techniques are
readily applicable to the problem of optimizing
transportation networks and assets in different areas
with varying conditions
– Many recent business improvement schools of thought
(Total Quality Management, Business Process ReEngineering, Six Sigma) are all based on Operations
Research theory
• Military and civilian businesses leadership increasingly
turning to analytic process to inform decisions
11
Example Naval Warfare OA Problem
• Issue: How many ships should the Navy have in the future?
– What types of ships should they be?
“The Global Concept of Operations requires a fleet of approximately 375 ships that will
increase our striking power from today's 12 carrier battle groups, to 12 Carrier Strike
Groups, 12 Expeditionary Strike Groups, and multiple missile-defense Surface Action
Groups and guided-missile submarines. These groups will operate independently
around the world to counter transnational threats and they will join together to form
Expeditionary Strike Forces—the "gold standard" of naval power—when engaged in
regional conflict.” Sea Power 21
The Navy's report provides few details about how many ships the service would have
to buy each year to implement either the 260- or 325-ship plan--and thus how big a
budget it would need for ship construction.
Some combinations may not be affordable, feasible or mission effective
12
Spectrum of Analysis
Basic Analysis:
Investigative Journalism
• What or Who is being studied?
– Objects or Methods
– Materiel or Doctrine
Who, What, When, Where, Why, and How
Who
What
• When and Where is it being studied?
– Micro or Macro level
– Phenomenological or Campaign
When
Where
Why
How
• How and Why is it being studied?
– Simple evaluation of one
– A choice between two or more
– An optimal mix of several
Who and What
Who or What is going to be analyzed
Military Application:
• What is the mission that needs to be accomplished?
• Can the mission be accomplish with the current
constraints?
• Is there a better way to use available resources to
accomplish the mission?
• If additional resources are applied how will it improve the
ability to accomplish to this mission?
Example Problem Statement:
“How many ships should the Navy have in 2020-2030?”
14
When and Where
Which missions are
important to the
overall campaign ?
Does it matter?
(SO WHAT?)
CAMPAIGN
What functions are
critical in those
missions?
What parameters
are critical to
those functions?
Can you use it?
MISSION
How does it work?
ENGAGEMENT
What is it that does it?
ENGINEERING
Doctrinal studies are usually
conducted at Campaign or
Mission levels, and sometimes at
the Engagement level
15
Technical studies are usually
conducted at Engineering levels
or below, and sometimes at the
Engagement level
Why and How
• Straight Evaluation of one subject against a set of benchmarks
– Grades
– Stock price or profit margin
• Choice between two or more subjects
– Fly-off
– Competition
• Optimization
– Determine the optimal mix of resources to apply to a specific (fixed) situation
• Force Structure
– Trade Studies measure the effect of different levels of capability in one area or
attribute versus another
• Cost / performance trade-offs determine the balance between design alternatives
based on both performance and cost
16
Total Spectrum of Analysis
Who / What
Optimization
Campaign
Doctrine
Campaign
Evaluation
Material
Mission
Evaluation
Doctrine
Mission
Evaluation
Engineering Engagement
Material
Campaign
Evaluation
Mission
Evaluation
Material
Doctrine
Engagement Engagement
Evaluation
Evaluation
Material
Engineering
Evaluation
Doctrine
Engineering
Evaluation
Material
Doctrine
Who / What
17
When / Where
When / Where
Choice
Evaluation
Optimization
Choice
• Regardless of where a
study falls in the who /
what / when / where / how
/ why spectrum, each has
the same characteristics
– Foundation is based on
scientific methods
– Features logical, common
study components and
stages
Foundation of OA
•
Regardless of subject, level or method, all Operations Analysis is based on
scientific methods
•
A more thorough definition:
– “...the application of scientific knowledge toward the solution of problems which occur
in operational activities (in their real environment). Its special technique is to invent a
strategy of control by comparing, measuring, and predicting possible behavior through
a scientific model of [the] activity”
Naval Operations Analysis, USNI Press, 1984
•
•
•
•
•
“Solution of problems” - the study Objective
“Operational activities” - the study Subject
“Real environment” - the study Context
“Comparing” - the study Basis of Comparison
“Measuring” - the study Metrics
18
Breaking a problem
down into these
factors is the first
step in solving it
A Typical OA Study Plan
2. Define
Studies
3. Execute
Studies
This type of plan
is scalable to
small and large
issues
1. Establish Plan
Foundation
Defense
Planning
Guidance
Prioritize Study Areas
Establish Study Spectrum
Develop Basis of Comparison
Select Metrics
May 2001
Define Objective
Generate Data
5a. Refine /
Focus Plan
4. Synthesize
Results
Establish Study Subject
Develop Alternatives
Define Context
Is the Study still
supporting the Objective?
5b. Check
Impact
What Happened?
Did it support the Objective?
19
5. Facilitate
the Decision
Process Data
Generate Information
Review with Decision-Makers
Descriptive Statistical Measures
• Graphical displays provide a better sense of the data by transferring
numerical information into a picture - another way is to use Descriptive
Measures
• Measures of Central Tendency (or location)
– Descriptions on where the middle lies
– Generally attempt to estimate the underlying population mean
– Examples include the mean, median & mode
• Measures of Dispersion (or spread)
– Descriptions on the spread of the data
– Examples include the range, variation, standard deviation & percentile
Central Tendency
looks for the middle
X
XX
20
X
X
X
X
X
XX XX
X
XX X X
Dispersion looks
for the spread
X
XX
X
X
X
X
XX XX
X
XX X X
Interpreting Statistics
• Sampling distribution is obtained by computing
statistics for a large number of samples drawn
from the same population
– Measures of Central Tendency / Location
• Mean
• Median
• Mode
– Measures of Dispersion
•
•
•
•
Range
Variance
Standard Deviation
Percentile
• Once observations have been made and
statistics derived, what can we infer about the
population from which the sample was
collected?
20
Student
Group A
15
AVG = 77
10
5
0
A
A-
B+
B-
C+
C-
D+
D-
F
20
Student
Group B
15
AVG = 77
10
5
0
A
A-
B+
B-
C+
C-
D+
D-
F
20
15
Student
Group C
AVG = 77
10
5
0
A
21
A-
B+
B-
C+
C-
D+
D-
F
Trend Analysis
104%
102%
100%
98%
96%
94%
92%
90%
88%
86%
84%
82%
% of BA filled E5-E9
C1/2 ERB
Trend
Trend
80%
Se
p95
De
c95
M
ar
-9
6
Ju
n96
Se
p96
De
c96
M
ar
-9
7
Ju
n97
Se
p97
De
c97
M
ar
-9
8
Ju
n98
Se
p98
De
c98
M
ar
-9
9
Ju
n99
Se
p99
De
c99
M
ar
-0
0
Ju
n00
Se
p00
De
c00
M
ar
-0
1
Ju
n01
Stock Price over time
C1/C2 Readiness
and Top 5 Manning
Source: Pers-452 (LOOMIS) C1/C2 vs. E5-E9 BA Filled
• An element of Time Series Analysis
• An investigation of data (i.e. system performance) over a
spectrum of one variable – usually time
• Trend Analysis can estimate or predict…
– within the data set (interpolation)
– outside the data set (extrapolation)
Analysis with Regression
• Regression Analysis evaluates the relationship between a
variable of interest (a dependent or response variable) and one
or more independent or predictor variables
– Distance from target and kill probability
– Years of training and promotion rates
– Time and target movement
• Often used to predict the response variable from the knowledge
of the independent variables
– Can evaluate either a linear relationship or more complex one
• Regression methodologies are complicated, but the EXCEL
spreadsheet package can perform both simple and complex
regression analysis quickly
A quick indicator is the coefficient of determination (R2) in the results.
You want this number to be close to one.
23
Regression: Interpolation Example
Missile X
Target Aspect vs. Miss Distance
• Test performance data set for Missile X
• In addition to the miss distance data, the
target aspect at launch time was also
recorded
• The chart shows the relationship between
the target aspect at launch and the
missile miss distance at intercept
Test Target Aspect
Miss
Number
at Launch
Distance
1
5.0
3.0
2
10.0
5.0
3
6.0
3.5
4
9.0
4.0
5
7.0
3.5
6
15.0
10.0
7
20.0
15.0
8
40.0
20.0
9
35.0
21.0
10
30.0
18.0
– For example for Test # 7, target aspect at
launch was 20 degrees and miss distance
was 15 feet
• Is there a mathematical relationship
between these two variables such that for
any given target aspect one can predict
the miss distance?
• What would you say the miss distance
would be for a target aspect of 25
degrees?
Target Aspect (Degrees)
Miss Distance (Feet)
Miss Distance (Ft)
Missile X Performance
25
20
15
10
5
0
0
10
20
30
Target Aspect at Launch (Degrees)
40
50
Regression: Extrapolation
•
Extrapolation estimates or predicts outside the data set
– When time is the independent variable, predicts system performance in the future based on
how it has functioned in the past
•
Validity depends on…
– …the consistency of the original data set
– …the assumption that current conditions will continue outside the data set
•
Regression works well for interpolation, but not necessarily for extrapolation
– R2 value (a measure of how well the line fits the data) only applies within the data set
•
When extrapolating, keep your window short and use some common sense
How far out would
you extrapolate
this data set?
Missile X Performance
Miss Distance (Feet)
25
y = 0.5626x + 0.3426
R² = 0.9477
20
15
10
y = -0.0126x 2 + 1.1116x - 3.5103
R² = 0.9801
5
0
0
10
20
30
Target Aspect at Launch (Degrees)
40
50
Extrapolation Example
• The day before the final launch of the Challenger Space Shuttle,
Morton Thiokol faxed 13 charts to NASA opposing the launch
due to possible O-ring failure.
– In the charts, the following information was scattered throughout and
summarized in the conclusions:
•
•
•
•
O-Ring damage at 53F
O-Ring damage at 75F
Temperature on 1-28-86 launch initially estimated to be 29F - 38F
RECOMMENDATION: O-Ring temperature must be > 53F at launch
– NASA officials recommended reconsideration because:
• Analytical graphics failed to communicate the magnitude of
risk that was in fact present
– Reassessing the situation, Morton Thiokol stated
– The evidence presented by their engineers was inconclusive,
– Cool temperatures were not linked to O-ring problems, and
– They now favored a launch the next day
Example from Visual Explanations, Edward R. Tufte, Graphics Press, 1997
26
Extrapolation Example (Cont.)
Damage Index
11
4
4
2
0
0
0
0
0
0
4
0
4
0
0
0
0
4
0
0
0
0
0
Analysis illustrates risk with launch day
temperature forecast of 26-29 degrees F
O-Ring Damage
12
Linear R2 = 0.4116
10
Damage Index
Temperature
53
57
58
63
66
67
67
67
68
69
70
70
70
70
72
73
75
75
76
76
78
79
81
Polynomial R2 = 0.6286
8
6
4
2
0
-2 0
20
40
60
80
100
Temperature
n
R2
 ( y  yˆ )
i 1
27
2
is the square of the
SSE
2
R

1

 1  i n1
residual errors from the line
SST
 ( yi  y ) 2
How well the line approximates the points
i
Why is basic probability theory important?
– Probability is the basis for statistics, data analysis, hypothesis
testing, sampling, surveys, experimentation, prediction … other
useful tools in the experimentation trade.
– Decision making under uncertainty is largely based on application
of statistical data analysis for probabilistic risk assessment of
decisions …probability provides the theoretical framework to be
certain of how uncertain you are.
Probability theory provides necessary foundations
for the ability to generate statistics
28
General Concepts
• Basic Probability Theory
– What is it?
• That branch of mathematics that is concerned with calculating the
likelihood of outcomes of experiments -- modeling the phenomenon
of chance or randomness
• That way of thinking in which we make inferences from a sample to a
population, and then measure the accuracy of those inferences
Probability of a strike by a tropical storm
Katrina
29
Operational Issues
How many enemy aircraft are
detected in a four hour
period?
Number of Enemy Aircraft
Detected in a four hour
period
Poisson Distribution
How reliable is
my
Air Search
Radar?
Probability that radar will fail
Exponential Distribution
Are there any anti-ship
mines
located in this area?
Mine Location in Chokepoint
Uniform Distribution
30
What are the chances
that
I can destroy an inbound
missile raid with
my overhead CAP?
Inbound Missile Raid
destruction
Binomial Distribution
What is the Pk for my
missiles?
Single Shot missile
kill
Bernoulli Trial
How well does my
Flight deck crew
perform?
Probability that flight deck
crew
will maintain advancement
rates
Normal Distribution
General Concepts
• What are statistics?
– Classical definition: a science of inferring generalities from specific
observations ,i.e., a way of working with numbers to answer questions
about various phenomena
1
• E.g. Analyzing missile test firings to evaluate overall missile performance
• Goals of statistics are to:
– Descriptive: Describe a set of numbers, and
– Inferential: Make accurate inferences about groups based upon
incomplete information
• E.g., Does the type of missile meet key performance parameters
• Making accurate inferences requires groundwork. In order to
reach an informed conclusion, one must:
– Gather data (numerical information); then,
– Organize it (sometimes in a graphic); then,
– Analyze it
Statistics use “observed data” to draw conclusions
about an “unobserved population”
1-Dixon and Massey, Introduction to Statistical Analysis
31
Missile AoA Dispersion Calculations
Let’s look at the spread miss distance dispersion data for our 3
candidate missile systems.
Missile 1
Missile 2
Missile 3
Accuracy of 5m with the spread data?
Accuracy of 10m with spread data?
Least amount of dispersed shots?
Greatest amount of dispersed shots?
Test 1
Test 2
Test 3
Test 4
Test 5
Test 6
Test 7
Test 8
Test 9
Test 10
5.0
5.0
5.0
5.0
5.0
5.0
5.0
5.0
5.0
100.0
10.0
10.0
10.0
10.0
10.0
10.0
10.0
10.0
10.0
10.0
3.0
3.0
3.0
3.0
3.0
10.0
15.0
20.0
20.0
20.0
Variance
Std Deviation
Range
Mean
Median
902.5
30.0
95.0
14.5
5.0
0.0
0.0
0.0
10.0
10.0
63.3
8.0
17.0
10.0
6.5
Miss distances in meters (m)
32
Value Statement
You can build a fairly comprehensive warfighting model
using Probability Distributions and Statistical Analysis
33
Operations Research and the Role of
Probability and Statistics in Military Analysis
21 February 2014
Dr. Rafael E. Matos
WBB, Inc.
(703) 448-6081 x108