Transcript Aug 2001 IJCAI2001 `Lesson Distribution Gap`

Lesson Distribution Gap
David W. Aha
Rosina Weber
Héctor Muñoz-Avila
Leonard A. Breslow
Kalyan Moy Gupta
Navy Center for Applied Research in Artificial Intelligence, Naval Research Laboratory
Booth # 214
Outline
Introduction


Contributions
Context:

lessons learned systems, process, organizations
Lesson distribution gap
How to bridge this gap? Monitored Distribution
Example
Evaluation, Results
Next Steps
Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA
Contributions
Describe lessons learned process
Identify gap in lesson distribution
Propose Monitored Distribution
Test hypothesis in evaluation
Monitored Distribution can improve plan quality
 Plan evaluator

Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA
Knowledge management context
Three types of KM initiatives
knowledge repositories
 knowledge access and transfer
 knowledge environment

From Davenport & Prusak’s (1998): Working Knowledge
Types of knowledge repositories
industry oriented (alert systems, best practices)
 organization oriented (lessons learned systems)
 for example, ..

Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA
government
non-government
Construction Industry Inst.
non-military
military
Honeywell
GM
Hewllet Packard
US
Air Force
Army
int’l
Bechtel Jacobs Company
European Space Agency
Italian (Alenia)
French (CNES)
Japanese (NASDA)
United Nations
Lockheed Martin E. Sys, Inc
Coast Guard
DynMcDermott Petroleum Co.
Xerox
IBM
BestBuy
Joint Forces
Siemens
Marine Corps
Navy
int’l
Canadian Army Lessons Learned Centre
US
Department of Energy: SELLS
NASA (Ames, Goddard)
Lessons learned systems
KNOWLEDGE
ARTIFACTS
Lessons learned systems are repositories of a
knowledge artifact called lessons learned
Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA
Lessons learned definition…
…or organizational lessons, lessons, lessons identified
Definition:
A lesson learned consists of knowledge or understanding
gained by experience. The experience may be positive, as
in a successful test or mission, or negative, as in a mishap
or failure. A lesson must be significant in that it has a real
or assumed impact on operations; valid in that is factually
and technically correct; and applicable in that it identifies a
specific design, process, or decision that reduces or
eliminates the potential for failures and mishaps, or
reinforces a positive result.”
(Secchi et al., 1999)
Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA
Lessons learned process
REUSE
REVISE
RETRIEVE
RETAIN
Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA
Lessons learned representation
indexing elements (case problem)


applicable task
preconditions
reuse elements (case solution)
lesson suggestion
 rationale

Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA
Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA
Lessons learned example
applicable task
Installing custom stereo speakers.
preconditions
The car is the Porsche Boxster.
lesson suggestion
Make sure you distinguish the wires leading to the speakers
from the wires leading to the side airbag.
rationale
Somebody has cut the wrong wire because they look alike
and the airbag went off with explosive force. This means
spending several thousand dollars to replace the airbag in
addition to be a potential hazard.
From article “Learning from Mistakes” about Best Buy
in knowledge management magazine, April 2001.
Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA
Lessons learned process
Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA
Lesson distribution methods
Broadcasting
bulletins, doctrine
Passive
standalone repository
Active casting
list servers,
information gathering tools
Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA
Problems with
lesson distribution methods
 Distribution
is
divorced
from
targeted
organizational processes.
 Users may not know or be reminded of the
repository, as they need to access a standalone
tool to search for lessons.
 Users may not be convinced of the potential
utility of lessons.
 Users may not have the time and skills to retrieve
and interpret textual lessons.
 Users may not be able to apply lessons
successfully.
Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA
Here is the gap
Repository of
lessons learned
Organization’s
members
Organizational
processes
Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA
How to bridge this gap?
Repository of
lessons learned
Organization’s
members
Organizational
processes
Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA
How to bridge this gap?
Repository of
lessons learned
Organization’s
members
Organizational
processes
Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA
How to bridge this gap?
Repository of
lessons learned
Organization’s
members
Organizational
processes
Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA
How to bridge this gap?
Repository of
lessons learned
Organization’s
members
Organizational
processes
Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA
How to bridge this gap?
Repository of
lessons learned
Organization’s
members
Organizational
processes
Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA
How to bridge this gap?
Repository of
lessons learned
Organization’s
members
Organizational
processes
Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA
How to bridge this gap?
Repository of
lessons learned
Organization’s
members
Organizational
processes
Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA
How to bridge this gap?
Repository of
lessons learned
Organization’s
members
Organizational
processes
Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA
How to bridge this gap?
Repository of
lessons learned
Organization’s
members
Organizational
processes
Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA
Monitored distribution
Repository of
lessons learned
Organization’s
members
Organizational
processes
Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA
Monitored distribution
Lesson repository
is in the same
context as
targeted processes
Organization’s
members
Repository of
lessons learned
Organizational
processes
Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA
Problems
&
Solutions
Distribution is divorced from
targeted organizational
processes.
Lessons are distributed to users
in the context of the
organizational processes.
Users need to access a
standalone tool to search for
lessons.
Users don’t need to access a
standalone tool.
Distribute in the same context does not suffice!
Problems
Users may not have the time
and skills to retrieve relevant
lessons.
&
Solutions
No significant additional time or
skills are required.
Users may not be convinced of Users can assess the potential
the potential utility of lessons. utility of lessons easily.
Users may not be able to apply Whenever possible, an ‘apply’
lessons successfully.
button allows the lesson to be
automatically executable.
Intrusive methods may cause
more problems than solutions.
Distribution tightly integrated to
the targeted processes.
Monitored Distribution Characteristics
 Distribution tightly integrated to the targeted processes
so that lessons are distributed
when and where they are needed.
• Represent lessons as cases (knowledge modeling).
• Lessons are indexed by their applicability.
 Additional benefits are:
• Case representation facilitates interpretation.
• Users assess potential utility with lesson rationale.
• Whenever possible, an ‘apply’ button allows the
lesson to be automatically executable.
Noncombatant Evacuation Operations (NEO)
Military operations to evacuate noncombatants
whose lives are in danger and rescue them to a safe haven
Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA
Assembly
Point
Campaign headquarters
Intermediate Staging Base
.
safe haven
NEO site
Example in HICAP
• HICAP is a plan authoring tool suite
• Users interact with HICAP by refining an
HTN (hierarchical task network) through
decompositions
• http://www.aic.nrl.navy.mil/hicap
• Muñoz-Avila et al., 1999
Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA
NEO site
Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA
safe haven
Selecting the Suggested Case…
Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA
Expanding yields…
Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA
And the user is notified of a lesson
RATIONALE:
TYPE: advice
Clandestine SOF should not be used alone
WHY: The enemy might be able to infer that SOF are involved, exposing them.
Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA
After applying the lesson
Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA
Evaluation
Hypothesis

Using lessons will improve plan quality
Methodology
Simulated HICAP users generated NEO
plans with and without lessons
 Plan evaluator implemented plans

Plan total duration
 Plan duration before medical assistance
 Casualties: evacuees, friendly forces, enemies

Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA
Plan evaluator
non-deterministic (100 plans 10 times each)
30 variables: 12 random
e.g., weather, airports
length of plans 18 steps
e.g., transportation mode, supplies, team
size of planning space 3,000,000
13 actual lessons
Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA
Plan implementation


Plans where evacuees were transported by land
modes have an increased chance of being attacked
by enemies.
When an attack happens it increases the number
of casualties among evacuees and friendly forces
(in proportion to # of evacuees).
Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA
Results
*The resulting values are averages
no lessons
with lessons
reduction
39h50
32h48
18 %
duration until
medical assistance*
29h37
24h13
18 %
casualties
among evacuees
11.48
8.69
24 %
casualties among
friendly forces
9.41
6.57
30 %
casualties
among enemies
3.08
3.14
-2 %
NEO plan
total duration*
Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA
Next Steps
Collection tool
Verify methods, reasoning
Integration of informal groups’ and user’s individual features
Evaluation with human subjects (simulated users in HICAP)
and let human subjects decide on applying lessons
Extend MD to other decision support systems and other
knowledge artifacts
Investigate distribution of experiential knowledge with
training knowledge
Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA
David W. Aha
Navy Center for Applied Research in Artificial Intelligence,
Naval Research Laboratory, Washington, DC
Rosina Weber
Department of Computer Science, University of Wyoming
Fall 2001 at Drexel University, PA
Héctor Muñoz-Avila
Department of Computer Science, University of Maryland
Fall 2001 at Leehigh University, PA
Leonard A. Breslow
Navy Center for Applied Research in Artificial Intelligence,
Naval Research Laboratory, Washington, DC
Kalyan Moy Gupta
IIT Industries, AES Division, Alexandria, Virginia
Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA
CBR Cycle and Knowledge Processes
Aamodt & Plaza 1994
Discussion
 Intrusive method requires good precision
 Knowledge representation is costly and so are lives!
 What’s the worth of 35,000 unused lessons?
 Knowledge representation can be also support validation
 Good news: collect lessons into case representation.
Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA
Plan evaluator: example
lesson
applicable task:
Assign security element.
Conditions for applicability:
There are hundreds or more evacuees as to justify a security
effort.
Lesson suggestion:
Recommend that EOD* personnel is utilized in security
element.
Rationale:
Success.
EOD two DET ten personnel were employed in a force
protection role and assisted USS Nassau security teams in
identifying and investigating suspect items brought aboard
by evacuees.
*EXPLOSIVE ORDNANCE DISPOSAL
Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA
Clarification
How is monitored distribution (MD) different from Clippie?
• In MD, case base task is applicability
• MD distributes experiential knowledge collected from
users in similar roles as the potential reuser
• Clippie is activated by single word
• Clippie distributes general instructions/information
Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA
additional lesson
Conditions for applicability:
There are representatives of different branches assigned to
participate.
Lesson suggestion:
Assign representatives of all forces to plan.
Rationale:
Lack
of
representatives
prevent
good
communication causing delays and miscommunication.
Rosina Weber IJCAI01 8 Aug 2001 Seattle, WA