Transcript power point

Taming Development and Data
Chaos in Military Logistics:
The ICIS Experience With Lightweight Languages
Gary Kratkiewicz
BBN Technologies
Geoffrey Knauth
BAE Systems
ICIS Background
•
•
•
•
Web-based software system
Analyzes future combat and humanitarian operations
Measures ability of Defense Logistics Agency to satisfy
wartime requirements for following items:
•
•
•
Fuel
Food
Replacement items (repair parts, clothing, medical items, etc.)
Identifies potential problems
•
•
•
When will I run out of stock?
What should I buy today for tomorrows needs?
Do I have adequate sources & transportation?
LL2 - November 9, 2002 - Gary Kratkiewicz & Geoffrey Knauth
2
Accomplishments
•
•
•
•
•
Early Web-based DOD system
Saved over $100 million
•
By computing reduced inventory needs for fuel and food
Adopted as part of DOD doctrine
Months of calculations reduced to hours
•
•
What-if capability leads to better decisions
Short analyses expanded role to crisis action
Used in military operations to:
•
•
Assess readiness
Compute operational requirements
LL2 - November 9, 2002 - Gary Kratkiewicz & Geoffrey Knauth
3
Development Notes
•
•
•
•
•
Started in 1994
Grew from small prototype to large system in 4
phases
User interface moved to Web in stages from
1996 to 1999
Approximately a third of code is in Lightweight
Languages
Significant data issues
LL2 - November 9, 2002 - Gary Kratkiewicz & Geoffrey Knauth
4
Data, Data, Data
•
•
Large amounts of data
• 800-1000 input files; 200 different formats
• Gigabytes of input and output per analysis
• Hundreds of analyses “active”
Data management issues
• Received from many different agencies
• Regular and irregular updates
• Meandering formats
• Duplication between analyses:
•
•
Computations
Intermediate & output files
LL2 - November 9, 2002 - Gary Kratkiewicz & Geoffrey Knauth
5
1. Prototype ICIS – 1994/1995
•
•
•
•
Original prototypes completely coded in Tcl/Tk
Vague, rapidly changing, and conflicting
requirements
Less than 1 MB of data
Ease of developing/tailoring lightweight
prototypes greatly contributed to initial success
LL2 - November 9, 2002 - Gary Kratkiewicz & Geoffrey Knauth
6
2. Production ICIS – 1995/1996
•
•
•
•
Lots of data: approaching 1 GB
Addition of (mostly) well-specified
computational modules
Real use with classified data in mid-1996
Mix of languages:
•
•
•
Tcl/Tk for GUI and glue for module execution
Perl for data preprocessing
C/C++ for computational modules
LL2 - November 9, 2002 - Gary Kratkiewicz & Geoffrey Knauth
7
3. Web ICIS – 1996-1999
•
•
•
•
Address remote access issues
Moved to Web in stages
On server:
•
•
•
•
Tcl/Tk – glue for module execution
Perl – data preprocessing
C/C++ and Java – computational modules
Perl and C++ – dynamic generation of HTML and
JavaScript
In browser:
•
•
•
HTML – static GUI elements
JavaScript – form-based GUI elements
Java AWT applets – interactive GUI elements
LL2 - November 9, 2002 - Gary Kratkiewicz & Geoffrey Knauth
8
4. Redesigned Web ICIS – 2001/2002
• Improve GUI and better manage data
•
•
•
Data approaching 1 TB
On server:
•
•
•
JScheme – managing data & module execution (sophisticated
“glue”)
Perl – data preprocessing
C/C++ and Java – computational modules
In browser:
•
•
•
•
•
HTML – static GUI elements
JavaScript (much less) – simple GUI elements
Java AWT applets – more complex GUI elements
Java Swing applets – most complex GUI elements
JScheme for understanding scenario
LL2 - November 9, 2002 - Gary Kratkiewicz & Geoffrey Knauth
9
Source Code Evolution
90000
80000
70000
Tcl/Tk
C++
C
Perl
SQL
HTML
Java
Scheme
Lines of Code
60000
50000
40000
30000
20000
10000
0
Prototype 1994
Production 1996
Some Web 1997
LL2 - November 9, 2002 - Gary Kratkiewicz & Geoffrey Knauth
Full Web 1999
Redesign 2002
10
Summary of Lightweight Language
Use in ICIS
• Rapid prototyping
• LL replaced by HL: completely for computations,
•
•
•
•
•
partially for GUI
Highly iterative, innovative design & development
• LL used in Data Set Manager, patent pending
Processing of meandering data
Gluing together heavyweight modules
• LL replaced with more sophisticated LL
Building little languages
Test code
LL2 - November 9, 2002 - Gary Kratkiewicz & Geoffrey Knauth
11
JScheme — Application Flexibility
• Interpreter for debugging
• Scripts for component functionality
• Can be a server for execution management of
multiple tasks
• Example implementation:
BBN’s Data Set Manager
LL2 - November 9, 2002 - Gary Kratkiewicz & Geoffrey Knauth
12
User Asks ICIS Questions
Example: “How much Fuel do I need?”
•
•
•
•
•
User interaction with ICIS is web-based
User defines composite scenario
User specifies results of interest
ICIS runs the assessment
• Numerous modules might be run
• It might take a while…
Results appear in browser
LL2 - November 9, 2002 - Gary Kratkiewicz & Geoffrey Knauth
13
An Example Small Assessment
LL2 - November 9, 2002 - Gary Kratkiewicz & Geoffrey Knauth
14
Problems
• Obvious approach of using shell
•
•
scripts and files doesn’t cut it here
Too much data
Many computations recomputed
needlessly
LL2 - November 9, 2002 - Gary Kratkiewicz & Geoffrey Knauth
15
A Full ICIS Assessment
Does Many Computations
The readable version of this
graph takes up a large wall.
LL2 - November 9, 2002 - Gary Kratkiewicz & Geoffrey Knauth
16
Data Set Manager
•
•
•
•
•
•
Data is described by features, not file names
A module spec describes
• Inputs and outputs of a module
• Process relationships with other modules
Modules can be implemented in any language
A module spec is applied to a composite
scenario to produce an execution script
The script is executed
All computations are memoized
LL2 - November 9, 2002 - Gary Kratkiewicz & Geoffrey Knauth
17
Execution Management Needs
Dynamic Capabilities
Data Set Manager takes full
advantage of Scheme
LL2 - November 9, 2002 - Gary Kratkiewicz & Geoffrey Knauth
18
Implementing the Data Set Manager
in JScheme was fairly easy because:
• Quasiquote was used to go from
•
•
module spec to execution script
A full language was required to
write module specs
Executing the script was just
walking a tree structure
LL2 - November 9, 2002 - Gary Kratkiewicz & Geoffrey Knauth
19
Murphy Strikes
Start
Goal
Mistrial!
LL2 - November 9, 2002 - Gary Kratkiewicz & Geoffrey Knauth
20
MVC: JScheme = M+C, Java = V
Draw graph from
knowledge of
assessment state
Generate
User Options
Show Details
Try Again
LL2 - November 9, 2002 - Gary Kratkiewicz & Geoffrey Knauth
21
Data Set Manager — Conclusion
•
•
•
•
•
Core implementation is 3804 lines of JScheme code
Supported by 287 lines of Perl, 102 lines of SQL
ICIS Module specs add another 1853 lines of JScheme
Current APIs for other languages are:
• 688 lines of C
• 496 lines of Perl
• 356 lines of Java (1 class)
• 120 lines of JScheme
BBN has applied for two patents
LL2 - November 9, 2002 - Gary Kratkiewicz & Geoffrey Knauth
22