Transcript Performance Evaluation

Using Data Compression to Increase the Bandwith of
Existing Tactical Control System
Malcolm Vant
Deputy Director General at DREO
Chairman of NATO IST
Problems
 Lots of bits in imagery
 Not enough pipe to push imagery to the field
 Visualisation quality is tied to compression
Information Management
Defence Evaluation
& Research Agency
Malvern
Sensors and
Worcestershire
Electronic Sectors
WR14 3PS
UK
Click to add sub-title
Margaret
Varga
Telephone: +1684 895712
Facsimile: +1684 894952
Email:
[email protected]
Background
 Huge volumes of diverse data to be analysed
 Existing systems are outpaced
 Manual examination is still the main approach
 Need for fast, cost effective and reliable semi-automatic or
automatic approaches
 To retrieve, screen, evaluate, correlate, disseminate,
archive and present information at the right time, place and
format
Information Deluge
An Overview of an Information Management Process
Analysis
Interpretation /
Evaluation
Data
Information
Data Mining
Intelligence
Transformation
Data Selection
Data Collection
Pre-processing
Knowledge
Patterns
Transformed
Data
Target Data
Pre-processed
Data
Raw Data
Visualisation
Information Management Process
(MoD)
 Data Collection - Reports (digital/hardcopy), formatted, application oriented
 Data Selection - Information Retrieval/Extraction/Routing
 Pre-Processing - Elimination/reduction of outliers
 Transformation - Conversion into appropriate format for processing
 Pattern Finding - Data mining
 Interpretation/Evaluation of Patterns - Knowledge discovery
 Analysis of Knowledge - Intelligence
 Summarisation
 Information visualisation
Information Management Principles
 Relevance
 Effectiveness and Efficiency
 Responsiveness
 Accessibility
 Custody
 Preservation
 Accountability
Open Source Textual Information Exploitation
(Military Relevance)
 Timely access to relevant, accurate, precise,
comprehensible and credible information (e.g. for decision
making)
 Ability to track what information is available (covert and
overt)
 Ability to define rapidly changing topics of interest from
diverse users
 Zoom or expand over levels of specialised knowledge (i.e.
summary or detailed analysis)
Limitations of Existing Information Retrieval Approaches
 Commercial Off The Shelf (COTS) products are only effective in a few domain
specific applications
 COTS based on user-specified ad hoc keywords, phrases or topics:
–
–
–
subject scope can become too general and unfamiliar
costly (time & money), depends on predefined search definition
irrelevant reports are retrieved and relevant reports may remain hidden
 Full reports are presented
 Duplicate information from same and different sources is presented
 Retrieved reports are ranked by existence and frequency of keywords
- not necessary equivalent to relevancy
Open Source Textual Information Exploitation Research Programmes
Delos - Textual (and speech) information sources
Indexation, Profiling,
Archival &
intelligence Analysis
DERA-OKAPI Demonstrator
Accurate Key
Generation &
Topic spotting
Relevance
Ranking
Redundancy
Elimination
Summarisation
Visualisation
Plumber, TextScape,
Delos
DERA-OKAPI Information Retrieval Demonstrator
DERA-OKAPI Information Retrieval Demonstrator
DERA-OKAPI Information Retrieval Demonstrator
User Interest Oriented Text Summarisation
 Accurate ‘keys’ are used as salient summary indicators of
the news feeds
 Extracted sentences with the occurrence of the accurate keys
 Extract the leading paragraph of the news item as a ‘rough’
summary of the reporting event
 Extract the sentence before and after the ‘key’ sentence reduce dangling problems
 Paragraphs with more than 3 ‘keys’ or more than 70%
relevant sentences
Text Summarization
Amount of text
News Items
Keywords
/phrases
First paragraph
Sentences with
‘keys’
Sentences before and after the sentence
with ‘keys’
Paragraphs with more than 70% of
sentences deemed relevant or > 3 keys
100%
1 - 3%
5 - 10 %
15 - 20%
25 - 30 %
30 - 35%
Salient Keys
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20
I-FOR
CONVINCED
DAYTON
CIVILIAN
CRIMES
MISSION
SERIOUS
INVESTIGATIONS
SITUATION
LIFE
SUCCESS
Nato
WAR
POLICE
STATEMENTS
PROCESS
PROBLEMS
keeping
peace
FORCE
A.1.1Original News
Record 0
weight 1000
Length: 2/~1 pages
<Subject:> BOSNIA/SMITH <L-ONLY> BY NICK SIMEONE /WASHINGTON
<ORGANISATION:> Defence Research Agency
DATE=/4/2/96
TYPE=CORRESPONDENT REPORT
NUMBER=2-195223
TITLE=BOSNIA/SMITH (L-ONLY)
BYLINE=NICK SIMEONE
DATELINE=WASHINGTON
CONTENT=
VOICED AT:
INTRO: THE COMMANDER OF NATO FORCES IN BOSNIA, ADMIRAL LEIGHTON SMITH, SAYS IT IS TOO
EARLY FOR MILITARY COMMANDERS TO RECOMMEND WHETHER U-S PEACE-KEEPING TROOPS
SHOULD REMAIN IN BOSNIA BEYOND THEIR SCHEDULED DEPARTURE IN DECEMBER. THAT’S A
DECISION, HE SAYS , THAT WILL HAVE TO BE MADE LATER THIS YEAR BUT NEARLY FIVE MONTHS
INTO THE PEACE-KEEPING OPERATION, ADMIRAL SMITH IS RAISING CONCERNS ABOUT CIVILIAN
ASPECTS OF THE BOSNIA PEACE AGREEMENT, INCLUDING THE ABILITY OF LOCAL POLICE TO
ACTUALLY ENFORCE LAW. WE HAVE THIS REPORT FROM V-O-A’S NICK SIMEONE.
TEXT: WITH WAR CRIMES INVESTIGATORS STARTING TO LOOK INTO MASS GRAVES, ADMIRAL
SMITH SAYS ENORMOUSLY DIFFICULT TASKS STILL LIE AHEAD AS THE WORLD WATCHES TO SEE
WHETHER BOSNIA’S FORMER WARRING PARTIES WILL BE ABLE TO FULFILL TERMS OF THE PEACE
ACCORD THEY SIGNED TO END THE WAR. // OPT // HE MADE THE STATEMENTS TUESDAY AT A
PENTAGON BRIEFING. //
LATELY, SOME POLITICAL LEADERS IN EUROPE HAVE BEEN PRIVATELY SUGGESTING THE PEACE
PROCESS IN THE FORMER YUGLOSLAVIA IS TOO DELICATE AND THE COUNTRY IS TOO LIKELY TO
SLIP BACK INTO WAR FOR NATO FORCES - PARTICULARLY U-S TROOPS ----- TO CONSIDER PULLING
OUT AT THE END OF THE YEAR. PRESIDENT CLINTON HAS REPEATEDLY SAID THAT U-S
PARTICIPATION IN THE PEAC-KEEPING MISSION IN BOSNIA WIIL ONLY BE FOR ONE YEAR. NATO
COMMANDER SMITH SAYS THE MILITARY, AT THIS STAGE AT LEAST, IS NOT MAKING PREDICTIONS
EITHER WAY.
// SMITH ACT //
THERE ARE SOME PEOPLE IN EUROPE WHO HAVE EXPRESSED AN
OPINION BOTH FOR AND AGAINST I-FOR REMAINING AFTER
DECEMBER. I WILL JUST TELL YOU THAT THE SMART MAN WOULD
GOO LOOK AT THE SITUATION, JUDGE IT AT THE TIME, MAKE A
DECISION AND THEN FROM THAT POINT ON BE GUIDED
ACCORDINGLY.
A.1.1
Leading Paragraph Summarisation
INTRO: THE COMMANDER OF NATO FORCES IN BOSNIA, ADMIRAL
LEIGHTON SMITH, SAYS IT IS TOO EARLY FOR MILITARY COMMANDERS TO
RECOMMEND WHETHER U-S PEACE-KEEPING TROOPS SHOULD REMAIN IN
BOSNIA BEYOND THEIR SCHEDULED DEPARTURE IN DECEMBER. THAT’S A
DECISION, HE SAYS , THAT WILL HAVE TO BE MADE LATER THIS YEAR BUT
NEARLY FIVE MONTHS INTO THE PEACE-KEEPING OPERATION, ADMIRAL
SMITH IS RAISING CONCERNS ABOUT CIVILIAN ASPECTS OF THE BOSNIA
PEACE AGREEMENT, INCLUDING THE ABILITY OF LOCAL POLICE TO
ACTUALLY ENFORCE LAW. WE HAVE THIS REPORT FROM V-O-A’S NICK
SIMEONE.
Salient key-based summary
INTRO: THE COMMANDER OF NATO FORCES IN BOSNIA, ADMIRAL
LEIGHTON SMITH, SAYS IT IS TOO EARLY FOR MILITARY COMMANDERS TO
RECOMMEND WHETHER U-S PEACE-KEEPING TROOPS SHOULD REMAIN IN
BOSNIA BEYOND THEIR SCHEDULED DEPARTURE IN DECEMBER. THAT’S A
DECISION, HE SAYS , THAT WILL HAVE TO BE MADE LATER THIS YEAR BUT
NEARLY FIVE MONTHS INTO THE PEACE-KEEPING OPERATION, ADMIRAL
SMITH IS RAISING CONCERNS ABOUT CIVILIAN ASPECTS OF THE BOSNIA
PEACE AGREEMENT, INCLUDING THE ABILITY OF LOCAL POLICE TO
ACTUALLY ENFORCE LAW. WE HAVE THIS REPORT FROM V-O-A’S NICK
SIMEONE
LATELY, SOME POLITICAL LEADERS IN EUROPE HAVE BEEN PRIVATELY
SUGGESTING THE PEACE PROCESS IN THE FORMER YUGLOSLAVIA IS TOO
DELICATE AND THE COUNTRY IS TOO LIKELY TO SLIP BACK INTO WAR FOR
NATO FORCES - PARTICULARLY U-S TROOPS ----- TO CONSIDER PULLING
OUT AT THE END OF THE YEAR. PRESIDENT CLINTON HAS REPEATEDLY
SAID THAT U-S PARTICIPATION IN THE PEAC-KEEPING MISSION IN BOSNIA
WIIL ONLY BE FOR ONE YEAR. NATO COMMANDER SMITH SAYS THE
MILITARY, AT THIS STAGE AT LEAST, IS NOT MAKING PREDICTIONS EITHER
WAY.
BUT IN EITHER CASE, ADMIRAL SMITH SAID THAT THOUGHT IS BEING
GIVEN TO SCALING BACK THE NATO FORCE, WHICH NOW STANDS AT
ABOUT 60-THOUSAND TROOPS, AS THE OPERATION HEADS INTO THE
SUMMER MONTHS. SO FAR NATO COMMANDERS SEE NO EVIDENCE THAT
THE WARRING PARTIES ARE PREPARING FOR WHAT IN YEARS PAST HAS
BEEN THE START OF A WARM WEATHER, SPRING OFFENSIVE.
HE SAYS A KEY TO THE SUCCESS OF THE BOSNIA PEACE PLAN WILL BE
THE BOSNIAN ELECTIONS, WHICH, ACCORDING TO THE PEACE PLAN, ARE
TO BE HELD IN SEPTEMBER, BY THEN NATO COMMANDERS, AS WELL AS
THE SPONSERS OF THE DAYTON PEACE ACCORD, WILL LIKELY HAVE A
GOOD IDEA WHETHER BOSNIA’S POST-WAR ATMOSPHERE IS FREE OF
THREATS AND IF, IN ADMIRAL SMITH’S WORDS, POLITICIANS CAN GIVE
SPEECHES WITHOUT BEING SHOT. SUCH INDICATIONS COULD INFLUENCE
THE DECISION ON WHETHER COME DECEMBER, U-S TROOPS STAY OR
LEAVE.
(SIGNED)
A.1.1
Simple Summarisation
(Nato/peace/keeping)
Based
on
User’s
Ad-hoc
Keywords
INTRO: THE COMMANDER OF NATO FORCES IN BOSNIA, ADMIRAL
LEIGHTON SMITH, SAYS IT IS TOO EARLY FOR MILITARY COMMANDERS TO
RECOMMEND WHETHER U-S PEACE-KEEPING TROOPS SHOULD REMAIN IN
BOSNIA BEYOND THEIR SCHEDULED DEPARTURE IN DECEMBER.
BUT NEARLY FIVE MONTHS INTO THE PEACE-KEEPING OPERATION,
ADMIRAL SMITH IS RAISING CONCERNS ABOUT CIVILIAN ASPECTS OF THE
BOSNIA PEACE AGREEMENT, INCLUDING THE ABILITY OF LOCAL POLICE
TO
ACTUALLY ENFORCE LAW.
TEXT: WITH WAR CRIMES INVESTIGATORS STARTING TO LOOK INTO MASS
GRAVES, ADMIRAL SMITH SAYS ENORMOUSLY DIFFICULT TASKS STILL
LIE AHEAD AS THE WORLD WATCHES TO SEE WHETHER BOSNIA’S FORMER
WARRING PARTIES WILL BE ABLE TO FULFILL TERMS OF THE PEACE
ACCORD THEY SIGNED TO END THE WAR.
LATELY, SOME POLITICAL LEADERS IN EUROPE HAVE BEEN PRIVATELY
SUGGESTING THE PEACE PROCESS IN THE FORMER YUGLOSLAVIA IS TOO
DELICATE AND THE COUNTRY IS TOOL LIKELY TO SLIP BACK INTO WAR
FOR NATO FORCES - PARTICULARLY U-S TROOPS - TO CONSIDER PULLING
OUT AT THE END OF THE YEAR. PRESIDENT CLINTON HAS REPEATEDLY
SAID THAT U-S PARTICIPATION IN THE PEACE-KEEPING MISSION IN BOSNIA
WILL ONLY BE FOR ONE YEAR.
NATO COMMANDER SMITH SAYS THE MILITARY, AT THIS STAGE AT LEAST,
IS NOT MAKING PREDICTIONS EITHER WAY.
BUT IN EITHER CASE, ADMIRAL SMITH SAID THAT THOUGHT IS BEING
GIVEN TO SCALING BACK THE NATO FORCE, WHICH NOW STANDS AT
ABOUT 60-THOUSAND TROOPS, AS THE OPERATION HEADS INTO THE
SUMMER MONTHS.
SO FAR NATO COMMANDERS SEE NO EVIDENCE THAT THE WARRING
PARTIES ARE PREPARING FOR WHAT IN YEARS PAST HAS BEEN THE START
OF A WARM WEATHER, SPRING OFFENSIVE.
WHAT WE WANT TO BE ABLE TO DO IS TO KEEP THE MILITARY APART AND
IF WE CAN DO THAT WITHOUT A LARGE FORCE OR WITHOUT A FORCE,
THAT REALLY THE GOAL.
Information Visualisation
 Represent both the structure & content of the retrieved
information
 Readily comprehensive manners
 Facilitate high level tasks
– Identification of similar/duplicate information
– Identification of topic trends
– Identification of anomalies
 Provides access to different levels of summary
DERA TextScape
DERA TextScape
DERA TextScape
DERA TextScape
DERA TextScape
DERA TextScape
Evaluation Issues
Retrieval effectiveness:
 Dynamic data sources
 User’s oriented/task oriented
 Key word saliency
Summarisation
 Summaries first
 Followed by full report
Visualisation
 Effective in showing the
– Identification of similar/duplicate information
– Identification of topic trends
– Identification of anomalies
 Inter-relationship? Temporal ?
Event Stream Analysis
 Analysis of large datasets generated during intensive
man/machine interaction
 The analysis of the large log file generated by these interaction is
assisted by DERA’s Plumber
 Plumber - a visual programming tool which enables a new visual
environment for data processing
 Part of a commercial package of DP+MS
Plumber
Main canvas
Built-in
visualisers such
as bar charts
Store results
to disk
Palette of
processing
widgets
Libraries of
pre-written
widgets
Access to external
processing tools
such as Excel and
Matlab
Java version