Transcript elicitExampleUncertainty1 - ITACS | International Technology

International Technology Alliance
In Network & Information Sciences
Analysing the ELICIT factoids
- example of uncertainty
David Mott (ETS, IBM UK)
April 2014
v1.1
ELICIT is…
•
An experimental framework for researching into collective sensemaking by teams
of collaborating humans
– A set of Natural Language sentences providing information about a scenario in a
fictitious geographic region with a number of different countries
– There is a possible attack by a group on a target in one of these countries
– The collaboration of teams must identify, from the information the who, what, where,
and when of the attack
– In different experiments, the teams are given only partial information, and the task is to
communicate and collaborate to do the identification
ELICIT sentences (factoids)
1 The Lion is involved
47 All high value targets of Omegaland are well protected
2 Word has it that an unprotected target is preferred to ensure the likelihood of success (can assume is true)
48 There has been an increase in messages intercepted in Psiland
3 The Lion doesn't operate in Chiland
49 The Lion was born in Tauland
4 The Lion attacks in daylight
50 There is no new information about Brown group operations in Chiland
5 The Azure, Brown, Coral, Violet, or Chartreuse groups may be planning an attack
51 Epsilonland is mountainous
6 The Azure and Violet groups use only their own operatives, never employing locals
52 Tauland is land locked
7 The Chartreuse group is not involved
53 The attack will be at 11:00
8 The Lion is known to work only with the Azure, Brown, or Violet groups
54 The Azure and Brown groups prefer to attack at night
9 The Purple or Gold group may be involved
55 The Tauland embassy in Epsilonland is hosting a international conference on the 10th
10 All of the members of the Azure group are now in custody
56 The Chartreuse, Purple and Gold groups are known to attack at any time of the day
11 Reports from the Coral group indicate a reorganization
57 Attacking buildings when there are many people present increases casualties
12 There is a lot of activity involving the Violet group
58 The Coral, Chartreuse and Purple groups are capable of attacking year round
13 The Brown group is recruiting locals - intentions unknown
59 The Lion is planning something in April on the anniversary of his father's death
14 The Lion will not risk working with locals
60 There are fewer attacks in the dead of winter (January thru March)
15 The Jackal has been seen in Tauland
61 The Violet and Chartreuse groups want to attach the interests of Chiland, Psiland and Omegaland
16 Members of the Purple group have been visiting Omegaland
62 The Violet group is planning something big on the 5th
17 The Chartreuse group has close ties with local media
63 The Violet group prefers to operate in daylight
18 The Azure group has a history of attacking embassies
64 The lion was born in June
19 The Purple and Gold groups have blood ties
65 The Coral group prefers to attack at night
20 The Brown group has been known to use IED's
66 The Purple group prefers to attack in daylight
21 Only the Coral and Violet groups have a capacity to hit protected targets
67 The Brown group needs time to regroup
22 All high value targets belonging to Tauland and Epsilonland are well protected
68 The Azure group does not attack on its holy days
23 The attackers are focusing on a high visibility target
24 Caches of explosives have recently been found in Epsilonland, Chiland, and Psiland
25 Financial institutions in Tauland, Chiland, and Omegaland were recently attacked there is evidence of more attacks
26 Reports that uniforms were stolen in Tauland, Epsilonland and Psiland
27 Bloggers are discussing the role of financial institutions in oppressing the Coral, Violet and Chartreuse groups
28 Members of the Violet and Chartreuse groups were active in planning protests at a recent financial summit
29 Security forces are providing highly visible, around the clock protection to all visiting dignitaries in the region
30 Dignitaries in Epsilonland employ private guards
31 Tau, Epsilon, Chi, Psi and Omega-lands are providing visible, around the clock protection to their own dignitaries at home
32 A new train station is being built in the capital of country Tauland
33 Tauland's embassy in Epsilonland has a flat roof
34 Until recently most of the dignitaries in Tauland rode in Mercedes
35 Dignitaries in Chiland have motorcycle escorts
36 Epsilonland's embassy in Tauland has two helicopter pads
37 The Azure, Brown, Coral, and Violet groups have the capacity to operate in Tau, Epsilon, Chi, Psi and Omega-lands
38 Locals in Tauland, Epsilonland and Omegaland are being recruited
39 Countries Chiland, Psiland and Omegaland are taking steps to protect their embassies abroad
40 The Brown group members have entered Tauland and Epsilonland
41 Reports from Tauland, Chiland and Psiland indicate surveillance ongoing at coalition embassies
42 The target is a coalition member embassy, visiting dignitary, or financial institution (Tau, Epsilon, Chi, Psi or Omega-lands)
43 No traces of members from the Coral group have been found in countries Psiland or Omegaland
44 Chiland is in the process of deploying troops to protect the embassies of coalition partners
45 The Azure, Brown, and Coral groups want to attack the interests of Tauland, Epsilonland or Chiland
46 The Coral and Violet group operatives have entered Psiland
WHO (agents involved),
WHAT (target of attack),
WHEN, WHERE?
A stretched goal for task 2
• I was told that the sentences were trivially easy to turn into a formal representation
which was then a “logic puzzle” to be solved, so why not:
1.
2.
3.
4.
5.
Pass the ELICIT sentences through the ERG
Convert the output to Controlled English, after modelling the domain concepts
Run a logic puzzle solver that interprets CE
Solve the problem!
Thus demonstrating the power of our system to perform analysis in a valid problem, allowing us
to participate in the sensemaking experiments of task 1?
• The sentences ran through the ERG system
–
Some minor syntactic changes
•
•
–
X’s Y did not parse, so I changed to the Y of X
…
Most sentences parsed, though I have not checked all of the parses for reasonableness.
• However there is a big problem…
Ambiguity
•
On analysing the sentences formally there are many ambiguities and difficulties of
interpretation:
– The Purple group or the Gold group may be involved
• What is the meaning of “or” in this context ? Inclusive? exhaustive?
– The attackers are focusing on a high visibility target
• What does high visibility mean? Only domain knowledge or common sense can tell
– The Lion will not risk working with locals
• Does this mean the Lion will not work with locals?
– Dignitaries in Epsilonland employ private guards
• Does “in” mean “belonging to” or “located in”?
– Reports from Tauland, Chiland and Psiland indicate surveillance ongoing at coalition embassies
• Are the reports from the embassies or the host countries?
– The Violet group prefers to operate in daylight
• Does this exclude artificial light, in this context?
• How do we compare this with “at night”
– The Azure and Violet groups use only their own operatives, never employing locals
• Actually this is a rule not a fact, (using present tense?) but how do we generalise the statement just
enough to capture the intent?
•
I analysed all 68 sentences in some detail as to ambiguities!
So even “simple”
sentences are difficult to
analyse; common sense
knowledge is required
Handling uncertainty
•
We have:
–
–
defined CE domain model
generated CE facts from sentences by


–
–
Run the sentences in the CE system, leading to a possible
solution to the who, what, where and when
Recorded the rationale for the reasoning

•
Constructing CE sentences “by hand”
This has required “interpretation” of the sentences, introducing
uncertainty factors as assumptions
Including how the conclusions are based upon assumptions
This should provide an example:
–
how the reasoning can be affected by the use of
assumptions
Flow of Information
Original
sentences
This shows the
approach in the
context of our
three different
possible
approaches
ERG system
Manual
Simplified
sentences
MRS
Manual
ERG system
MRS
CE system
CE system
Domain CE
Domain CE
Domain CE
Domain
Model
(Concepts,
Rules)
Reasoning
Engine
WHO
WHAT
Explanation
WHERE
WHEN
Problem
Solving
Strategy
Uncertainty about protected targets (1)
Original sentence:
Word has it that an unprotected target is preferred to ensure the likelihood of success
(can assume is true)
We could just take this preference as
absolutely true, and infer that nonprotected things are possible targets
Actually, because we are taking a
constraint-based problem solving strategy,
we would represent this in the opposite
negative form
if
( the thing T is a non-protected thing )
then
( the thing T is a possible target ).
if
( the potential target T is a protected thing )
then
( the potential target T is a non-target ).
(The negative form could be derived automatically from the
positive form via modus tollens, but this latter reasoning rule has
not been implemented in the CE reasoning engine)
Uncertainty about protected targets (2)
However the assumption that this is actually true is an “interpretation” of the
sentence in the context of the ELICIT domain and may be too strong
We wish to explicitly record this
interpretation as part of domain
knowledge
and we ASSUME that this interpretation
can be made by the rule as a necessary
premise to infer its conclusion
there is a domain interpretation named prot_targ_non_target that
has "the preference is treated as an absolute truth" as description.
it is assumed by the agent dm that the domain interpretation
prot_targ_non_target can be made by the rule
rule_prot_targ_non_target.
[ rule_prot_targ_non_target ]
if
( the potential target T is a protected thing ) and
( the domain interpretation prot_targ_non_target can be made by the rule rule_prot_targ_non_target )
then
( the potential target T is a non-target ).
Uncertainty about protected targets (3)
Now any application of the rule will make the conclusions dependent upon the
assumption as well as the basic premise propositions
[ rule_prot_targ_non_target ]
if
( the potential target T is a protected thing ) and
( the domain interpretation prot_targ_non_target can be made by the rule rule_prot_targ_non_target )
then
( the potential target T is a non-target ).
it is assumed by the agent dm that the domain interpretation
prot_targ_non_target can be made by the rule
rule_prot_targ_non_target.
the potential target t1 is a protected thing
Providing this information in the
rationale can show the analyst this
source of uncertainty
the potential target t1 is a non-target
Uncertainty about security in the region (1)
Original sentence:
Security forces are providing highly visible, around the clock protection to all
visiting dignitaries in the region
We want to represent the inference that
visiting dignitaries in the region are
protected, as in this rule
[ visiting_digs_protected ]
if
( the visiting dignitary D is visiting the country HOSTC ) and
( the country HOSTC is located in the region thisregion )
then
( the visiting dignitary D is a protected thing ).
But how do we interpret “in the region”?
We could take the view that all of the
countries mentioned in the ELICIT
sentences are in the region we have called
“this region”
if
( there is a region named thisregion ) and
( there is a country named C ) and
then
( the country C is located in the region thisregion ).
Uncertainty about security in the region (2)
However the assumption that this is actually true is an “interpretation” of the
sentence and may be too strong
We wish to explicitly record this
interpretation as part of interpreting
the sentence
and we ASSUME that this interpretation
can be made by the rule as a necessary
premise to infer its conclusion
the sentence interpretation si29 has "we take the region to include
all countries" as description.
it is assumed by the agent dm that the sentence interpretation si29
can be made by the rule assume_allcountries_in_region.
[ assume_allcountries_in_region ]
if
( there is a region named thisregion ) and
( there is a country named C ) and
( the sentence interpretation si29 can be made by the rule assume_allcountries_in_region )
then
( the country C is located in the region thisregion ).
Note we have different types of
interpretation, those general to the domain
and those specific to a sentence
Identification requires assumptions
• The reasoning that identifies the WHAT and WHERE seems
dependent upon these and other assumptions e.g.:
– That protected things are non-targets (domain interpretation)
– That all countries are in “this region” (sentence interpretation)
– That we know the total set of possible participants and possible targets
(domain interpretation)
– That “in” means working for, not located in (sentence interpretation)
– That there is only one attack situation being talked about (domain
interpretation)
• We can generate the rationale for the reasoning, and this shows
the assumptions used
“Proof tables”
We will show the
rationale in the
form of “proof
tables”
Why cant the Lion work with the Coralgroup?
propositions
•
Each column is a rule
application
•
•
Red cell is the conclusion
Salmon cells are the premises
•
In the second column a green
cell indicates a predefined
fact and pink represents an
assumption and a word
represents the name of a rule
leading to the conclusion
•
The Equivalent Rule in CE
rule
[ no_time_overlap : ]
if ( the operative A operates in the time interval TA ) and
( the group B operates in the time interval TB ) and
( the time interval TA does not overlap the time interval TB ) then
( the operative A cannot work with the group B ) .
Determining the “WHAT”
• In our constraint-based approach we find the WHAT
(i.e. the target) by eliminating all other possibilities
• One such possibility is “ChilandDVE” representing
the Chiland dignitaries visiting Epsilonland:
– there is a dignitary named ChilandDVE that is an official of
the country Chiland and is located in the country
Epsilonland.
• We eliminate ChilandDVE by proving:
– the possible target ChilandDVE is a non-target.
Reasoning for “Chiland is a non-target”
Assume all
countries
are in the
region
Assume
protected
things are
nontargets
Assumptions used
• It can be seen that the proof of “the possible
target ChilandDVE is a non-target” involves
several assumptions, based upon an
interpretation of the original sentences
– Word has it that an unprotected target is
preferred to ensure the likelihood of success (can
assume is true)
– Security forces are providing highly visible, around
the clock protection to all visiting dignitaries in the
region All countries mentioned are in this “region”
Could Argumentation be used?
• Could arguments be had in order to construct
the assumed interpretations of the sentences
and the domain logic?
– The paper on “On Interpreting the ELICIT
sentences” https://www.usukitacs.com/node/2603
has some examples of informal arguments and
what the sentences mean
For example:
• For the sentence:
– Security forces are providing highly visible, around the clock protection to all
visiting dignitaries in the region
• In the initial analysis (done on first exposure to the sentences, with no
preconceived knowledge on the author’s part), it is stated:
– This sentence is highly ambiguous, since it does not state the region in
question. It seems unlikely that this refers to all regions, so it is hard to see
how this information could be used in inference.
• In subsequent work to formulate the sentences, the view changed to the
interpretation that “all countries are in the region”, since the alternative
view that “only some are in the region” would not make sense given that:
– there is no information given about the membership of countries in any
region
– There is no mention of specific regions, just the idea of “the region”
– The sentence would be meaningless if either the initial analysis were true or
there were just “some” countries in the region without stating which
– It is unlikely that a sentence is given that is meaningless