Transcript 90minMeeting

Classifying Email into Acts
Verb
Commisive
Deliver
Verbs
Directive
Request
Commit
Propose
Amend
Noun
Activity
Event
Ongoing
Meeting
Other
• From EMNLP-04, Learning to
Classify Email into Speech Acts,
Cohen-Carvalho-Mitchell
• An Act is described as a verbnoun pair (e.g., propose
meeting, request information)
- Not all pairs make sense.
One single email message
may contain multiple acts.
Delivery
Opinion
Data
Nouns
• Try to describe commonly
observed behaviors, rather
than all possible speech acts
in English. Also include nonlinguistic usage of email (e.g.
delivery of files)
Some Improvements
1. With more labeled data (1743 msgs), using
1g+2g+3g+4g+5g features
2. Using careful (and act specific) pre-processing
of message text
3. Using act specific feature selection scheme
(Info Gain, ChiSquare, etc)
- Significant performance improvements.
Some Examples of 4-grams
and let mmee know
know what pppeople think
would be able to
do pppeople want to
do pppeople need to
do not want to
pppeople need to get
please let mmee know
pppeople think pppeople need
mmee know what pppeople
what do pppeople think
pppeople be able to
pppeople don not want
pppeople would be able
that would be great
Call mmee at home
Request
Req
is fine with mmee
is good for mmee
i will be there
i will look for
will look for pppeople
i will see pppeople
as soon as I
$numbex per person
i will bring copies
our meeting on dday
is ok for mmee
look for pppeople
in will try to keep
-numbex i will
i will try to
i will check my
I do not have
Commit
meet at horex pm
horex pm on ddday
on ddday at horex
pppeople meet at horex
to meet at horex
would like to meet
please let mmee know
ddday at horex am
ddday at horex pm
lets plan to meet
would pppeople like to
pppeople will see pppeople
is fine with mmee
numbex-numbex pm
can pppeople meet at
ddday numbex/numbex
is good for mmee
Meeting (noun)
Results
1716 msgs, 5-CV
Paper Best = (best classifier in emnlp04)
Paper++ = (unigram, no feature selection, maxent, more data),
Best = (1g?2g3g4g5g,maxent,InfoGain feature selec.)
paper best
paper++
best
0.3
Error Rate
0.25
0.2
0.15
0.1
0.05
0
Req
Dlv
Cmt
Prop
meet
dData
Results
1716 msgs, 5-CV
Paper++ = (unigram, no feature selection, maxent, more data),
Best = (1g?2g3g4g5g,maxent,InfoGain feature selec.),
human = (vitor-susan agreement)
paper++
best
human
1
Kappa
0.8
0.6
0.4
0.2
0
Req
Dlv
Cmt
Prop
meet
dData
Collective Classification: Predicting
Acts from Surrounding Acts
Request
Request
Delivery
Parent message
???
Proposal
Commit
Child message
Content versus Context
•
•
•
•
Content: Bag of Words features only (using only 1g features)
Context: Parent and Child Features only ( table below)
8 MaxEnt classifiers, trained on 3F2 and tested on 1F3 team dataset
Only 1st child message was considered (vast majority – more than
95%)
Context
Request
Delivery
Content
???
Request
Proposal
Commit
dData
Meeting
Parent message
Commissive
Child message
Directive
Propose
Parent Boolean
Features
Child Boolean
Features
Parent_Request,
Parent_Deliver,
Parent_Commit,
Parent_Propose,
Parent_Directive,
Parent_Commissive
Parent_Meeting,
Parent_dData
Child_Request,
Child_Deliver,
Child_Commit,
Child_Propose,
Child_Directive,
Child_Commissive,
Child_Meeting,
Child_dData
Commit
Deliver
Request
0
0.1
0.2
0.3
0.4
0.5
Kappa Values (%)
Kappa Values on 1F3 using Relational (Context) features
and Textual (Content) features.
Set of Context Features (Relational)
Collective Classification Model
Commit
Other acts
…
…
…
Request
Deliver
Parent Message
Current Msg
Child Message
Collective Classification algorithm
(based on Dependency Networks Model)
New inferences are accepted only if confidence is above the Confidence
Threshold. This Threshold decreases linearly with iteration, and makes the
algorithm works as a temperature sensitive variation of Gibbs sampling – after
iteration 50, the threshold is 50% and then a pure Gibbs sampling takes place
Collective Classification Results
Deliver
Commissive
Request
0.55
Kappa
0.5
0.45
0.4
0.35
0.3
0.25
0
10
20
30
Iteration
40
50
Act by Act Comparative Results
Baseline
Collective
43.44
44.98
dData
38.69
42.01
Deliver
40.72
36.84
Propose
49.55
47.25
Request
58.37
58.27
Directive
Meeting
47.81
52.42
32.77
30.74
Commit
Commissive
37.66
0
10
20
30
42.55
40
50
60
70
Kappa Values (% )
Kappa values with and without collective classification, averaged over the
four test sets in the leave-one-team out experiment.
What goes next?
1. Extend Collective classification by using the new
SpeechAct classifiers (1g-5g, feat selection)
2. Online(incremental) and semi-supervised learning –
CALO focus.
3. Integration of new Speech Act package to Minorthird &
Iris/Calo.
4. Role discovery – network-like features + speech act