Transcript Speech Grammars for Textual Entailment Patterns in Multimodal

LREC 2010
Speech Grammars for Textual Entailment
Patterns in Multimodal Question
Answering
Daniel Sonntag, Bogdan Sacaleanu, DFKI
21/05/2010
Daniel Sonntag
|1
Outline
»
Semantic Dialogue Shell
»
Textual Entailment
»
Processing Example
»
Conclusions
Daniel Sonntag
|2
Acknowledgements
» Thanks go out to Robert
Nesselrath, Yajing Zang,
SPARQL
Günter Neumann, Matthieu
Deru, Simon Bergweiler,
Gerhard Sonnenberg, Norbert
Reithinger, Gerd Herzog,
Alassane Ndiaye, Tilman
Becker, Norbert Pfleger,
Alexander Pfalzgraf, Jan
Schehl, Jochen Steigner, and Ease the interface to
Colette Weihrauch for the
external third-party
implementation and
components.
evaluation of the dialogue
Robust
infrastructure.
Question
Understanding
Daniel Sonntag
|3
Semantic Dialogue Shell
Daniel Sonntag
|4
Dialogue Shell Workflow
Speech
Interpretation
Text
Interpretation
Text
Summarisation
Modality
Fusion
Interactive
Semantic
Mediator
Graphic
Generation
- Domain Model
- Context Model
- User Model
Text
Generation
Speech
Interpretation
Visualisation
Daniel Sonntag
Presentation
Planning
Personalisation
Gesture
Interpretation
Dialogue
External
Information
Sources
SPARQL
Remote
Linked Data
Services
and
Interaction
eTFS/SPARQL
Management
SPARQL
Interactive
Service
Composition
SPARQL
OWL-API
Semantiic (Meta)
Services
RDF
KOIOS
(Yago
Ontology)
OWL
AOIDE
(Music
Ontology)
Visualisation
Service
|5
THESEUS’s Semantic Dialogue
Shell: Goals and Requirements
» Multimodal interaction with the Semantic
Web and the Internet of Services
» Components customisable to different use
case scenarios
» Flexible adaptation to
» Input and output modalities
» Interaction devices
» Knowledge bases
» To understand a greater number of
queries:
» Robust question understanding (NLU) when
using both speech and written text input
» Semantic (i.e., a RDF or OWL based) query
interpretation
» The combination of robust question understanding
and ontology-based answer retrieval
Daniel Sonntag
|6
SPARQL Query Editor
SPARQL
Daniel Sonntag
|7
Daniel Sonntag
|8
Speech Grammar
<utterance name="SHOW_CV_OF_PERSON">
<phrases>
<phrase>zeige ?mir den [werdegang lebenslauf] [von zu] PERSON</phrase>
<phrase>sage ?mir mehr über den [werdegang lebenslauf] von PERSON</phrase>
<phrase>wie ist der [werdegang lebenslauf] von PERSON</phrase>
</phrases>
<semantic-interpretation>
<object type="odp#TaskRequest">
<slot name="odp#fusion-confidence">
<value type="Float">1.0</value>
</slot>
<slot name="odp#hasContent">
<object type="dialogshell#BackendRetrievalTask">
<slot name="dialogmanager#backendComponent">
<value type="String">mediator:summarizer</value>
</slot>
<slot name="odp#hasContent">
<variable name="PERSON"/>
</slot>
</object>
</slot>
</object>
</semantic-interpretation>
</utterance>
Daniel Sonntag
|9
Textual Entailment
Our idea is that an NLU grammar for speech input
can be reused to build more robust multimodal
text-based question understanding by automatically
generating textual entailment patterns.
Daniel Sonntag
| 10
Textual Entailment &
Information Access
Conceptual
Ontology
(RDF/OWL)
Method1: Speech / Semantic
Grammars
• RDF/OWL reasoning
Request
Method2: RTE
• textual reasoning
Information
(RDF)
Textual
Daniel Sonntag
| 11
Textual Entailment through
Alignments
» For textual entailment to hold we need:
» text AND background knowledge  hypothesis
» but background knowledge should not entail hypothesis alone
» Background Knowledge
» Unsupervised acquisition of linguistic and world knowledge from
general corpora and web
» Acquiring larger entailment corpora
» Manual resources and knowledge engineering
» Alignment-based TE and Background Knowledge
» Preprocessing: POS, morphology, cognates
» Representation: bag-of-words
» Knowledge Sources: WordNet, Roget‘s Thesaurus, Wehrle
Thesaurus
Daniel Sonntag
| 12
Argumentation
» Input modalities are usually interpreted according to separate
models and aligned to a shared model (often coarse-grained).
» Present a method of interpretation based on a common model
(propagated changes to multiple modalities).
» Built on the grammar for
speech inputs =
Leveraging Existing
Speech Grammar
Knowledge
Daniel Sonntag
| 13
Processing Example
Daniel Sonntag
| 14
Entailment Patterns and Possible
Hypotheses
Daniel Sonntag
| 15
Entailment Patterns and Alignment
Engine
» Association-based word
alignment. Three steps:
» lexical segmentation, when
boundaries of lexical items
are identified;
» correspondence, when
possible similarities are
suggested in line with some
correspondence measures;
» alignment, when the most
likely semantically similar
word is chosen.
Daniel Sonntag
| 16
Entailment Patterns and Alignment
Techniques
» Question: What is the
birthplace of Angela Merkel?
» Pattern: Where is Angela
Merckel born?
» Filters on a full alignment.
Daniel Sonntag
| 17
Entailment Patterns and Alignment
Techniques
POS Filter: Exclude unlikely alignments
based on POS. Allow for the additional
mappings:
verb to noun
(i.e., born
vs.
Lexical Semantic
Resource
Filter:
birthplace)
WordNet (synonyms); Roget Thesaurus
String Similarity Filter: Dice coefficient,
(conceptually related words)
Longest common subsequence ratio;
submatches, misspellings
System of weights: nouns, verbs, and adjectives are better scored than function words.
Daniel Sonntag
| 18
Dialogue Example
» (1) U: “Open my personal address book. What do you know about
Claudia?”
» (2) S: “There’s an entry: Claudia Schwartz. The personal details are
shown below. She lives in Berlin.” + Google Map Display of street
coordinates.
» (3) U: “Which is Claudia’s favorite kind of music? Do you know the
bands she likes most?”
» (4) S: “Nelly Furtado” + Displays videos obtained from YouTube.
(Rest API)
» (5) U: “How did experts rate her last album?”
» (6) S: Shows an expert review according to the BBC Linked Data Set.
» (7) U: “Show me other news.”
» (8) S: Opens a browser + Text field and a new agency Internet
page (featuring Angela Merkel)
» (9) U writes: “Where was Angela Merkel born? / In which town was
Angela Merkel born?” etc.
» (10) S: “She was born in Hamburg.”
» (11) U speaks again: “And Barack Obama?”
» (12) S: “He was born in Honolulu.”
» (13) U: “Show me Angela Merkel’s career.”
Daniel Sonntag
| 19
Touchscreen Installation
Daniel Sonntag
| 20
Image Analysis in
Biomedicine MEDICO
Retrieval and examination of 2D picture series
Daniel Sonntag
| 21
Conclusions
Daniel Sonntag
| 22
Conclusions
»We described a multimodal dialogue shell for QA
and focussed on the robust multimodal question
understanding task.
»The textual interpretation is based on automatically
generated textual entailment patterns.
»As a result, we can deal with written text input and
different surface forms more flexibly according to
the derived entailment patterns.
Daniel Sonntag
| 23
Method Comparison
» Method 1: Speech Grammars
» Speech grammars are verbose
» Requires full coverage of expected input
» Hard-coded reasoning in rules
» Example:
» Show me all pictures of X.
» What pictures does X have?
» Show me all images of X.
» Method 2: NLU Grammars
» Use of Textual Entailment
» NLU grammars are compact
» Requires partial coverage of possible input
» Example:
» Show me all pictures of X.
» Entailed utterances:
» What pictures does X have?
» Show me all images of X.
Daniel Sonntag
1
2
| 24