Transcript iee2000-04PresAFinal

Signing for the Deaf using Virtual
Humans
Ian Marshall
Mike Lincoln
J.A. Bangham S.J.Cox
(UEA)
M. Tutt M.Wells
(TeleVirtual, Norwich)
SignAnim
School of Information Systems, UEA
Televirtual, Norwich
Subtitles to Signing Conversion
Funded by
Independent Television Commission (UK)
Tessa
School of Information Systems, UEA
Televirtual, Norwich
Speech to Signing of Counter Clerk Turns
in PO Transactions
funded by Post Office
ViSiCAST
School of Information Systems, UEA
Televirtual, Norwich
Independent Television Commission (UK)
Post Office (UK)
RNID (UK)
IvD (Holland)
University of Hamburg (Germany)
IST (Germany)
INT (France)
EU funded 5th Framework Project
Background – Deaf Community
Deaf
v
Hard of Hearing
Signing
v.
Subtitles
60,000
v.
1 in 8 of population
300 Level 3 signers
Background – Sign Language
Signed
Sign Supported
British Sign
English
English
Language
(SE)
(SSE)
(BSL)
educated deaf community
preferred first language
SignAnim – Aims and Aspirations
Exploration of (semi-)automatic conversion of
subtitles to sign language …
… to increase access for the Deaf ...
… with a potential of providing access to up to
50/80% of TV broadcasts.
SignAnim – Natural Language Processing
Subtitle stream up to 180 words min -1
Sign rates typically 50% of speech rate (100 signs min -1)
SE – too verbose to be signed in full
SSE – elision of low information words
BSL – translation to multi-modal signs
SignAnim Components – Simon the Avatar
Sign Stream
Data Capture
Sign
Dictionary
Avatar
Motion Capture
Cybergloves
Magnetic Sensors
Video face tracker
Schematic of SignAnim system
Audio/Video
Stream
TV Capture Card
Eliser
Teletext
Stream
P1
D1
P2
Avatar
… P
n
D2
Software Mixer
SignAnim Components – Eliser
Requirements
Resolution of Lexical Ambiguity
Elision
If @ receiver Timeliness of signing
v
If @ transmitter prioritising of parts of sign sequence
Eliser - Summary
Sign Stream
Subtitle Frames
CMU Parser
Prioritiser
Elision Level
Sign
Dictionary
SignAnim – Natural Language Processing
‘Last night we brought you the tale of the duck that could not swim
and had to learn while a guest of the RAF in Norfolk.’
26 words
in 2 subtitle frames
time to speak / time subtitles on screen
time to sign in full
finger spelling significant overhead
7 secs
18 / 14 / 9 secs
SignAnim – Natural Language Processing
‘Last night we brought you the tale of the duck that could not swim
and had to learn while a guest of the RAF in Norfolk.’
Resolution of some lexical ambiguity by p.o.s. tagging
- duck
- had
- swim
- in
noun/ verb
auxiliary/ verb
noun/ verb
participle/ preposition
to facilitate correct sign selection
SignAnim – Natural Language Processing
‘Last night we brought you the tale of the duck that could not
swim and had to learn while a guest of the RAF in Norfolk.’
Potential elision
determiners
auxiliary verbs
modifying phrases
adjectives and adverbs
in extreme cases jettison entire sentences
SignAnim – Natural Language Processing
‘Last night we brought you the tale of the duck that could not
swim and had to learn while a guest of the RAF in Norfolk.’
Additional problems
structural ambiguity
appropriate sign
no sign for guest, default finger spell
SignAnim – CMU link grammar
Positive features
Lexically driven sentence parser
Robust
Prioritorises multiple analyses
On failure returns partially parsed word sequence
Modifiability
SignAnim – CMU link grammar example
Xp
Jp
Wd
Pv
CO
Ds
Perhaps the
E
Ssi
hen
Dsu
was
actually
MVp
reared
A
by
a
broody
duck
!
CMU link grammar parser - a shell
<noun>
:
<adj>
<det>
<verb>
<prep>
:
:
:
:
( {A-} & {D-} & Wd- & S+ ) or
( {A-} & {D-} & O- ) or
( {A-} & {D-} & PN- );
A+ ;
D+ ;
S- & O+ & {@PP+};
PP- & PN+;
book.n books.n report.n reports.n room person
yellow green
the a
book.v books.v report.v reports.v brings
on in
CAPITALIZED-WORDS
"."
LEFT-WALL
:
:
:
:
:
<noun> ;
<adj> ;
<det> ;
<verb> ;
<prep> ;
: <noun> or <adj> or <det> ;
: FS- ;
: (Wd+ & FS+) ;
CMU link Grammar Parser - link construction
books
.n :
( {A-} & {D-} & Wd- & S+ ) or
( {A-} & {D-} & O- ) or
( {A-} & {D-} & PN- ) or
(S- & O+ & {@PP+});
.v :
Wd
S
O
PN
{D}
{D}
{D}
{A}
{A}
{A}
books
books
Wd
S
{D}
O
S
{@PP}
books
books
O
S
{@PP}
{A}
person
books
linkparser> A person reports the book.
Found 1 linkage (1 had no P.P. violations)
Unique linkage, cost vector = (UNUSED=0 DIS=0 AND=0 LEN=7)
+----------------FS---------------+
+---Wd---+
+------O-----+
|
|
+--D-+---S---+
+--D--+
|
|
|
|
|
|
|
|
///// a person reports.v the book.n .
(m)
(m)
(m)
(m)
(m)
/////
/////
a
person
reports.v
the
FS
Wd
D
S
O
D
<---FS---->
<---Wd---->
<---D----->
<---S----->
<---O----->
<---D----->
FS
Wd
D
S
O
D
.
person
person
reports.v
book.n
book.n
Eliser - elision stategy
Augment CMU dictionary with further p.o.s. information
e.g. has.aux
v.
has.v
Rules for word and path priorities
#Link Weight
CO
3
D
1
Ds
1
Left Path
X
-
Left Word
X
X
X
Right Path
-
Right Word
-
G
AN
A
4
4
4
X
-
X
X
X
-
-
RS
4
-
X
X
X
Eliser - Prioritorising
Xp
Jp
Wd
Pv
CO
Ds
Perhaps the
E
Ssi
hen
Dsu
was
actually
MVp
reared
A
by
a
broody
duck
!
Eliser - Prioritorising
Xp
Jp
Wd
Pv
CO
Ds
Perhaps the
10
10
Dsu
E
Ssi
hen
was
10
10
actually
10
MVp
reared
10
A
by
a
10
10
broody
duck
10
10
!
Eliser - Prioritorising
Xp
Jp
Wd
Pv
CO
Ds
Perhaps the
3
10
Dsu
E
Ssi
hen
was
10
10
actually
10
MVp
reared
10
A
by
10
a
10
broody
duck
10
10
!
Eliser - Prioritorising
Xp
Jp
Wd
Pv
CO
Ds
Perhaps the
3
1
Dsu
E
Ssi
hen
was
10
10
actually
10
MVp
reared
10
A
by
10
a
10
broody
duck
10
10
!
Eliser - Prioritorising
Xp
Jp
Wd
Pv
CO
Ds
Perhaps the
3
1
Dsu
E
Ssi
hen
was
10
10
actually
2
MVp
reared
10
A
by
10
a
10
broody
duck
10
10
!
Eliser - Prioritorising
Xp
Jp
Wd
Pv
CO
Ds
Perhaps the
3
1
Dsu
E
Ssi
hen
was
10
10
actually
2
MVp
reared
10
A
by
9
a
9
broody
duck
9
9
!
Eliser - Prioritorising
Xp
Jp
Wd
Pv
CO
Ds
Perhaps the
3
1
Dsu
E
Ssi
hen
was
10
10
actually
2
MVp
reared
10
A
by
9
a
1
broody
duck
9
9
!
Eliser - Prioritorising
Xp
Jp
Wd
Pv
CO
Ds
Perhaps the
3
1
Dsu
E
Ssi
hen
was
10
10
actually
2
MVp
reared
10
A
by
9
a
1
broody
duck
4
9
!
Eliser - Elision
Xp
Jp
Wd
Pv
CO
Ds
Perhaps the
3
1
Dsu
E
Ssi
hen
was
10
10
actually
2
MVp
reared
10
A
by
9
a
1
broody
duck
4
9
!
TESSA - Overview
Aim : To give access to Post Office services for
those whose first language is not English.
TESSA Input : Speech Recognition
•
Restricted Number of sentences (115)
•
Variable quantities (monetary amounts,
days of the week)
•
Grammar defined as FSN
•
MLLR acoustic adaptation
•
Entropic recognition engine
TESSA Output : BSL and Foreign Language
•
BSL sign sequences
•
Signs for variable quantities blended into
standard phrases
•
Customer may ask for phrases to be
repeated
•
Text translations into 4 languages for
non-English speakers
•
English text for the hard of hearing
Conclusions
SignAnim and Tessa demonstrated
+ replay of motion captured sequences readable
+ usefulness of existing NLP and speech
recognition technologies
+ desirability of BSL (rather than SSE)