Transcript slides

FIRE 2013
Presentation on
:
Transliterated Search using Syllabification
Approach
By:Hardik Joshi1,
Apurva Bhatt1,
Honey Patel2
{hardikjjoshi,apurva.bhatt7,Honeypatel.39}@gmail.com
1Department
of Computer Science, Gujarat University, Ahmedabad, India.
2L.J. College of Engineering, Ahmedabad, India
Dec
@FIRE 4rth Dec 2013
Content
 Introduction
 Our Approach
 Syllabification
 Our Results
 Error And Analysis
 Conclusion
Introduction
 There is need to provide local language support in web based
applications because various domains such as ecommerce sites
require English knowledge.
 The challenge in transliteration is take the word “राष्ट्रपति” for
this word “rashtrapati”, “rashtrapathi”, “raashtrapathy”,
“raashtrpati” are various possible combinations may possible
which one should be correct is again an issue.
 Transliteration tasks become difficult in presence of out of
vocabulary words (OOV) and noisy words.
 In both the subtasks, the transliteration was performed using
syllabification approach.
 In the subtask-1, we had done the morphological analysis of
English words , then a corpus based approach used to identify
frequently occurring Hindi words.
 In the subtask-2, the queries were formulated that contained
both Roman and Devanagari script and Roman script for
separate run submissions.
Syllabification Approach
syllable
 Linguists have different languages have constraints on possible
consonant and vowel sequences that characterize not only the
word structure for the language but also the syllable structure.
Onset
Rhyme
 Vowels @ center (nucleus)
 consonant @ beginning (onset)
 End is coda
coda
nucleus
Syllable Structure Example
 Word
 Sprint
Training Format
Source
sudakar
chhagan
jitesh
narayan
shiv
madhav
mohammad
Target
स ◌ु द ◌ुा क र
छगण
ज िु◌ ि ◌ु श
न ◌ुा र ◌ुा य ण
श िु◌ व
म ◌ुा ध व
म ◌ु ह म ◌ु म द
Algorithm for subtask-I
 Step 1: First of all words are fetching in English dictionary.
 Step 2: perform spell-check ,stemming and also morphological
analysis for English language, if no spell error and match found then
label the word as English =E.
 Step 3: If English word are not found then check with English
corpus of US News paper.
 Step 4: If English word found then check with English corpus of
Indian news paper.
 Step 5: If English word found in US News paper and not found in
Indian news paper then word=E.
 Step 6: Step 2 and step 5 are parallel apply for English
words and label as =\E.
 Step 7: Remaining words would be transliterate into
Hindi words and Label the word as = \H.
 Step 8: Apply to Moses tool ,which one is help English
words transliterate into Hindi words.
RESULT OF SUBTASK-1
Results For Subtask 2
 Run 1 “मर सापन न कक रानी काब आयगी ि mere sapnon ki rani
kab aayegi tu”.
 Run 2 “mere sapnon ki rani kab aayegi tu”.
Metrics
Run-1
Run-2
Maximum
Score
Median
Score
nDCG@5
0.5627
0.5262
0.8052
0.5620
nDCG@10
0.5619
0.5232
0.8002
0.5608
MAP
0.2546
0.2163
0.4236
0.2355
MRR
0.5835
0.5730
0.8440
0.5884
Error And Analysis
 There are some problems in the transliteration which
decreased the precision.
 Error in the maatra : “sapnon” => “सापन न”, “ki” => “की”,
“kab” => “काब”, “main” => “ममन” & “mein” => “मीन” , na
=> न & ka => क
 Multiple Mapping of the words e.g. T = ि, ट, i.e. tera=>टरा,
tum => िूम, to => ट , teri =>टरर .
 Missing sounds (फ, ख, छ ‘chh’, ksh) i. e. for word “accha” we
got “आक्का”, for , “poochho” we got “पछ
ू ट”.
 Multiple Transliterations- c,k
 The vowel are not giving perfect answers
i.e. “lo” => “लॉ” , “ho”=> “ह र”, “ko” => “कॉ”
 Spelling Variations(shree,shri)
 Conjuncts formation(“kya” => “कया”)
 Missing of vowels ‘ak tr khan’ (अक ु िर खान)
 ‘y’ As Vowel: ‘anthony’ & ‘Shyam’
Conclusion
 We used the syllabification approach and considered the most
probable term in the transliteration process. The word labeling
task was performed assuming that a term either belongs to
English language or Hindi language. We were able to get high
accuracy in English recall as the labeling approach used
morphological analysis and dictionary approach. However due
to syllabification model, the transliteration did not give high
precision resulting in lower precision of transliteration tasks
and subsequently lower precision metrics in the song lyrics
retrieval tasks.