Transcript slides - Department of Computer Science, Hong Kong Baptist

Content-based
Music Information Retrieval
Student: DENG Jie. Supervisor: Prof. LEUNG, Clement
Department of Computer Science
Hong Kong Baptist University
March, 2010
1
Outline
What will be Covered:






Introduction
A Brief Review of Music
MIR in the Real World and Challenges
Current Content-based MIR Key Techniques
Evaluation of MIR
Conclusion and Future Work
2
Introduction
 Music Information Retrieval (MIR) is the interdisciplinary
science of retrieving information from music.
– Mainly based on three-filed subjects: traditional
information retrieval, musicology and digital audio.
 Content-based MIR is the science of extracting features
from musical content, such as melody, rhythm and tempo
and so on to facilitate tasks such as analysis and music
retrieval.
 Aim:
– To better understand “music” in the music work
– To really search music by “music”
3
Motivation - Music Discovery
Music Channels
Chart Shows
Record store
Gigs
+
Radio
Mate’s
recommendation
4
Music IR Application
 Digital Music Libraries / Sound Archives
–
–
Seeking for content-based access music libraries
Combined with metadata search of existing catalogues
 Music Education
–
Voice or instrumental teaching
 Music Related Legal and Copyright
–
Is the creative content of this music work based on
something for which others hold the rights?
 Musicology
–
–
–
Is this piece of music work similar to any other works?
Dose any part of this piece closely resemble any part of
any other works?
Is this piece of music work is based on others?
5
A Brief Review of Music
Music Concepts
 Three basic features of a musical sound
–
–
–
Pitch
Intensity / Dynamics
Timbre / Tone color
 There are many other terms describing music
–
–
–
–
Tempo
Tonality
Time Signature
Key Signature
6
A Brief Review of Music
Music Characteristics
 Music can be defined as the art of disposing and producing
sounds and silences in time
–
Has horizontal and vertical dimensions
 The main dimensions of music can be used for music
retrieval (Reference Nicola Orio)
–
–
–
–
–
–
–
Timbre : Quality of the produced sound
Orchestration : Sources of sound production
Acoustics : Quality of the recorded sound
Rhythm : Patterns of sound onsets
Melody : Sequence of notes
Harmony : Sequence of chords
Structure : Organization of the musical work
7
A Brief Review of Music
Music Representation
–
–
–
–
Visual (musical scores, manuscripts)
Aural (digital music)
Text
Hybrid (visual representation of an audio music file )
__________________________________________________________________________________
Example: Visual Representation
Common
Music
Notation
Tablature
E---------------0---------3-3------3--1-1-------------3-1-1---------------B---1-------------3-2-----2-2------2--3-3-----3-2---2---3-3------------1--G---0--0--0h1-------0-----0-0------0--2-2---2---0-0-----2-2------------2--D---2--2------------------------------0---0-------------0----0-1-2-3-3---3A-3-------------------0-0-----0--0--------------0-------------------------E-------------0------------------------------------------------------------
8
MIR in the Real World
There are mainly three-category USERS in MIR
“Amateur”
“Professional”
Librarians
Just about
anyone!
Publishers
“Academic”
Musicologists
Educators
Producers
Performers
Composers
Lawyers ...
Vast numbers
Very many
Significant
numbers
9
MIR in the Real World
Common Music Data and Format
 Audio recordings
–
Sampled sound
–
Wave, MP3, AAC, etc.
 Symbolic recordings
–
Abstract musical instructions, MusicXML
–
Scores, MIDI, Humdrum, etc.
10
MIR in the Real World
Overview of some existing music search systems
 Search by music related metadata: (artists, albums, tracks, music
reviews, new release, etc.) Yahoo! Music and Allmusic are the examples of
this search type
 Search by music lyrics: Lyrics.com and SongLyrics.com
 Music Media Management and Track Identification: Identify metadata for
music tracks, for example Gracenote and MusicIP
 Recommend similar music: by mining some music feature elements
(melody, rhythm, tone color, etc) to recommend user some similar music
 Recommend personalized music: by mining some users’ information to
recommend them some their favorite music
11
Challenges in MIR
 Began in the 1950’s, still an emerging discipline
 Subjectivity and Versioning
 Many levels of music knowledge
 Lack of bibliographic control and data quality
________________________________________________________________________
MIR Pipeline
Hearing
Representation
Signal Processing
Understanding
Analysis
Machine Learning
Reacting
Interaction
Human Computer Interaction 12
A Simplified MIR Map
This very schematic diagram highlights trends
Extracted
or produced
information
Actions
External
data
integration of audio visual, symbolic and textual data
13
Basic Steps of Content-based MIR
 Representation of music contents
– Features: melody, rhythms, etc.




Feature extraction from music data
Feature indexing
Query interface
Matching query features against the feature
index
14
Content-based MIR
 Representation of Music Content
– Most music features used to represent music are
always melody.
– Rhythm feature only consider the rhythm
omitting the melody.
– Melody contour method uses three characters to
express the contour of melody.
15
Content-based MIR
 Feature extraction from music data
– There are two category algorithms: time domain
(Autocorrelation function, Average magnitude
difference function and Simple inverse filter
tracking) and frequency domain models
(Spectrum and Cepstrum)
– Common extracted tools in the following:
• Short-term Fourier Transform features (FFT)
• Mel-Frequency Cepstral Coefficients (MFCC)
• Daubechies Wavelet Coefficient Histogram (DWCH)
– Pitch is the main feature extracted in practice
16
Content-based MIR
 Feature Indexing
– Index terms: play a similar role of words in textual
documents.
– Sequence matching techniques: consider both the
query and the documents as sequences of symbols
and model the possible difference between them.
– Geometric methods: cope with polyphonic scores and
also exploit the properties of continuous distance
measures.
– Based on the above methods, there are mainly three
category music search on the melody feature.
• Melodic retrieval based on index terms (N-grams)
• Melodic retrieval based on sequence matching
• Melodic retrieval based on geometric methods
17
Content-based MIR
 Query Interface
– Query by text (keywords: album, artist, track, etc.)
– Query by aural (singing or humming)
• Wave input (sing the whole or part of the songs)
• Music notes segmentation
• Thematic melodies are extracted, translated into text
representations of intervals, pith, and harmony
• Comparison procedure
– Query by tapping
• Wave input by tapping
• Compute the duration of each note
• Similarity comparison
18
Content-based MIR
 Matching Query Features against the Feature Index
–
–
–
–
–
Approximate/Partial matching
Similarity measure (MFCC, GMM, KNN)
Precision: how many of the answers are in fact correct
Recall: how many of the correct answers are in fact retrieved
Relevance feedback
 Vector space model
– Documents and queries are presented by vectors
– Each element in a vector is determined by an indexing scheme (Ngrams or others)
– The value of each element is determined by a weight scheme
– The similarity between document di and query qj :
19
Evaluation of MIR
 The community has established an array of software
tools to support this work
– see http://music-ir.org/evaluation
 In 2004, Audio Description Contest first attempted
to build comparative benchmark of MIR algorithms.
 Downie has already given us the foundations and
future of the scientific evaluation of MIR systems.
 Traditional information retrieval evaluation can also
be adopted in MIR, for example precision and recall
measures.
20
Conclusion
 Music is a complicated art form of information and requires
special retrieval systems
 MIR technology is improving, but the real application is still lacking
 Basic music concepts and characteristics
 Basic steps and models of MIR
 Current Content-based MIR Key Techniques
 Scientific evaluation of MIR
___________________________________________________________
Future Work
Mining the semantic information in multimedia, especially in digital
audio music, and then propose a comprehensive and adaptive method
to automatically analysis and retrieve the high level semantic
information of music, for example, emotion, mood, and style, etc.
21
Related Researches and Projects
 ISMIR since 2000
– International Symposium on Music Information
Retrieval
 WOCMAT since 2005
– Workshop On Computer Music and Audio Technology
 Digital archive application
– Data mining in digital music archive
 Free music audio, sound processing tools and musicrelated visualization and mining tools
– http://www.music-ir.org/evaluation/tools.html
 Music IR evaluation since 2005
– http://www.music-ir.org/mirexwiki/index.php/Main_Page
– Test collection: music documents, query sets, and
judgment
– Major handle: copyright issue
22
Reference
•
•
•
•
•
•
•
•
•
Michael S. LEW, Nicu Sebe. Content-Based Music Information Retrieval: Current
Directions and Feature Challenges. Proceedings of the IEEE, April 2008
Nicola Orio. Music Retrieval: A Tutorial and Review. Foundations and Trends in
Information Retrieval, Volume1, Issue 1, Pages 1-96, 2006.
J. T. Foote, "An Overview of Audio Information Retrieval." In ACMSpringer Multimedia Systems, vol. 7 no. 1, pp. 2-11, ACM Press/Springer-Verlag,
January 1999
Remco C. Veltkamp, Frans Wiering, Rainer Typke. Content Based Music Retrieval.
In B. Furht (Ed.), Encyclopedia of Multimedia. Springer, 2006.
Giovanna Neve, Nicola Orio: A Comparison of Melodic Segmentation Techniques
for Music Information Retrieval. ECDL 2005: 49-56.
Hwei-Jen Lin, Hung-Hsuan Wu. Efficient geometric measure of music similarity.
Information Processing Letters, Volume 109, Issue2, Page 116-120, 2008.
Iman S. H. Suyoto, Alexandra L. Uitdenbogerd, and Falk Scholer. Searching
Musical Audio Using Symbolic Queries.IEEE. Transactions onAudio, Speech and
Language Processing, 16(2):372–381, 2008.
The Scientific Evaluation of Music Information. Retrieval Systems: Foundations
and Future. Computer Music Journal, Computer Music Journal, 28:2, pp. 12–23,
Summer 2004.
Michael Fingerhut. Real music libraries in the virtual future: for an integrated
view of music and music information. Digitale bibliotheken voor muziek, 2005.
23
Thank you!
Questions?
24