Transcript Musicology, Music Cognition and Musical Similarity

Melodic Features and
Retrieval
ISMIR Graduate School, Barcelona 2004
Musicology 3-4
Frans Wiering, ICS, Utrecht University
Outline






yesterday’s assignment
demo: MIR outside academia (7:20; 44:10)
one-dimensional melody retrieval
Gestalt view of melody
advanced melody retrieval
assignment
one-dimensional melody retrieval

common assumption is (was?) pitch-only retrieval is
sufficient



e.g. CCGGAAGGFFEEDDEC
mechanisms for fuzzy matching
variants


interval (distance between 2 pitches)
pitch-contour



same/up/down (Parson’s Code)
RURURDRDRDRDRUD
examples:


www.musipedia.com (Rainer Typke)
www.themefinder.org (CCARH)
Results from Musipedia


query is ranked 3
other hits are
very unlikely


unfortunately no
notation/sound
available
Haydn: evident
false positive

why?
Themefinder

Several 1-dimensional
search options, e.g.








pitch
interval
contour
rhythm
wildcards
each theme stored as a
number of strings
matching by regular
expressions
ca. 40.000 themes



Barlow and Morgenstern
(1948)
ESAC encodings
Lincoln, 16th Century Motet
(DARMS project)
results from Themefinder
Query: +m2 +M2 P1 -M2 -m2 -M2


Example from Byrd &
Crawford (2001)
other hits





not as far-fetched as
musipedia’s
different rhythm
different meter
still not very similar
is this what people have in
mind?
Nice one we’ve just discovered



www.tuneteller.com
Pitch-only search of
MIDI on the internet
many more MIR
systems in Rainer
Typke’s survey.
URL is in your
mailbox
Why pitch-only retrieval is unsatisfactory

information contribution of other 3 parameters
(estimate for Western music; Byrd & Crawford
2001)




pitch: 50%
rhythm: 40%
timbre + dynamics: 10%
melodic confounds (Selfridge-Field 1998):




rests
repeated notes
grace notes, ornamentation
Mozart example
Why pitch-only retrieval is unsatisfactory

information contribution of other 3 parameters
(estimate for Western music; Byrd & Crawford
2001)




pitch: 50%
rhythm: 40%
timbre + dynamics: 10%
melodic confounds (Selfridge-Field 1998):




rests
repeated notes
grace notes, ornamentation
Mozart example
Gestalt and melody

melody: coherent succession of pitches


coherence important for similarity: creates musical
meaning



from New Harvard Dictionary of Music
bottom-up (pitches and durations)
top-down: segmenting, Gestalt
Gestalt theory of perception




late 19th/early 20th century, Germany, later US
perception of wholes rather than parts
explanations: Gestalt principles of grouping
application in visual and musical domain
Low-level Gestalt principles

Snyder mentions:

proximity



similarity




duration
articulation
continuity


rhythmic
intervallic
melodic
these produce closure of
wholes
Example: Beethoven 5th
symphony: beginning 1st
movement

also illustrates high-level
principles
from Snyder (2001)
Low-level Gestalt principles

Snyder mentions:

proximity



similarity




duration
articulation
continuity


rhythmic
intervallic
melodic
these produce closure of
wholes
Example: Beethoven

also illustrates high-level
principles
from Snyder (2001)
High-level Gestalt principles

parallellism


very strong in Mozart, Ah
vous, second half of
melody
intensification


important organisational
principle in variations and
improvisations
Mozart’s last variation
from Snyder (2001)
Application in analysis and retrieval


Gestalt reduces memory
overload: we can ignore the
details
Analytical: Schering (1911)




14th century Italian songs
basic melodic shape
might be nice for retrieval
Problem with Gestalt
principles:



many different formulations
overlap; no rules for conflict
intuitive, cannot be
successfully formalized
from New Grove, Music analysis
The cognitive interpretation: chunking

what creates a boundary





interval leap
long duration
tonality (stable chords)
etc
Example of quantification: Melucci & Orio (2004)

using local boundary detection (Cambouropoulos 1997)



apply weight to intervals and durations
boundary after maximum
chunks forther processed for indexing
Organising chunks

STM problem: max. 5-7
different elements



very short span
solution: hierarchical
grouping
melody schemas

contours of melody




cf. Schering ex.
examples: axial, arch, gapfill
Mozart begins with gap-fill
next level: form

A-B-A
from Snyder (2001)
mental model of a song
Ah, vous dirai-je maman
melody level
A
A
B
analysis
chunk level
synthesis
phrase level
subchunk level

analysis: from ear to LTM

(sub) chunks created by similarity and
continuity


synthesis: from LTM to focus of attention

recollection

a lot of parallellism
boundaries by leaps and harmony


chunks may have a harmonic aspect too
(I, V, V->I)

using general characteristics of phrases and
chunks
performance

notes are reconstitued through some musical
grammar
Problems of melody retrieval

People remember high-level concepts, not notes



melodic variability and change






often confused with poor performance abilities
theme-intensive music (fugues) stimulate formation of such
concepts
transposition
augmentation/diminution
ornamentation
variation
compositional processes: inversion, retrograde
other factors


polyphony
harmony
Set-based approaches to melody retrieval
in polyphony

General idea:




Clausen, Engelbrecht, Meyer, Schmidt (2000):




compare note sets: find supersets, calculate distance
usually take rhythm and pitch into account
hopefully more tolerant agains some of the problems of melodic variety
PROMS
matches onset times; wildcards
elegant indexing
Lemström, Mäkinen, Ukkonen, Turkia (several articles, 2003-4)


C-Brahms
algorithms for matching line segments





P1: onsets
P2: partial match onset times
P3: common shared time
attention to time complexity
Typke, Veltkamp, Wiering (2003-2004)

Orpheus system
Earth Mover’s Distance


The Earth Mover’s
Distance (EMD)
measures similarity by
calculating a minimum
flow that would match
two set of weighted
points. One set emits
weight, the other one
receives weight
Y. Rubner (1998); S.
Cohen (1999)
Application to music


represent notes as
weighted point sets
in 2-dimensional
space (pitch, time)
weight represents
duration


other possibilities
contour/metric
position etc
other possible
application:
pitch event +
acoustic feature(s)?
here, the ‘earth’ is only moved along the temporal axis
Another example

interesting
properties





tolerant against
melodic
confounds
suitable for
polyphony
continuous
partial matching
disadvantage


triangle inequality
doesn’t hold
less suitable for
indexing:
after alignment, the ‘earth’ is moved both along the
temporal axis and along the pitch axis
Test on RISM A/II
Matching polyphony with the EMD



EMD’s partial matching property is essential
MIDI example used as query for RISM database
gross errors in playing are ironed out
Proportional Transportation Distance (PTD)



Giannopoulos &
Veltkamp (2002)
EMD, weigths of sets
normalised to 1
suitable for indexing


triangle inequality
holds
no partial matching
Test on RISM A/II


only hits with
approximately
same length
need 4 queries to
find all known
items
False positive (EMD)

problems arise when length and/or number of notes differs
considerably
Segmenting




overlapping segments of 69 consecutive notes
not musical units
search results are
combined
better Recall-Precision
averages
Example of new search
http://teuge.labs.cs.uu.nl/Rntt.cgi/mir/mir.cgi
Concluding remarks about melodic retrieval

lots of creativity go into melody; difficult to give rules






not a ‘basic musical structure’ (Temperley 2001)
essential to use multiple features
 pitch, rhythm
 harmony
segmentation
 finding perceptually relevant chunks is not easy
 finding complete melodies may be harder
 arbitrary segments may also work
indexing strategies for melody
melodic change over time
several projects have tentative results for polyphony
 gut feeling: false positives are big issue
 notion of salience (Byrd and Crawford)