Transcript Automatic music classification and the importance of instrument

Automatic music classification
and the importance of
instrument identification
Cory McKay and Ichiro Fujinaga
Music Technology Area
Faculty of Music
McGill University
Montreal, Canada
Overview
Examination of the relative importance of
different high-level features in automatic
music classification
Performed an experiment involving
automatic genre classification of MIDI files
Found that features based on
instrumentation (an abstraction of timbre)
were of particular importance
10 March 2005
CIM Montreal / McKay & Fujinaga
2/25
Topics
Introduction to automatic music
classification
Related research
Details of experiment performed
Features used
Feature weighting
Taxonomies used
Classifiers and training data used
Results
Conclusions
10 March 2005
CIM Montreal / McKay & Fujinaga
3/25
Introduction to automatic music
classification
There are many ways in which computers can
classify music
Genre
Composer
Performer
Geographical/temporal/cultural origin
etc.
Music classification can be difficult for both
humans and computers
Rarely have precise, clear and consistent guidelines
delineating the musical characteristics of categories
10 March 2005
CIM Montreal / McKay & Fujinaga
4/25
Applications of automatic music
classification
Discovery of probable authorship of anonymous
compositions
Sociological and psychological research into how
humans construct the notion of musical similarity and
form musical groupings
Automatic sorting of large databases
Music recommendation systems
Sorting of personal music collections
e.g. based on mood or listening scenarios
Automated transcription
Detection of pirated recordings
10 March 2005
CIM Montreal / McKay & Fujinaga
5/25
Advantages of automatic music
classification
Computers can perform classifications
faster and more consistently than humans
Computers can analyze music in novel
and non-intuitive ways that might not occur
to humans
Computers can avoid human
preconceptions that might contaminate
experimental results
10 March 2005
CIM Montreal / McKay & Fujinaga
6/25
How automatic classification works
“Feature” extraction
Properties or characteristics of recordings
Percepts that classifiers base decisions on
Can be extracted from audio (e.g. MP3) or symbolic
(e.g. MIDI) recordings
Good features are essential to successful
classification
Classification can be done using
Expert systems: utilize pre-set heuristics
Machine learning (AI): supervised or unsupervised
learning
10 March 2005
CIM Montreal / McKay & Fujinaga
7/25
Features
Low-level features
Signal processing quantities
e.g. spectral centroid and spectral flux
Can be effective practically
Can have psychoacoustic significance
Have little direct theoretical meaning musicologically
or sociologically
High-level features
Based on musical abstractions
e.g. tempo and meter
Currently difficult or impossible to extract from audio
recordings
Have more theoretical relevance than low-level
features
10 March 2005
CIM Montreal / McKay & Fujinaga
8/25
Overview of this experiment
Empirical examination of which features
are most useful to classifiers
Used high-level features because of their
theoretical significance
Used test task of genre classification
A particularly difficult type of classification
Related to many other types of classification
Features useful for this task likely to be
particularly robust
10 March 2005
CIM Montreal / McKay & Fujinaga
9/25
Related research
Relatively little work has been done on features
that could be useful for arbitrary types of music
Cantometrics project (Lomax 1968)
Tagg (1982)
Cope (1991)
Arden and Huron (2001)
Studied the correlation between musical features and
geographical regions
Automatic genre classification has received
considerable attention recently
Audio classification work of Tzanetakis and Cook
(2002) is often cited
Best results to date with symbolic data have been
achieved by McKay and Fujinaga (2004)
10 March 2005
CIM Montreal / McKay & Fujinaga
10/25
Bodhidharma
Experiments carried
out with the
Bodhidharma system
A general-purpose
symbolic feature
extraction and
classification system
Easy-to-use
Portable
Applicable to a wide
range of research
tasks
10 March 2005
CIM Montreal / McKay & Fujinaga
11/25
Features studied
111 high-level features implemented:
Instrumentation
e.g. whether modern instruments are present
Musical Texture
e.g. standard deviation of the average melodic leap of different lines
Rhythm
e.g. standard deviation of note durations
Dynamics
e.g. average note to note change in loudness
Pitch Statistics
e.g. fraction of notes in the bass register
Melody
e.g. fraction of melodic intervals comprising a tritone
Largest available set of implemented high-level features
42 more features have been proposed, but have not been
implemented yet
More information available in Cory McKay’s master’s thesis (2004)
10 March 2005
CIM Montreal / McKay & Fujinaga
12/25
Features to use
An insufficient number of features can fail to
provide classifiers with enough information to
make good decisions
Too many features can overwhelm and
confuse classifiers
Can be difficult to predict in advance which
features will work well together
Individual performance of a feature is not necessarily
indicative of its performance in combination with other
features
10 March 2005
CIM Montreal / McKay & Fujinaga
13/25
Feature weighting
Feature weighting is a technique for
experimentally determining the importance of
various features by assigning weights to them
Used genetic algorithms here
“Evolves” a good set of weights
The weights produced by the genetic
algorithm provides an indication of the
importance of particular features in particular
contexts
10 March 2005
CIM Montreal / McKay & Fujinaga
14/25
Types of classification performed
The choice of “best” features is context-dependant
e.g. best features for distinguishing between Baroque
and Romantic different than when comparing Punk and
Heavy Metal
Performed three types of classification:
Flat
Hierarchical
Round-robin
Hierarchical and round-robin feature weighting
allowed classifiers to use specialized weightings in
order to improve performance
10 March 2005
CIM Montreal / McKay & Fujinaga
15/25
Taxonomies used
Used hierarchical taxonomies
A recording could belong to more than one
category
A category could be a child of multiple parents
in the taxonomical hierarchy
Performed experiments with two
taxonomies:
Large (38 leaf categories):
Used to test system under realistic conditions
Small (9 leaf categories):
Used to loosely compare system to existing sytems
10 March 2005
CIM Montreal / McKay & Fujinaga
16/25
Large taxonomy
10 March 2005
CIM Montreal / McKay & Fujinaga
17/25
Small taxonomy
Jazz
Bebop
Jazz Soul
Swing
Popular
Rap
Punk
Country
Western Classical
Baroque
Modern Classical
Romantic
10 March 2005
CIM Montreal / McKay & Fujinaga
18/25
Training and testing
Used ensembles of k-nearest neighbour
and neural network classifiers
950 MIDI files
Hand-classified for training based on a variety
of on-line databases
5 fold cross-validation
80% training, 20% testing
10 March 2005
CIM Montreal / McKay & Fujinaga
19/25
Average success rates
Root Genres
Classification Performance
Leaf Genres
9 Category
Taxonomy
38 Category
Taxonomy
Leaf: 57%
Root: 81%
10 March 2005
90
80
Success Rate (%)
Leaf: 90%
Root: 98%
100
70
60
50
40
30
20
10
0
Classical
Jazz
CIM Montreal / McKay & Fujinaga
Pop
Average
Chance
20/25
Success rates achieved in previous
research
Audio results:
Many systems have been implemented
Generally only used 10 categories or less
Success rates generally below 80% for more than 5
categories
Symbolic results:
84% for 2-way classifications (Shan & Kuo 2003)
89% for 2-way classifications (Ponce de Leon & Inesta
2004)
63% for 3-way classifications (Chai & Vercoe 2001)
60-70% for 6-way classifications (Basili, Serafini & Stellato
2004)
10 March 2005
CIM Montreal / McKay & Fujinaga
21/25
Feature performance
Feature Group
Number of Features
Weighting Scaled by Number of Features (%)
Instrumentation
20
46.1
Pitch
25
24.5
Rhythm
30
14.3
Melody
18
11.6
Texture
14
1.7
4
1.6
Dynamics
Features based on instrumentation were assigned
46.1% of all weightings (after scaling)
At least one instrumentation feature played a major role
in almost every classifier
Two of the top three features were based on
instrumentation
10 March 2005
CIM Montreal / McKay & Fujinaga
22/25
Importance of instrumentation
Features based on instrumentation clearly dominant
A high-level abstraction of timbre
Implies that audio classification systems could
benefit from instrument identification modules
Caveat:
These results present the overall averages of weightings
Other features played a dominant role in certain stages of
classification
The best results were achieved by including a wide variety
of features and applying feature weighting
10 March 2005
CIM Montreal / McKay & Fujinaga
23/25
Conclusions
Features based on instrumentation can play an
essential role in automatic music classification,
and should be used if possible
High-level features can produce good results,
and should not be neglected in favour of lowlevel features
Bodhidharma’s large feature library combined
with feature weighting is an effective approach
Very good genre classification success rates can
be achieved with small taxonomies, and we are
at least approaching a point where large
taxonomies can be dealt with effectively
10 March 2005
CIM Montreal / McKay & Fujinaga
24/25
10 March 2005
CIM Montreal / McKay & Fujinaga
25/25