Transcript Document

QuickTime™ and a
TIFF (Uncompressed) decompressor
are needed to see this picture.
Music and shape:
Exploring cross-modal
representation
Examples, a model and analyses
QuickTime™ and a
TIFF (Uncompressed) decompressor
are needed to see this picture.
Professor Adam Ockelford
University of Roehampton
11th March 2010
Music and shape
How can the two relate?
One set of dynamic
relationships can be
QuickTime™ and a
TIFF (Uncompressed) decompressor
created through
movement:
are needed
to see this picture.
• more or less freely associated with sound
(eg, dance, gestures in performance)
• creating, causing or controlling sound
(eg, playing instruments, conducting)
Music and shape
Another set of relationships can arise through
the visual representation of sound – the
focus of this presentation
QuickTime™ and a
TIFF (Uncompressed) decompressor
are needed to see this picture.
• Seven examples
• Attempt to construct common theoretical
framework
• Indicative analyses
• Discussion
Children’s ‘picture scores’
7 year old’s representation of a rhythm
(after Bamberger, 1982)
Children’s ‘picture scores’
7 year old’s representation of a rhythm
(after Bamberger, 1982)
Blind children’s drawings on
‘German Film’
After Welch (1981)
Blind children’s drawings on
‘German Film’
Conventional Western
music notation
Any features in common with children’s informal
notations?
Braille music
Key differences with print?
Guitar chord tablature
What processes are at work here?
Similarities and differences with previous examples?
Graphic notation
QuickTime™ and a
TIFF (Uncompressed) decompressor
are needed to see this picture.
Excerpt from Stockhausen, Electronic Study No. 2 (1954)
Score produced through
synaesthetic response to
music
Jamie Roberts’s score of Jean Michel Jarre’s Oxygene (Track 4)
How does it all work?
A common underlying
principle or principles?
QuickTime™ and a
TIFF (Uncompressed) decompressor
are needed to see this picture.
Or fundamentally different processes at work?
Perceptual domains
• ‘Domains’ = way of modelling the fact
that one sensation can have different
‘states’
• Touch: eg, temperature and pressure.
QuickTime™
and a
• Vision: eg,TIFFhue,
saturation
and
(Uncompressed) decompressor
brightness. are needed to see this picture.
QuickTime™ and a
TIFF (Uncompressed) decompressor
are needed to see this picture.
QuickTime™ and a
TIFF (Uncompressed) decompressor
are needed to see this picture.
Perceptual domains
In music:
- Pitch: (chroma, octave), harmony, tonality
QuickTime™ and a
- Perceived time:
onset,decompressor
duration
TIFF (Uncompressed)
are needed to see this picture.
- Timbre
- Loudness
- Perceived location of sound source
Different categories of domain
Can be conceived as being
QuickTime™ and a
single dimensional,
bidimensional
or multidimensional
TIFF (Uncompressed) decompressor
are needed to see this picture.
and as being
linear or cyclical or both, or complex
Examples …
Duration
Pitch
QuickTime™ and a
TIFF (Uncompressed) decompressor
are needed to see this picture.
‘Values’
• Each state in which a domain is perceived to exist can be
conceived of as a ‘value’.
• May be simple or complex (like the domains to which they
pertain)
• Example:
Relationships within domains
• Musical structure is not to do with individual
values, but the relationships between them
within each domain - particularly pitch and
QuickTime™ and a
perceived time
TIFF (Uncompressed) decompressor
are needed to see this picture.
• We can assume that the relationships
between values normally pass listeners by as
series of qualitative experiences
• However, they may be conceptualised
(culture specifically?) as differences or ratios,
or they may reflect the complexity of the
values to which they pertain
Examples of intra-domain relationships
‘Primary’ relationships
Examples of intra-domain relationships
‘Secondary’ relationships
Relationships between
domains
• Two main types
QuickTime™ and a
TIFF (Uncompressed) decompressor
are needed to see this picture.
– ‘Regular’ (isomorphic in the domain of ‘shape’)
– ‘Irregular’ (through association)
Regular inter-domain
relationships
Guiding principle:
to map values
systematically
QuickTime™ and a
TIFF (Uncompressed) decompressor
are needed to see
this picture.
between
domains,
a relationship must involve
a function that is
not specific
to the perceptual domains concerned.
Regular inter-domain relationships
For example, differences are domain-specific, so
single values of difference cannot be mapped
systematically between domains.
Regular inter-domain relationships
Whereas, ratios are abstract, are not bound by the
context in which they occur. Hence they permit
regular mapping between domains.
Regular inter-domain relationships
Ratios may occur between differences (as secondary
relationships). Hence the regular inter-domain relationships
would be tertiary.
‘Irregular’ inter-domain relationships
Guiding principle:
formed through association …
either through repetition
or indirect connection
Irregular inter-domain relationships
Different types:
(a) arbitrary (repetition)
(b) indirect (indirect connection)
(c) synaesthetic (indirect connection)
Arbitrary inter-domain relationships
Any relationship between a sound and its
visual representation (shape) is possible.
For example:
>
accent
•
staccato
Quick Time™ a nd a
TIFF ( Un compr ess ed ) de co mp res sor
ar e n eed ed to s ee this pic tur e.
fermata
Arbitrary inter-domain relationships
Example (‘staccato’) – relationship created through
repetition (or verbal explanation = proxy repetition):
Indirect inter-domain relationships
Example (‘note cluster’) – relationship created through
action / object as common visual and auditory source
Can be reinforced through repetition
Synaesthetic inter-domain relationships
b
Example – green corresponds to E major (and vice versa)
Again, can be
reinforced
(for others)
through repetition
From theory to analysis
Children’s ‘picture score’
Simplest interpretation: arbitrary relationships – given shape
(circle) maps onto note
Children’s ‘picture score’
More advanced interpretation: regular mapping of distance
and duration; sequence and temporal order
Score on German Film of pitch glide
Coordinated regular mapping of vertical and horizontal distances
with pitch and perceived time (implied tertiary relationships)
Print score
Arbitrary and semi-regular mappings combine
(the latter reinforcing the former)
Braille score
Only arbitrary mappings survive transcription into Braille
(which makes using this medium more challenging)
Guitar tablature
Combines both indirect and regular representations of pitch
Stockhausen
score:
comprises
regular
representations
of pitch,
perceived time
and loudness
Jamie’s score:
comprises
regular and
indirect
representations
of pitch,
perceived time
and timbre
Conclusions
• Evidence from children’s untutored
representations of music suggests that
sophisticated inter-domain mapping
between sound and shape occurs early
and intuitively
• This view is reinforced by blind
children’s representations of changing
pitch – which are also sophisticated,
and exist in the absence of any visual
model to guide them
Conclusions
• The precise nature of inter-domain mapping
may vary from individual to individual, and
according to cultural convention.
• However, it appears to be relatively easy to
grasp regular mappings (consider, for
example, widespread use of horizontal
dimension for time and vertical dimension for
pitch) or arbitrary symbols (as in standard
notations)
Conclusions
• Some inter-domain mappings may be
hard-wired through synaesthesia.
• These seem to vary from individual to
individual.
• However, such representations can be
learnt and appreciated by others
Conclusions
• What next?
• More exhaustive search of different forms of
symbolic visual representation of sound to
check broad applicability of the model
• Use the model to generate new forms of
sound / shape installations
• More research into how mappings aries and
are learnt, and into sonic/visual synaesthesia
Adam Ockelford
[email protected]
QuickTime™ and a
TIFF (Uncompressed) decompressor
are needed to see this picture.
07818-456 472