Transcript Sonification and Data Representation

Sonification and Data
Representation
John E. Bower
Department of Music
Sonification
Concept
Tools
Implementation
Concept
Sonification is the representation of some data through
metaphorical structures of sound
Absolute, quantitative data is interpreted qualitatively
The ideal is overt, but reticent immediacy and clarity of
message
Interpreting the realization can be a learned activity
A successful sonification should efficiently convey the metrics
of the source data without an inappropriate amount of user
fatigue
Concept
Ambient Device’s
Orb
Concept
John Hancock Building
Boston, Mass.
Tools
Michael Winslow?
Csound
Progeny from the Music-N languages developed by Max Mathews et al at Bell
Labs, Princeton, and Stanford Universites in the 1960s
Csound evolves in the 1980s under Barry Vercoe at M.I.T.
Freeware GNU LGPL
Runs on most every platform, from an Amiga to a Sharp Zire
Proprietary “assembly” language
“Batch” processing paradigm: sound is rendered from a pair of text files, one
denoting user-designed “patches”, the other the parameters to inform those
patches
Most mature computer music app with close to 500 opcodes
Not ideal for real-time use
Max/MSP | jMax
Original author: Miller Puckette
Initially developed at IRCAM, Paris
Available commercially as Max/MSP for Mac/Win32
jMax is the open-source (GNU GPL) version implemented in Java with a C-based
rendering engine for OSX, Linux, Win32, IRIX
Created specifically for real-time use
Graphical “programming” paradigm
Extensible by external C plugins
Well-supported and used by musicians, media artists, clinicians, architects, etc.
“Clone” app, PD (Pure Data), by Puckette available as an open-source app for
OSX/Linux/IRIX/Win32
Max “graphical” patch
SuperCollider
Original author: James McCartney
Originally a commercial app; open-source (GPL) with SuperCollider3
Available for OSX/Linux; Win32 port somewhat likely
Object-oriented programming language based upon Smalltalk but with C language
family syntax
Ideal for real-time use
Operates on a distributive paradigm (not shared-memory) facillitated by OSC
networking protocol
Extensible with Smalltalk/Objective C classes, C++ plugins
Arguably the most powerful/efficient language to date: “bleeding edge”
Documentation? What’s documentation?????
Implementation
Much like a musical composition, elements of a
sonification may include:
Pitch and register
Rhythm and tempo
Phrase and cadence
Timbre and orchestration
“Modality”
Spatialisation
The sonification may be overtly musical or ambient
(musical or naturalistic) in design
Research: examples
Peep
Peep, a network “auralizer” by Michael Gilfix
First presented in a paper at Usenix, 2000
Discrete events are mapped to short, naturalistic sounds; for instance, a user
logging in may be represented by a cricket chirping
Continuously variable states such as system load are mapped to continuous sounds
like water or wind
Research: examples
IBM Research
Conducted by IBM Research Division’s David A. Rabenhorst et. al.
System for complementary visualization and sonification of semiconductor
models in the FIELDAY program
Program examines any two of the electrostatic potential, electron
concentration, and hole concentration of the model in complementary
windows
A 3D cursor navigates one window visually while the sonification generates
sound based upon the 3D vector gradients of the chosen scalar field at the
same focal point in the solid as displayed in the second, non-focused, window
Allows the user to simultaneously concentrate on one data field visually and
one aurally
Research: examples
IBM Research, continued
The scalar data field is sonified by a musical triad with voice doubling on each
individual component (six voices total)
Each signed component of the 3D vector gradient maps to both the detuning and
stereo location of a note in the triad
Rough precision of each vector gradient (x, y, z) is measured by the panning
(direction and intensity) of the doubled-tone, fine precision by the intonation
difference (up to +- one semitone) between the two notes
The three structural components of the triad are assigned sounds with different
spectral characteristics to facilitate differentiation between the voices
Static vector gradients have their volumes attenuated so they are not as present in
the sonification as actively-changing gradients
Research: examples
Spatial Data Mining
Tools developed at FSU by Myke Gluck for visualization/sonification of GIS data
Tools are intended for general users
One represents data through melodies that undergo variations in tempo, key,
articulation, and embellishment
Others utilize:
Range depiction (pitch, volume, timbral quality: “brightness”)
Multiple tones (octaves represent data classes, particular notes within an
octave represent data values)
Different musical scales (diatonic, pentatonic, etc.)
Different chord qualities (major, minor, etc.)
Research: examples
Life Music: Protein Sonification
Collaboration between biologist Mary Anne Clark and artist John Dunn at Texas
Wesleyan University
Sonification designed with the following:
Alpha and beta regions of proteins differentiated by changes in
instrumentation and/or pitch
Pitch material determined by assigning a fixed pitch to each amino acid, or
assigning pitch based on a frequency histogram of the amino acids with the
more “consonant” intervals representing the most frequently occurring amino
acids
Register determined by the water solubility of the amino acids, with the most
insoluble getting the lowest pitches, the most soluble the highest
Research: examples
Other research
IBM’s Sonnet “visual” programming language for interface design.
Possessed features for adding sound to interfaces to monitor data flow and
limit GUI clutter.
Used internally at IBM. Never materialized commercially?
Spatialisation of air traffic data with aural cues for aircraft proximity, rate of
approach, various “emergency” situations, etc. (NASA Ames Research Center)
Various tools designed to monitor smog conditions in Los Angeles, Yellowstone
forest-fire data; monitoring of patient blood oxygen level, blood pressure, and other
vitals for operating theater use; sonification of nuclear power plant control room
data; stock and other financial trends
Implementation
Short user study of sonification examples, each
utilizing a different means of interpreting data
Indicate on the sheet your perception of either
an increasing, decreasing, or unvarying metric
for each example
Please rate each example with a “fatigue”
level from 1 to 10 as well as a clarity ranking
from 1 to 10
At the end of the sheet, please indicate if you
have demonstrable musical training
Sonification and Data
Representation
John E. Bower
[email protected]
www.stillmovement.org