Transcript Document

Musical Acoustics &
Psychoacoustics
Donald Byrd
5 Nov. 2007
Copyright © 2006-07, Donald Byrd
Review: Representations of Music
• Three basic forms (representations) of music
– Audio: most important for most people (general public)
– MIDI files: often best/essential for some musicians,
especially for pop, rock, film/TV
– Notation: often best/ essential for musicians (even
amateurs) & music scholars
– Essential difference: how much explicit structure
• Music holdings of Library of Congress: over 10M
items
– Includes over 6M pieces of sheet music and 100K’s of
scores of operas, symphonies, etc.: all notation!
• Differences are profound
rev. 8 Sep. 2006
2
Review: Representations of Music &
Audio
Digital Audio
Audio (e.g., CD, MP3):
like speech
Time-stamped
Time-stamped
Events Events
(e.g., MIDI file): like
unformatted text
Musiclike
Notation
Music Notation:
text with complex
formatting
1 Sep. 2006
3
Review: Rudiments of Musical
Acoustics
• Need some musical acoustics for almost
anything in digital audio
• Acoustics: branch of physics that studies
sound (of any kind)
– Concepts like frequency & amplitude
• Psychoacoustics: study of how sound is
perceived; mostly psychology
– Concepts like pitch, loudness, timbre
1 Sep. 2006
4
Materials for Studying Audio
• What are interesting sounds really like?
– Sine waves, etc. are boring (cf. addsynenv)
– Sounds of acoustic instruments are “rich”
– Vary in every way: with pitch, loudness, time
• Musical instrument samples
• Audacity audio editor
– For Windows, Mac OS 9 and X, Linux
– Download from
http://audacity.sourceforge.net/
• Programs in (e.g.) R
10 Sept. 2006
5
Review: Parameters of Musical Sound
• Four basic parameters of a definite-pitched
musical note
1. pitch: how high or low the sound is: perceptual analog
of frequency
2. duration: how long the note lasts
3. loudness: perceptual analog of amplitude
4. timbre or tone quality
• Above is decreasing order of importance for
most Western music
• …and decreasing order of explicitness in CMN
(Conventional Music Notation)!
10 Sept. 2006
6
Scholars (and others) Beware! (1)
• Plausible (at the time!) assumptions
– Men have more teeth than women (ancient)
– Diseases can’t be transmitted by invisible
organisms (19th century)
– Stomach ulcers can’t be caused by organisms
(20th century)
• What you expect & what you see/hear
– A recent discovery about kitchen sponges
– 1923 New York Times headline: “Dinosaurs
Cavort in Film for Doyle”
rev. 31 Oct. 2007
7
Scholars (and others) Beware! (2)
• What you expect & what you hear
–
–
–
–
Don & the Kurzweil 250 flute sound
Hammond organ vs. a real pipe organ
Don, a famous musician, & K250 handclaps
Huron on what he “knew” & learned
• R. Moog at Kurzweil & piano touch
rev. 20 Sep. 2006
8
Musical Acoustics (1)
• Acoustics involves physics
• Musical (opposed to architectural, etc.) acoustics
–
–
–
–
Frequency (=> pitch)
Amplitude (=> loudness)
Spectrum, envelope, & “other” characteristics => timbre
Partials vs. harmonics
• Psychoacoustics involves psychology/perception
– Perceptual coding (for “lossy” compression: MP3, etc.)
• Timbre
– Old idea (thru ca. 1960’s?): timbre produced by static
relationships of partials, plus envelope
– …but attack often more distinctive than “steady state”!
– Rich (interesting) sounds are complex; nothing is static
– Time domain (waveform) vs. frequency domain (spectrum,
spectrogram) views
rev. 11 Sept. 06
9
Materials for Studying Acoustics
• Most useful
–
–
–
–
–
–
Piano (preferably a grand)
String instruments (preferably large & bowed)
Drinking glasses filled to different levels
Simple percussion: sleighbell, “rainegg”, etc.
Recording equipment
Audio editor, spectral-analysis program
• Also helpful
–
–
–
–
Other musical instruments
Tuning fork
Strobe light
Echoic & anechoic chambers
rev. 31 Oct. 07
10
Modes of Vibration &
Simple Vibrating Systems
• Waves can be transverse or longitudinal
– Which is water? Which is sound?
• Simple vibrating systems & waveforms
– Tuning fork
• Not many other examples!
– What’s the waveform?
4 April 07
11
Complex Vibrations & Resonance
• Modes of vibration
–
–
–
–
Air column closed at both ends vibrates at f, 2f, 3f…
…where f = frequency determined by length
Closed at one end, open at other: f, 3f, 5f…
…where f = half the frequency of previous case
• Resonance
– “When a system that can vibrate at a certain frequency is
acted on by periodic disturbance at same frequency,
vibrations of large amplitude can be produced.” —Backus
– Ex: swing in playground
– Ex: string & body of musical instrument
– Resonance: tendency of a system to oscillate at maximum
amplitude at a certain frequency.
4 April 07
12
How String Instruments
Make Sounds
• Vibrating strings
• Waveform is almost a sawtooth => lots of
harmonics!
• Bowing
– Rosin to increase friction
– Special effects: sul ponticello, col legno, etc.
• Plucking
4 April 07
13
How Woodwinds Make
Sounds
• Vibrating air columns
• Without reeds: flute
– Acts like open column
• With reeds: oboe, clarinet, bassoon
– Clarinet acts like half-open column
– Why?
– Same length => an octave lower
• Overblowing
• What about brass instruments?
4 April 07
14
How Percussion
Instruments Make Sounds
• Guess
4 April 07
15
Acoustic Phenomena
• Sympathetic vibration
• The harmonic series
• Beats
– Chorus effect
• Harmonics (on bowed strings) show:
– Modes of vibration
– Nodes
– Relationship to sul ponticello?
• Architectural acoustics
– Resonance (as in I Am Sitting in a Room)
– Standing waves
31 Oct. 07
16
Psychocoustic Phenomena
• A matter of psychology (and, to some extent,
music theory)
• Basis for auditory illusions
– Endless glissando
– “Mystery melody”
• Practical application: perceptual coding used in
lossy compression (MP3, WMA, etc.)
– Masking
– Threshhold of audibility
4 April 07
17
Creating Interesting Sounds (1)
• Periodic, nearly-, & non-periodic signals
• Approaches to creating sounds
–
–
–
–
Additive synthesis: like painting
Subtractive synthesis: like sculpture
Sampling: like collage
Others: modulation, physical modeling
• Most possible with analog or digital
hardware, but analog is limited
11 Sept. 2006
18
Creating Interesting Sounds (2)
• Cf. “Electronic Music Tutorial” (Ishkur)
• Additive Synthesis
• Fourier's Theorem
– Any periodic signal => sum of harmonically-related
sine waves
• Envelopes: continuous & piecewise linear
– Important special case: “ADSR”
– Attack, Decay, Sustain, Release
• Phase, interference, & beats
– Phase by itself rarely important, but relationships are
– Diagrams & demo in CECM Acoustics Primer, Sec. 8
– Interference between channels or very close partials
rev. 13 Sept. 2006
19
Additive Synthesis & Envelopes
• addsynenv does additive synthesis of up to six partials
–
–
–
–
–
Each has arbitrary partial no., starting phase, "ADSR" envelope
Partial no. can be non-integer => not harmonic
ADSR = Attack/Decay/Sustain/Release (3 breakpoints)
…but addsynenv allows much more complex envelopes
Plays one note with waveform specified by partial nos. & their
envelopes (maybe also phases)
– Simultaneously displays “spectrogram” or “sonogram”
– …but not waveform
– Phase in real world normally has little effect, but can be
critical in recording & digital worlds (e.g., cancellation)
• Additive synthesis can’t create aperiodic (non-definite
pitch) sounds
• ...or many realistic attacks
rev. 13 Sept. 06
20
Creating Interesting Sounds (3)
• Early CCRMA (Stanford) studies of acoustic
instrument sounds
– Envelope for each partial with a few segments
– Similar to addsynenv
• Subtractive Synthesis
• Early (analog: Moog, etc.) synthesizer model
• Signal source: sine/square/sawtooth (for
additive)
• …or noise & filters (for subtractive)
• LFO to add vibrato, tremolo, glissando, etc.
• ADSR envelope for entire sound
11 Sept. 2006
21
Periodicity & Definite Pitch
Periodic waveform: clearcut
definite pitch
With sharp corners: much
high-freq. energy
square wave; loud trumpet
Without sharp corners: little sine wave; low, soft flute
high-freq. energy
Almost-periodic waveform:
definite pitch
piano
Somewhat periodic
waveform: complex or
multiple pitches
bells
Aperiodic waveform: noisy,
no definite pitch
cymbal, bass drum
31 Oct. 07
22
Real-World Musical Sounds (1)
• The “Attack/Sustain/Release” (ASR) model for notes
– Attack, Sustain, Release modified from recordings
– Attack includes Decay from ADSR model
• Used in the Kurzweil 250 (1984), etc.
–
–
–
–
Original version had only 2 MB for all samples
Piano had diff. samples for 2 loudness levels
…and diff. sound for every 4-6 semitones
1-2 sec. per sample for A+S+R
• How good did the K250 really sound?
– COUNTDOWN, by Christopher Yavelow
– “An opera for the nuclear age”
• “the ‘orchestral accompaniment’ is in reality a Kurzweil250 digital sampler, synchronized to the baton of the
conductor…”
– http://www.yavelow.com/docs/countdown.html
12 Feb. 07
23
Real-World Musical Sounds (2)
• Nowadays, can afford “unlimited” sustain
• …but also need diff. sounds for many (8?) diff.
loudness levels (multisampling)
– “All Together Now”, Electronic Musician, Jan. 2007
• …and diff. sound for every semitone or two
• W/ unlimited sustain, takes gigabytes just for
piano!
19 Feb. 07
24
Timbre Space & Envelopes (1)
• Timbre space
– Includes all possible timbres
– Not well understood; may be 3-dimensional
• Original “FourierRandomTimbre” covers only a
tiny part
• Some ways to enlarge it:
– Overall envelopes
– Allow (inharmonic) partials
– Envelopes for partials
• Formants show timbre involves context: not
really a property of a note!
• Timbre as "the psychoacoustician's
multidimensional wastebasket category"
29 Oct. 07
25
Timbre Space & Envelopes (2)
• Envelope
– Cf. Wikipedia article on timbre
– Amplitude structure of a sound, so called because sound just
"fits" inside its envelope
• Piecewise-linear envelope
– Envelope that can be drawn with a relatively small number of
straight lines and no curves
– Important in music synthesis because can approximate many
complex real-world sounds well with very little data
– Special case: "ADSR" (Attack - Decay - Sustain - Release)
envelopes; common on synthesizers, etc.
– Samplers may combine A & D
29 Oct. 07
26
Uncompressed Audio Files are Big
• 1 byte = 8 bits (nearly always)
• How much data on a CD?
– CD audio is 44,100 samples/channel/sec. * 2
bytes/sample * 2 channels = 176,400
bytes/sec., or 10.5 MByte/min.
– CD can store up to 74 min. (or 80) of music
– 10.5 MByte/min. * 74 min. = 777 MBytes
– Actually more: also index, error correction
data, etc.
22 Sep. 2006
27
Compressed Audio: Lossless & Lossy
• Don’t confuse data compression and dynamicrange compression (a.k.a. audio level
compression, limiting)
• Codec = COmpressor/DECompressor
• Lossless compression
– Standard methods (LZW: .zip, etc.) don’t do much for
audio
– Audio specific methods
• MLP used for DVD-Audio
• Apple & Microsoft Lossless
• Lossy compression
– Depends on psychoacoustics (“perceptual coding”)
16 Feb. 06
28
Specs for Some Common Audio Formats
Format
Encoding Type
Details
Fidelity
“Red Book” (CD)
Uncompressed,
linear
44.1KHz, 16 bit s/sample,
st ereo
Very high
Bandwidth
(Kbps)
ca. 1400
Early game audio
Uncompressed,
linear
22.05KHz, 8 bit s/sample,
mono
Low
176
MLP, Apple
Lossless
Compression, etc.
MP2 (Variat ions1)
Lossless comp.
compression ca. 2:1
Very high
ca. 700
Lossy comp.
compression ca. 3:1
High
ca. 400
MP3 (Variations2),
AAC, WMA
Lossy comp.
compression ca. 7:1 to
over 10:1
High t o
very high
ca. 128-192
compression more than
20:1
Medium
ca. 28-64
AAC (Variations2), Lossy comp.
WMA
13 Feb. 06
29
Psychoacoustics & Perceptual Coding
• Pohlmann, Ken (2005). Principles of Digital
Audio, 5th ed., Chapter 10: Perceptual Coding
• Rationale: much better data compression
• Physiology of ear and critical bands
– Not fixed frequency: any sound creates one or more
critical bands
• Masking
– Depends on relative loudness & frequency
– Noise is much better than pitched sounds
• Threshhold of hearing
– Depends greatly on frequency
22 Sep. 2006
30
Compressed Audio: Lossy Compression
• General method
1. Divide signal into sub-bands by frequency
2. Take advantage of:
• Masking (“shadows”), via amplitude within
critical bands
• Threshhold of audibility (varies w/ frequency)
• Redundancy among channels
• MPEG-1 layers I thru III (MP-1, 2, 3), AAC get better &
better compression via more & more complex techniques
– “There is probably no limit to the complexity of
psychoacoustics.” --Pohlmann, 5th ed.
– However, there probably is an “asymptotic” limit to
compression!
• Implemented in hardware or software codecs
22 Feb. 06
31
Another “Analog vs. Digital” Battle
• Merton, Orren (2006, February). The Sum of All
Tracks. Electronic Musician 22,2, pp. 57-63
• About summing (mixing) with analog vs. digital
hardware
• Tested with excellent hardware, panel of experts
• Methodology not really scientific
• But is this fair? Does it really matter? How can
you do better with subjective phenomena?
– Electronic Musician isn’t a scholarly journal
– Critical: blind study => results probably meaningful
– Cf. http://createdigitalmusic.com/2006/09/06/analogsumming-pm8-for-people-who-dont-trust-softwaremixing/
rev. 25 Sep. 2006
32
Scientific Evaluation of Subjective
Phenomena: Lossy Compression (1)
• Pohlmann, Ken (2005). Principles of Digital
Audio, 5th ed., pp. 403-413
• Discusses perceptual coding performance
evaluation
• Want to evaluate as objectively as possible!
• One idea: measure artifacts
– Difference between original & compressed =>
distortion
– Psychoacoustic model can estimate NMR (Noise-toMasking Ratio)
25 Sep. 2006
33
Scientific Evaluation of Subjective
Phenomena: Lossy Compression (2)
• Pohlmann: “best way to evaluate is to
exhaustively listen”
• Tests with excellent hardware, panel of experts
• …but serious scientific methodology
• Double-blind, ITU-R methods, CCIR 5-pt scale,
>=50 subjects, etc.
• Similar methods used in validation of MP3, AAC,
etc.
• …and ff123 et al’s evaluation of Ogg Vorbis
• Our listening test: at
http://www.informatics.indiana.edu/donbyrd/N
560Site-Fall06/Worst/
25 Sep. 2006
34
Auditory Illusions 1
• Wikipedia article “auditory illusion”
• Deutsch's Musical Illusions
– E.g., Mysterious melody, Chromatic illusion
– http://psy.ucsd.edu/~ddeutsch/psychology/d
eutsch_research1.html
• Visual and Auditory Illusions (from UBC)
– Often analogous illusions
– E.g., Shepard's Tones, Tritone Paradox
– http://www.cs.ubc.ca/nest/imager/contributi
ons/flinn/Illusions/Illusions.html
25 Sep. 2006
35
Auditory Illusions 2: Note, Chord, or
Scale?
•
•
•
•
•
Issue is partials each with its own envelope
Clue: relative frequencies of partials
Clue: shapes of envelopes
Try to simplify the sound…
Do you experience fusion?
rev. 25 Jan. 2007
36
Auditory Illusions: Shepard’s Tones
• Also called (less accurately) endless glissando
• Demo on Web
– http://www.cs.ubc.ca/nest/imager/contributions/flinn/Illusions/Illu
sions.html
– Wikipedia version has a more interesting timbre
• Again, it’s partials & envelopes
• Clue: relative freqs. of partials are important,
unusual
• Clue: envelopes of partials are important
• Used by J.C. Risset in computer-synthesized
music
– in one case, with rhythm equivalent
rev. 5 Nov. 2007
37
Visual Analogue of Shepard’s Tones
rev. 25 Jan. 2007
38
Auditory Illusions: Mysterious Melody
• Hard to recognize, unless you already
know what it is!
• Clue: probably easier for Americans
rev. 5 Nov. 2007
39