Transcript Notes_11_7_a

Virtual Worlds: Audio and
Other Senses
VR Worlds: Output Overview
• Visual Displays:
– Visual depth cues
– Properties
– Kinds: monitor, projection, head-based, handheld
• Aural Displays:
– Aural cues
– Properties, including spatialization
– Kinds: headphones, speakers
• Haptic (touch) Displays:
– Properties
– Kinds
• Other Displays: vestibular, olfactory
Audio Displays: Human Perception
Outer ear (inc the pinna) collects sound as
air vibrations, converts them to mechanical
vibrations in the middle ear (vibrations of
the tympanic membrane produce
vibrations in the 3 bones called the
ossicles) which transmit vibrations to the
inner ear’s cochlea (filled with fluid, motion
of the fluid causes the hairs of the basilar
membrane to vibrate) and then sent via
nerve impulses to the brain
Audio Displays: Sound
• Sound is the propagation of waves (pressure variations)
through a medium; humans hear sounds from 20 Hz
(cycles per second) to 20 kHz; low frequency are deeper
• Sound properties include amplitude, frequency (rate at
which the pressure varies) and phase (where they are in
time), spectral components which determine timbre (type
or quality)
• Sounds have a temporal component; there can be
masking
• Pitch and loudness are perceptions related to the sound
properties (frequency and amplitude); for example, the A
that orchestras tune to is 440 Hz
• Humans can do auditory scene analysis (grouping into
discrete objects) and auditory stream segregation
(isolating a sequence of sounds as one event) and
cocktail party effect
• Localization of sound: ability to pinpoint the source
Questions
• What issues are important for localized
sound?
• When would it be important to have
spatialization of sound?
Cues for Localization of Sound
• Interaural level difference (ILD): different
volumes reach each ear; better for high
frequencies where the head provides
interference
• Interaural time difference (ITD): same
sound reaches the ears at different times;
better for low frequencies (wavelength is
large relative to head size)
Called duplex theory; one problem is the
cone of confusion where ITD is the same;
head movement can help
Cues for Localization of Sound
(con’t)
• People with hearing loss in one ear can
localize so there are other factors
• Doppler effects: intensity increases means
getting closer; intensity decreases =
receding
• Reverberation/reflection
• Acoustic characteristics of speech
• Pinna filtering and head-related transfer
functions (HTRF)
Head-Related Transfer Functions: HRTF
• Pinna filtering (outer ear): distorts
incoming sounds; depends on frequencies
and position of sound
• Experiments done: microphones to
measure the changes that happen from
different locations (~30 years ago)
• Produced mathematical functions to
change the sound
• “Trick” the ear into localization
• Should be individualized but can be
general; not as good when sound is
behind or in low frequency range
Auralization: simulation of sounds
• Room or environment has acoustic properties
(RIR: room impulse response); uses wavebased modeling and geometric modeling (sound
is traced)
• Acoustic environment and the listener together
are the BRIR (binaural room impulse response):
generally done by combining the RIR and the
HTRF, with the majority falling on the HTRF
• Generally use standard HTRF; doesn’t take into
account different size pinna, diff ways to
measure HTRF, perturbation of microphones,
changes in head position
• Another techniques is to have arrays of
microphones to reproduce sounds
Aural Presentation Properties
• Less computationally intensive than CG
• Number of channels: 1, 2 or more
• Sound stage: world referenced
(loudspeakers) or head-referenced
(headphones)
• Localization: HRFT, can simulate
interaural differences, reverb
• Masking
• Amplification
Aural Displays
• Headphones: immersive, cables, HRTF, tracking
needed
• Headphones can isolate listener but
cumbersome and only good for one person and
sounds originate “inside the head”
• Loudspeakers: environment, masking by
projection screens, world-referenced, good for
more people
• Loudspeakers: more difficult to get diff signals
for diff ears, amplitude panning to simulate ILD
(volume)
• Loudspeakers with wave field synthesis; large
number of closely-packed loudspeakers
Aural Logistic Properties
•
•
•
•
•
•
•
Noise pollution
User mobility because of cables
Interference from tracking equipment
Environment needs: reflections
Combination with other displays (visual, haptic)
Portability
Throughput: number of people, time to change
headphones
• Encumbrance
• Safety
• Cost
Vestibular and Other Senses
• Vestibular sense: inner ear – to sense
equilibrium, acceleration, gravity
– Flight simulators, motion platforms, shaking,
low-gravity
– Issues with nausea, dizziness
• Olfactory: difficult to describe, individual
variance
• Taste?
Olfactory Sense
• Hardware to generate odors: inkjet,
solenoid valves, mass-flow controllers,
autosamplers- blending of odor
components
• Odor concentration
• Odor duration and strength: continuous air
flow, air pump, headsets with nose
interface
• Odor sensing an individual differences
Sources
Understanding Virtual Reality by Sherman & Craig, Morgan
Kaufman, 2003
Computer Graphics and Virtual Environments by Slater et
al
Virtual Audio Systems by Kapralos et al, Presence, Dec
2008
Cooking up an Olfactory Game Display by Nakamoto et al,
IEEE Compputer Graphics and Applications, 2008