Transcript How Sound Can Aid The Information Seeker In

How Sound Can Aid The Information
Seeker In Information Retrieval...
Or...
If You Can’t See The Forest For The Trees,
Will The Sound Of The Tree Falling On You
Help?
Dennis Hage
Sherry Meredith
KEY QUESTIONS
• How can sound help in information
retrieval?
• What is the difference between
speech, non-speech, and music?
• Who benefits from audio-based
information retrieval?
Sonically enhanced retrieval
allows for more natural
communication between computer
and user, allowing users to employ
two senses to solve a problem.
- Stephen Brewster
Definitions
• Sound is produced when the air is
disturbed in some way (a vibrating
object).
• Sonification - The use of non-speech
audio to
convey information.
Sonification Versus Speech
• Speech:
– Slow speed & serial
– Language dependent
– Many predigested words
• Non-speech sounds:
– Similar to graphical icons
– Can quickly represent a concept
Definitions (cont.)
• Auditory Display - Non-speech audio to
convey information in the computer
interface.
• Auditory Display techniques
– Sonification
– Auditory Icons
– Earcons
Sonification
• Sound generation that need not have any
direct relationship to the data being
generated.
• Removing the display one more step from the
data, may also make it richer and more
coherent.
Why Use Sound?
“Who the hell wants to hear actors
talk?”
- Harry Warner
(on the future of “talkies”- movies with sound)
Why Use Sound? (Cont.)
• Archaeologists of the future
• Human communication is primarily carried out
via speech
• Audio cues in our day to day lives - car honks
to computer beeps to angry dog snarls.
Why Use Sound? (Cont.)
Acceleration of auditory display
interest:
•
•
•
•
Information theory
Powerful and affordable computers
Sound synthesis technology
All of the above (synergy)
Why Use Sound? (Cont.)
The increasing number, size, and
complexity of data sets challenge existing
visualization techniques.
• Seismic data overview 24 hours in several
minutes.
• Difficulty sorting out relationships as the
number of variables to examine increases.
Sounds like, You know…
• ...time itself…is well-suited to represent
temporal information.
• Sound can be used to:
– Detect changes over time
– Detect anomalies in data
– Replace a distracting visual element
on a map display.
• Example: Pitch is used to replace the
time bar.
Why should we use sound at
all?
• Less interference between tasks using
different senses.
• Individuals who are deprived of vision.
• Signaling the ongoing status of background
activities.
• Can also provide a sense of place.
• Insistent quality
“Today’s biology suggests that
it’s the arts that lay the
foundation for later academic
and career success.”
• Eric Jensen, Teaching with the Brain in Mind.
Learning is discovering what
is already built into our brain
that will allow us to respond to
the immediate challenge.
Brain
Model
Neural Networks similar
to jungle organisms:
• Respond variously to
environmental challenges.
• Made up of highly connected
components.
Cortex
Limbic
System
Cortex
(reasoning/logic)
Brainstem
Cerebellum
R. Sylwester, A Celebration of Neurons
• Cortex size of
large dinner
napkin.
• Contains about 10
billion neurons.
Neuron
• Soma-body
• Dendrites-receptor
sites
• Axon-releases
neuro-transmitter
• Synapse
G. Edelman, Bright Air, Brilliant Fire
Columns of Neurons
Six
Cellular
Layers
Key to
Getting
Smarter:
Area of web of
axon
connections
R. Sylwester, A Celebration of Neurons
• Grow more connections between cells.
• Not lose existing connections.
Brain customizes itself.
• Frequent behavior and thought
patterns stimulate connections.
• This creates dominant pathways
for nerve impulses.
• Unneeded cells and unused
connections get pruned away.
Modular Brain
• Interconnected combinations of neural
networks process more complicated,
related phenomena,
– from sounds to phonemes to words,
– from lines to triangles to pyramids.
Two critical ways
to enrich the brain:
• Make learning challenging.
• Provide interactive feedback.
Challenging
sensory stimulation
is a brain “Nutrient.”
Brain is artist and scientist:
• Attempts to discern
and understand patterns.
• Creates patterns of its own.
Facts and skills in isolation
• are organized differently by the brain,
• need more practice and rehearsal.
Feedback
• reduces uncertainty, lowers stress.
• most useful when immediate
and self-generating.
• should involve choice.
“Doing worksheets in school
prepares a student
emotionally to do worksheets
in life.”
• Robert Sylwester, A Cellebration of Neurons
Learning in context
• Brain tends to string things
together:
– notes into songs,
– steps into dances,
– events into stories,
– actions into games.
• Robert Sylwester, A Cellebration of Neurons
Cerebellar system
• first system to mature
(along with inner ear).
• information gathering and
feedback source for movement.
• filters and integrates incoming data for
complex decision making.
When several senses
are used to register
objects and events:
• information is stored in several
interrelated memory networks,
• memory is more accessible and
more powerful.
“Processes we had
considered pure ‘thinking’ are
now seen as phenomena in
which the cognitive and
emotional aspects work
synergistically.”
Elias, et al. Promoting Social and Emotional Learning.
It Depends?
• In some cases, redundant information is
less effective than audio-only
information to the phenomenon of visual
dominance. (Tzelgov et al, 1987)
• Our sense of vision often seems much
more dominant than our sense of
hearing. (Ackerman 1990, Tuan 1993)
It Deep ends? Don’t Go
Overboard: Sound Quality
Problems
•
•
•
•
•
•
•
•
Low resolution of many auditory variables
Limited spatial precision
Lack of absolute values
Annoying
Interferes with speech communication
Absence of persistence
No printout
User limitations: The aural equivalent of color
blindness
Sound Advice
• Visualization is often insufficient for
comprehending certain features in the data
(Voyager 2 data analysis).
• Need to comprehend an abundance of data
... need to interpret and understand.
• Audio’s natural integrative properties are
increasingly being proven suitable for
presenting high dimensional data without
creating information overload for users.
Early History History Uses Titanic Sound Bites
• Sonar.
– Leonardo da Vinci (1490)
– Titanic (1912).
Early History History Uses
• Drums and horns
• Morris code
• Diagnostic listening - mechanics, doctors
• Programmers
Some other successful ones...
•
•
•
•
•
•
Hans Geiger in early 1900s...
Pulse-oximeter - 1980s. oxygen in a patient’s blood
Voyager 2 - rings of Saturn.
Quantum Whistle
Cartography
Yeung (1980) investigated sound as a means of
representing the multivariate data common in
chemistry
• These cases illustrate the ability of the auditory
system to extract underlying structure and
temporal aspects of complex signals that are
often important in scientific exploration and
YOU AUTO KNOW BETTER
• Imagine driving to work, listening to
auditory cues, navigate to the file menu
which has a print option. You print reports
so that they are waiting for you when you
arrive. Start your email application and
listen to any new email messages, delete
and file others.
YOU AUTO KNOW BETTER
• Audio glance presents a similar
overview aurally rather than visually of
the overall properties of an object.
Specifically, email messages.
Sound to Drive by... Maps
• John Krygier (1994) identified a range of sound
variables that are similar to graphic variables:
– Location:
Location of a sound in a two- or three- dimensional
sound space.
– Loudness: The magnitude of a sound.
– Pitch:
The highness or lowness (frequency) of a sound.
– Register: The relative location of a pitch in a given range of pitches.
– Timbre: The general prevailing quality or characteristic of a sound
– Duration: The length of time a sound is (or is not) heard.
– Rate of change: The relation between the duration of sound and
silence over time.
– Order: The sequence of sounds over time.
– Attack/decay: The time it takes a sound to reach its
maximum/minimum.
Pitch and Curve
One Sound Use
• Changes in magnitude of a single variable (pitch).
• Pitch - frequency of a simple tone. It is the feature
of a sound by which listeners can arrange sounds
on a scale from “lowest” to “highest”.
• CHILDREN BY AGE 10...
• Pitch variation over time may be nearly as effective
as variation in curve height on a visual graph.
• May be better than tactile for blind users.
Auditory Icons
• Caricatures of naturally occurring sounds which convey
information by analogy with everyday events. (Gaver, 1986)
• 2 advantages:
– Users tend to be more engaged.
– Increases flexibility, users don’t have to always attend to
the screen for information.
Auditory Icons
•
•
•
•
Heighten awareness of a distributed community
Mapping between the sounds and their meaning
Sources not sounds
Advantage is also its weakness... no real-world
analogy such as renaming a file.
• Frivolous? -- but how about visually impaired or in
critical applications where sound can reduce risk of
error...
MUSIC
• Pitch - primary basis for traditional melody
• Rhythm- relative changes in the timing of the
attacks of successive events
• Tempo- the speed of events
• Dynamics - the relative loudness of events (static
or varying)
• Timbre - the difference of spectral content and
energy over time (differentiates a saxophone from
a flute)
• Location - where the sound is coming from
Earcons (Micons and Picons)
• Earcons are “non-verbal audio messages that are used in
the computer/user interface to provide information to the
user about some computer object, operation or
interaction.” (Meera M Blattner, 1989)
• Based on synthetic musical tones.
• Earcons are constructed from simple building blocks
called motives. (1-3 notes long)
Earcons (Micons and Picons)
• By using several dimensions of sound, such as pitch,
timbre, and rhythm, information can be communicated to
the user.
• Hudson and Smith (1996) , have constructed longer
earcons which convey useful information about incoming
email messages, referred to as audio glances.
• Tool Palettes (Brewster)
• http://ftp.dcs.glasgow.ac.uk/~stephen/demos.sht
ml
Virtual limitations
Virtual limitations
• One of the main deprivations caused by
blindness is the problem of access to
information.
• This information bottleneck must be
overcome in order for visually disabled
users to access information.
• As GUI has become the standard
developing alternative modes of interaction
for the blind becomes even more important.
Easier for sighted people, disadvantage for
impaired.
In Search of Kramer...
• The continued trend towards complex
tasks and data challenge our abilities to
make sense of things via abstractions
such as numbers and categories.
Perceptual presentations such as
sonification present a means for tapping
innate meaning-making skills. Matching
these perceptual and cognitive capacities
with appropriate tasks and designing
effective display systems to effect this
match is going to be a large and
THE FUTURE IS HEAR...
Or…
Not GUI…
But
SUI (Sound User Interface)
When Pigs Can Fly, Will you
hear them squeal?