Transcript PSYCHOLOGY AND INFORMATION SYSTEMS

6. Sound, Language, Thought &
Sense Integration
I think I know what you mean
Brian Whitworth
17 July 2015
© 2001 Brian Whitworth
1
Aim
How does the brain represent sound and
other sense information and combine it?
How does this affect the design of multimedia systems?
17 July 2015
© 2001 Brian Whitworth
2
Gregory: Hearing, p303
Sound?
• Closely connected to emotions
• Based on air pressure waves on the eardrum
Pressure
Frequency (2 cycles per second or 2 Hertz) ~ Pitch
Amplitude
(decibels)
~ loudness
Time
Wave Shape (this is a pure tone) ~ timbre
17 July 2015
© 2001 Brian Whitworth
3
Loudness
• Measured in decibels = Log of air pressure
– Double loudness = +10db = x10 air pressure
– 1 db = just detectable , 60-70db = conversation,
100 db or above = uncomfortable
• Loud sounds can threaten, be unpleasant or
embarrass, so volume control is critical:
– A real estate shop’s computer sales system is too loud, &
has no volume control, the user response is:
• Says “Is it talking to the whole neighborhood?”
• Steps back a few paces embarrassed, then walks away
17 July 2015
© 2001 Brian Whitworth
4
Pitch
• Pitch is the repetition rate of the wave (frequency)
– Hear from 20Hz (very low) to 20,000Hz (very high)
– Each musical note has a certain pitch
• Application:
– Low sounds associate with men, with large musical instruments
(bassoon) or animals, and hence with strength
• E.g The Terminator & Darth Vader have low voices
– High sounds associate with women or children, with small
instruments (piccolo) or animals, and hence less harmfulness,
– Lower pitched voices/sounds if bad can be more threatening, if
good can be more reassuring, higher pitched voices are less
threatening of harm, but can be taken less seriously
17 July 2015
© 2001 Brian Whitworth
5
Timbre
• Most sounds are not pure sine waves
• The “shape” of the pressure wave distinguishes a
piano from a violin playing at the same pitch and
volume
• Applications:
– Each person or creature’s voice has a unique timbre that is
recognizable
– We deduce from voice if the person is male or female, old or
young, their culture, race, country, and status
– Choose the voice to fit the listener (e.g. adult voice for a
children’s tutorial, but child’s voice for a children’s game)
17 July 2015
© 2001 Brian Whitworth
6
Fourier analysis
Gregory: Fourier
• Complex sounds can be broken down into pure
sine waves which recombine to give that sound
• The ear appears to carry out a fourier analysis we hear a chord as a set of notes combined, not a
new sound, we can hear a frequency spectrum
– Sounds with high frequencies in them sound “sharp”
– Sounds with low frequencies in them sound
“mellow”
17 July 2015
© 2001 Brian Whitworth
7
Ear - a frequency analyzer
• Ear’s parts
– Ear drum: membrane vibrates with air pressure changes
– Inner ear bones: amplify vibration (adjustable) to ..
– Cochlea: a fluid filled spiral whose base, the basilar membrane,
vibrates
• Like a guitar string to match the frequency (for low sounds)
• In different places for different frequencies (for high sounds)
– Sensitive hair cells along the basilar membrane create sound
– With age lose ability to hear higher frequencies as those cells
are lost and not replaced
• Application: Lower frequencies are more easily heard
17 July 2015
© 2001 Brian Whitworth
8
Gregory: Music, p499
Harmony - frequency combinations
• While combining light frequencies gives new colors,
combining sound frequencies gives chords
• Some sound frequencies “go together” harmoniously,
just as some colors do
• All music scales use the octave frequency difference,
even animals respond to it
• Must stay with the same octave of notes or “key”
– Major keys are happy and joyful
– Minor keys are sad and plaintive
17 July 2015
© 2001 Brian Whitworth
9
Attention and sound
• Sudden volume or pitch changes get attention
• For sound, time order is critical, listen to the “tada”
sound played in reverse
• Application: Sound easily catches attention as the
channel is usually free (cf visual)
– This sound “says” there is an error
– Note if you use speech the person
tends not to listen and read (same channellanguage) and listening is a lot slower
than reading, by a factor of up to 4
17 July 2015
© 2001 Brian Whitworth
10
Gregory: Music
Music
• Music can be considered to consist of:
– Rhythm: loudness pattern over time, is the most basic form
of music (e.g. drums)
– Melody: frequency patterns over time
– Harmony: frequency combinations (chords and
accompaniment)
• Gestalt principles of proximity and simplicity apply
to sound patterns (see Gregory diagram p501- listeners transpose notes
that are proximate)
• Application: Music is a universal “language” that
every type of person can relate to - use it to create
mood
17 July 2015
© 2001 Brian Whitworth
11
Foreground and Background
• Sounds
– B/G: No pattern or steady rhythmic one (e.g waves), quiet,
soothing
– F/G: Sudden loud noises, varying patterns
• Music
– B/G: Repetitive, harmonic changes (e.g muzak, store music)
• Repetitive but sharp
more mellow, harmony
Ideal
– F/G: Strong rhythm, catchy tune, clear sharp notes
• Speech
– B/G: many people talking (a hubbub)
– F/G: Single speaker almost always gets attention
17 July 2015
© 2001 Brian Whitworth
12
Individual differences
• Just as some people are color blind, some cannot hear
certain notes or tunes (cf right vs left hemisphere
specialization)
• Some people prefer
–
–
–
–
rhythm (e.g heavy rock music)
melody (e.g. pop music)
harmony (e.g. orchestra), or
All combinations
• Application: Choose music style and complexity to
suit the audience
17 July 2015
© 2001 Brian Whitworth
13
Binaural hearing
Gregory: Binaural hearing
• Each ear hears slightly different signals - we
use the difference to locate sound in space
– For low frequencies use time or phase difference
(up to one millionth of a second)
– For high frequencies use loudness difference
• Application: Stereo sound adds another
dimension or channel to listening - it allows
3D localization analyzers to operate
17 July 2015
© 2001 Brian Whitworth
14
Auditory illusions
• Alternating high/low tones in
each ear heard as high in one
and low in the other
• Switch earphones - same result!
• Dominant LH processes right
ear notes only
• But location involve a separate
channel, and uses the higher
frequency note
• Same sounds usually come from
the same source
17 July 2015
© 2001 Brian Whitworth
Gregory: Auditory illusions
I heard
in
alternating ears
15
Gregory: Phonetics
Phonetics
• Phonemes, the basic sounds of speech, often vary
with the sounds around them (context effect) • Bough
• Not necessarily related to spelling, e.g.
• Rough
– Same Chinese characters are pronounced very
differently (e.g. Mandarin vs Cantonese)
– Some cultures have sounds that others don’t
• Cough
• Through
• Dough
(English “th” and “r’, German umlaut, Chinese tonal vowels)
• Cultures recognize foreigners by how they talk
• Application: Use a native speaker of your
audience for familiarity, or a foreigner for the
exotic (e.g New York vs Scottish vs Japanese English)
- was that the same person as before??
17 July 2015
© 2001 Brian Whitworth
16
Gregory: Frish, p278
Signals and communication
• Frisch discovered the “language” bees used to tell other
bees where the honey was
– Direction was the direction of their “dance” relative to the hive
axis being the angle to the sun
– Distance was the dance type (round or waggle) and speed
• Any action can be a signal used in communication if
sender and receiver process it the same way
• Application: Communication problems arise when we do
not process the same signal the same way
e.g. different cultures, men vs women, young vs old
• Language is a common way of processing certain signals
17 July 2015
© 2001 Brian Whitworth
17
Body Language
Gregory: Body language, p99
• In conversations people synchronize behaviors
– Sit the same way (postural congruence)
– Use hands the same way
• This signals rapport, i.e. that they process (think, act
and feel) in the same way
• Application:
– People will ask of your web site “Is this me?”, and judge if
the site is like them (or done by people like them)
– In web site design always identify your audience, and
initially at least, show congruence to them
17 July 2015
© 2001 Brian Whitworth
18
Formal Language
Gregory: Language: learning word meanings,
p421
• Language occurs when arbitrary signals (words),
link to or symbolize concepts
• Language must be learned (cf instinctive sounds)
– Instinctive sounds mean the same everywhere
– Words and meanings vary with language
• Mensa, table, mesa
– Speech involves both learned words and instinctive
sounds and tones
– Application: Natural sounds are universally
understood
Wake up!
17 July 2015
© 2001 Brian Whitworth
19
Words and concepts
• Words can represent concepts
– Concepts develop apart from words & vocabulary
• Children know things before they say them
– Concepts begin broad, but become discriminated into
sub-concepts, giving conceptual hierarchies
• Child uses “Ball” for a ball, a balloon, an egg, the sun
– Language is situational, e.g. “Mummy shoes” means
• “Where are my shoes?”, or
• “Help put on my shoes”, depending on the situation
• Only after ~4 do words have meanings apart from the present
situation, and only after ~11 do children analyze ideas
17 July 2015
© 2001 Brian Whitworth
20
Application - the KIS principle
• Keep It Simple. To communicate effectively
– Use words whose meaning is clear and simple
– Use concepts that are clear, especially for children
(cf children’s color preferences)
– For example children may struggle with idea of someone
who is both good and bad
• Examples
–
–
–
–
Reduce the number of words
Reduce the size of the words
One idea/concept per sentence
Use the situation (graphics, colors etc) to support meaning
17 July 2015
© 2001 Brian Whitworth
21
Gregory: Concepts, p157
Concepts
• A concept mentally represents an experience
– Concepts can be connected to other concepts
e.g. baby ~ cuddly, fish ~ cold (association)
– Concepts can contain other concepts
e.g dog < animal < thing (abstraction)
– Concepts can imply other concepts
e.g. falling ~> landing (analysis)
– Concepts are the building blocks of conceptual
thinking, a generic system to model anything
(cf spatial models, which only model space)
17 July 2015
© 2001 Brian Whitworth
22
Conceptual thinking
Gregory: Thinking, how it
can be taught, p774
• Formatory or associative thinking, which uses pointto-point connections between concepts
(e.g. conversations often just associate ideas)
• Abstract or holistic thinking, which forms abstract
concepts using conceptual similarities/differences
(e.g. no one has ever seen a “three”, it is an abstract concept, and much
mathematics involves such concepts)
• Analytical thinking which can derive and assess
conceptual relations using logic and argument
(e.g. People are stupid, Socrates is a person, therefore Socrates is stupid:
Socrates is somebody, somebody is stupid, therefore Socrates is stupid; etc
are examples of faulty argument)
17 July 2015
© 2001 Brian Whitworth
23
Gregory:Language areas in the brain p425
Language and meaning
• The analysis of language to create conceptual
meaning, and expressing meaning via language,
seem to occur in distinct areas of the LH
– Damage to one area (Broca’s) seriously affects
language, but can leave melodies unaffected
– Damage to another area (Wernicke’s) seems to affect
the connections between concepts and words
• The analysis of document content seems a distinct
channel with distinct analyzers, just like color etc
17 July 2015
© 2001 Brian Whitworth
24
Document Structures
Gregory: Internal models, p387
• Concepts exist in a conceptual structure
• A document’s content represents a conceptual
structure which the reader must process
• Document’s structure is supported by
– Headings, sub-headings, paragraphs and sentences
• Paragraph should be about one idea, begin with an
introduction, and end with a summary or conclusion
– Framing/coloring of main idea sections
17 July 2015
© 2001 Brian Whitworth
25
Application - Document structure
• Types of conceptual relations
– Associative (concepts connect to
concepts) e.g a “flow” of ideas,
hypertext mesh connections
– Abstract (concepts contain concepts)
e.g. a menu or headings, hierarchical
connections
– Analytical (concepts imply concepts)
e.g. a set of base ideas/information that
a logical argument uses to give a
conclusion, usually a text sequence
• These relations combine to give the
document’s content structure
17 July 2015
© 2001 Brian Whitworth
A
A
B
C
D
A1
A2
A2a
A2b
A2c
A
B
C
Conclusion
X
D
E
Argument
F
Assumptions
26
Content analysis
Abstract
Headings
Detail
Paragraphs
Sections
Sentences
Sequence
Beginning (issue, problem, interest, purpose)
Associations
Hypertext Browsing
Middle (information, argument)
Links to appendices, more
End (summary or conclusion)
17 July 2015
information, other sources)
© 2001 Brian Whitworth
27
Analytical thinking
• Not easy or intuitiveusually it hurts
• Avoid premature
closure from other
processes
A
Which way does the ball go?
A
B
B
C
Bob cycles to from A to B at 10mph, how fast must
he cycle back to average 20mph for the whole trip?
17 July 2015
© 2001 Brian Whitworth
Answers: B,
He cant
28
Content applications
• Previous principles apply to content processing:
– Use priming: Declare key ideas, like site purpose,
early to prepare or “lead into” them
– Manage attention: The first sentence can be critical
for keeping (or losing) attention
– Chunking: Group same ideas in the same paragraph
or headed section, in a logical sequence
– Minimalism: Remove unnecessary ideas
– Consistency: Avoid contradictions (e.g. bad spelling
contradicts claims for professionalism!)
17 July 2015
© 2001 Brian Whitworth
29
Remembering concepts
Gregory: Remembering, p679
• How to memorize information?
– Simplest method is repetition
• Conceptual memory is based on embedding concepts
in a conceptual structure - so one part can lead to
another part
– E.g. remember letters in the alphabet sequence, remember
dentist at 1pm as after lunch
– Remember by recalling some aspect that links to the rest
• Sensory memory is based on impact based on change
– E.g. Say it LOUDLY or UnUsUally to remember it
17 July 2015
© 2001 Brian Whitworth
30
Things remembered
• Things will be remembered if
–
–
–
–
Repeated (rote learning)
They have sensory impact (clear, bold, obvious)
They have emotional impact (see later lesson)
They have conceptual impact
• Link into existing conceptual structures
• Provide strong argument based on agreed facts
– Can use sensory modality to retrieve conceptual modality
(e.g. Mnemonic like KIS, or KISS)
• If you understand how people think, what you say
can fit into their structures, cf being a “red flag”
17 July 2015
© 2001 Brian Whitworth
31
Gregory: Touch p778
Touch
• Involves skin receptors for:
–
–
–
–
–
Light touch (hairs)
Pressure
Vibration
Pain
Temperature
• We detect changes, e.g. place one hand in cold
water and one in hot. Now tepid water seems
“hot” to one hand but “cold” to the other
17 July 2015
© 2001 Brian Whitworth
32
Cognitive maps
Gregory: Localization of Brain
Function and cortical maps, p436
• See the strange diagram on p436 of areas of the
brain devoted to touch (somatosensory) processing
• Large parts of the skin receive little processing,
while smaller parts (e.g. the lips) involve
extensive processing
• Illustrates that cognitive impact depends on the
amount of processing, not the amount of signal
17 July 2015
© 2001 Brian Whitworth
33
Gregory: Taste p767
Taste
• Tongue recognizes types of dissolved molecules
– Primary tastes are salty, sweet, sour and bitter?
– But some tastes are none of these (Japanese umami)
– Different types of primary taste, e.g. soya salty vs brine salty
• With smell, taste guides food selection and good diet
e.g. avoiding rotten food
• Depends on a varying sensory baseline of saliva (x10)
and current mouth content (x100)
– Fasting sensitizes taste, eating desensitizes it
– Each taste affects the following one (e.g. wine)
– With a purified baseline, tasters can distinguish once from
twice distilled water
17 July 2015
© 2001 Brian Whitworth
34
Gregory: Smell p719
Smell
• Nose can recognizing types of airborne molecule
• Smell sense connects to older parts of the forebrain,
rather than the neocortex that allows conceptual analysis
• Smells enervate or stimulate the body directly
• Pervasive impact often overlooked
– Relationships - Deodorants, perfumes, air fresheners
– Eating - Flavors of foods come largely from smell
– Danger - Detection of leaking gas
• Considerable individual differences, women may be
more sensitive than men
• No satisfactory classification scheme
17 July 2015
© 2001 Brian Whitworth
35
Applications
• Few IS applications at this time for other senses ..
–
–
–
–
–
–
Touch screens where you feel the difference?
Scratch’n’sniff throwaway screen covers?
A pain port for more realistic action gaming?
Motion chairs for a fuller roller coaster experience
Galvanic skin response “truth channels” for e-mails?
Brain wave (EEG) connections to bypass the “middle man” of
the senses?
• “Primitive” senses are simple but pervasive - can have
powerful effects if activated e.g. a bad smell
17 July 2015
© 2001 Brian Whitworth
36
Gregory: Cross-modal sensory integration p173
Sensory Integration
• Is there a common (sense) “pool” for sensory
experience?
– Can you recognize a seen object by touch?
• Yes - cross-modal sensory integration occurs
• No - each sense has its own analyzers that direct actions
• In young children and animals, cross-modal
sensory integration is weak (no real need)
• But language/concepts act as cross-modal
bridges, connecting sensory analyzers
17 July 2015
© 2001 Brian Whitworth
37
Gregory: Blindness, recovery from p94
Example - recovery from blindness
• Blind subject received corneal grafts at 52
– Easily learned to see CAPITALS which shapes he
had learned by touch
– Could not recognize lower case letters (which had
not been learned by touch)
– Unaffected by Necker cube illusion, but looking out
a window several stories up thought the ground
outside was touchable (no depth processing)
– With a familiar object (a lathe), had to touch it first,
then said “Now that I have felt it I can see it”
17 July 2015
© 2001 Brian Whitworth
38
Application - preferred modality
• Each person has different preferred primary
channels and analyzers which may act as
“gatekeepers” for general acceptance/processing
• For those who prefer/attend document meaning
–
–
–
–
–
Give valid/useful information
Avoid factual or logical inaccuracies (red flags!)
Provide a content overview or summary early
Use a logical sequence of ideas and sections
Define and use terms unambiguously, correctly and
literally rather than metaphorically or colorfully
17 July 2015
© 2001 Brian Whitworth
39
Summary
• Sound analyzers separate the signal frequencies
and look for patterns over time rather than space
• Conceptual processing involves distinct aspects,
represented in the content of a document
• Language is a cross-modal bridge for the senses,
as concepts become apart from the senses
• Analytical thinking is powerful but non-intuitive,
and tends to be overridden by simpler processes
if decision making is not delayed
17 July 2015
© 2001 Brian Whitworth
40
Homework
• The following are key readings for this lesson:
–
–
–
–
–
–
–
–
Hearing, p303 - Basic mechanics of the ear
Binaural hearing - basis of stereo
Language, learning word meanings - distinguish concepts from words
Thinking, how taught - Distinguish analytical thinking (difficult) from
formatory thinking (easy) and read the examples
Language areas in the brain - Skim, but recognize there are distinct
areas that have particular functions in language and thought
Remembering - Learn how to remember things and to make things
memorable
Touch, taste and smell - briefly review so you can see the same
principles operate. Understand how these senses are more pervasive
and less specific compared to sight and sound
Cross-modal sensory integration - Read the examples, and understand
the role of language and thought in bridging the senses
17 July 2015
© 2001 Brian Whitworth
41
Next:
• Even given every type
of visual processing to
every level there is still
something missing!
• What is it?
• User interaction and
navigation
– “Where am I?”
– “What do I do?”
17 July 2015
© 2001 Brian Whitworth
42