Transcript Eriksholm Workshop V *Cognitive Energy - (CSD)

Fifth Eriksholm Workshop
“Hearing Impairment & Cognitive Energy”
FRAMEWORK FOR
UNDERSTANDING
EFFORTFUL
LISTENING
Kathy Pichora-Fuller (U of Toronto)
Sophia Kramer (VUMC Amsterdam) – Workshop Co-convenor
Participants & First Authors of Papers

















Mark Eckert
Brent Edwards
Ben Hornsby
Larry Humes
Sophia Kramer
Ulrike Lemke
Thomas Lunner
Carol Mackersie
Mohan Matthen
Graham Naylor
Natalie Phillips
Kathy Pichora-Fuller
Michael Richter
Mary Rudner
Mitchell Sommers
Kelly Tremblay
Arthur Wingfield
International and Interdisciplinary
Canada, Denmark, Netherlands, Sweden, Switzerland, USA (UK)
Philosophy
Cognitive Psychology
Gerontology
Neuropsychology
Speech Perception
Health Psychology
Neuroscience
Social Psychology
Engineering
Motivation Psychology
Audiology
Eriksholm Workshop V
“Hearing Impairment and Cognitive Energy”
June 3-5, 2015
“Cognitive Energy” ~ Titchener (1908) “Psychic Energy”
Consensus to address the following questions:

What is known and what gaps exist in our knowledge?

Can we reconcile the cognitive/behavioural and the stress/physiological views?

Do measures of “listening effort” in the lab reflect real-world life experiences?

What is potential/feasibility for translation to clinical/engineering applications?
The workshop and consensus paper focused on three main areas:
1) theories, models, concepts, definitions, & frameworks
2) methods and measures
3) knowledge translation
McGarrigle, R., Munro, K. J., Dawes, P., et al. (2014). Listening effort and fatigue: What exactly are we measuring?
A British Society of Audiology Cognition in Hearing Special Interest Group ‘white paper’. Int J Audiol, 53(7), 433-440.
Framework (Borg)
According to Borg, Bergkvist, Olsson et al. (2008):
"[A] model is defined as a set of related concepts
that can quantitatively predict an outcome on the
basis of certain premises.
[A] framework is a series of defined concepts that
are less precisely related and that are not formulated
in a way that allows quantitative testing."
Paradigm Shift (Kuhn)
Scientific turmoil when discrepancies between data
and existing theories cannot be resolved.
Sociology of science and readiness for new ways of
thinking about old problems.
New conceptualization shifts paradigm.
Ear and Hearing

July 2016 Special Issue (FREE online)

Consensus plus 16 papers

Setting the stage


Behavioural approaches and cognition


(6 papers: Humes, Sommer, Rudner, Lemke, Edwards, Pichora-Fuller)
Physiological approaches: motivation, stress, and fatigue


(3 papers: Matthen, Wingfield, Phillips)
(6 papers: Eckert, Richter, Mackersie, Kramer, Hornsby/Naylor)
Knowledge translation

(2 papers: Lunner, Tremblay)
Key definitions
etc.
etc.
Learning Objectives
You will be able to
 Explain how a classic cognitive model of attention
can be applied to listening effort;
 Describe cognitive behavioral measures and
physiological measures to assess listening effort;
 Apply theories of motivation and fatigue to
rehabilitation approaches to address listening
effort.
Applying a Classical Model of
Attention to Understand Listening Effort
Audiology & Psychology
Language, Perception, Memory, Attention, Motivation
“The psychological
processes (at the top)
are not assigned to
any particular level,
but in general they
require the
participation of the
cerebral cortex.”
H. Davis, 1964.
1970 edition of the
audiology textbook
Hearing and Deafness
Why put effort into effort?

Patient experience…..
Better rehabilitation for patients…..
Better hearing accessibility in society….

Word recognition is insufficient…


From the words of a person who is hard
of hearing to FUEL….
“When you are hard of hearing you struggle to hear;
When you struggle to hear you get tired;
When you get tired you get frustrated;
When you get frustrated you get bored;
When you get bored you quit.
-- I didn’t quit today.”
Difficulty hearing can
increase demand on cognitive processing resources and
increase emotional and physiological stress
that individuals may avoid by withdrawal from social interaction….
unless the individual has motivation not to quit!
We hear with our ears,
we listen with our brain….
and when and how much effort
we expend during listening in
everyday life depends on our
motivation to achieve goals and
attain rewards of personal or
social value.
Effort could yield pleasure (Matthen)
Definition of (Listening) Effort
We defined effort as
the deliberate allocation of mental resources to overcome
obstacles in goal pursuit when carrying out a task,
with
listening effort applying more specifically when tasks involve
listening.
....Not only speech understanding (scene analysis, alarms,
music, emotion...)
Kahneman, 1973 Attention and Effort
FUEL
Capacity Supplied x Capacity Demanded
EFFORT
(Kahneman, 1973)
Behavioral Cognitive Measures
Behavioral Cognitive Measures
 Working memory
 Dual task
Cognition & HA Benefit Correlated

Landmark studies > 10 years ago (Gatehouse; Humes; Lunner)

Those with higher cognitive function


Those with lower cognitive function




do better with complex, fast-acting signal processing
do less well to such complex devices
Cognition matters in challenging conditions
Why?
How/why measure cognitive status?


To predict or guide treatment
(HA fitting, training)
As a new outcome measure
Audiograms and Age (ISO 7029)
Women
Men
3kHz


HF audiometric threshold elevation
 OHC (also noise-induced hearing loss)
 Endocochlear potentials ~ stria vascularis
Neural – loss of synchrony
(Mills, Schmeidt, Schulte, & Dubno, 2006)
3kHz
Speech Understanding in Noise
(CHABA, JASA, 1988)

Little problem in ideal listening conditions





Difficulty in challenging listening conditions








Quiet
One talker
Familiar person, topic, situation
Simple task, focused activity
Noise
Multiple talkers
Strangers, accents, new topic, novel situation
Complex task, many concurrent activities
Fast pace
Hearing aid
Health care encounters?
Avoid by withdrawal from social interaction!
Speech, Spatial & Qualities of Hearing Scale (SSQ)
(SSQ; Gatehouse & Noble, 2004;
Bahn, Singh, Pichora-Fuller, JAAA, 2012)
Older “normal” vs HL
Older “normal” hearing for age
clinically normal audiograms up to 4 kHz
(N = 48; average age ~ 70 years)
Older vs Younger Adults
SSQ and Behavioral Hearing Tests:
Not Significantly Correlated
Pure-tone Average
(dB HL)
Words-in-Noise (WIN)
50% threshold (dB SNR)
SSQ Items with Largest Age-related Differences

Speech


Young - Old score
(10-point scale)
Conversing in adverse environment
 Conversation in echoic environment
 Talking with a person in continuous noise
1.7
1.6
Focusing, switching attention
 Ignore interfering voice of different pitch
 Following conversation switching in a group
 Ignore interfering voice of same pitch
1.9
1.6
1.5
Hearing Aid Compression & Cognition
(Lunner & Sundewall-Thorén, JAAA 2007)
Explained SNR variance
from hearing loss and cognitive performance
45%
Explained variance
40%
Slow, Steady
35%
30%
25%
PTA(6)
20%
VLM
15%
10%
5%
0%
Slow&Unmod.
Slow&Mod.
Fast&Unmod.
Test condition
Fast&Mod.
Controlling Audibility
(Humes, JASA 2002;JAAA 2007)


Audibility is main factor for
UNAIDED measures of
speech (in noise)
Aided (or amplified to 4 kHz):
other factors account for over
half of variance: age, central
processing (SNR), and
cognitive factors (memory)
Bottom-up & Top-down Processing
Effortful listening


Bottom-up (ear to brain)

Analysis of acoustic signal




Better signal (faster)
Poorer signal (slower)
@ amount & type of distortion
Knowledge
Top Down

Bottom-up processing less efficient
Top-down processing more necessary
Meaning
Top-down (brain to ear)

Priming (pre-signal)


Disambiguation (post-signal)


expectations facilitate recognition (faster)
knowledge constrains alternatives (slower)
Repair (post-signal)

Fill in gaps or correct errors (slower)
Bottom up

Sound
Coordinate Bottom-up & Top-down Processing
Working memory

System responsible for the PROCESSING and
temporary STORAGE of information
during the performance of all complex cognitive tasks,
including comprehension
 assumed to have a limited capacity that must be
shared between processing and storage

(Baddeley, 1976)
Speech
Perception
in
Noise
(Pichora-Fuller, Schneider, Daneman, JASA, 1995)

3 dB
8 lists: 50 sentences in babble

Half low-context
John did not talk about the spoon.

YOUNG
Half high-context
Stir your coffee with a spoon.



3 dB
Repeat last word of sentence
Vary S:N (signal-to-noise ratio)
(Sometimes also recall test)
OLD
OLD HL

Old need 3 dB better S:N

Context helps old 3 dB more
Airplane
Living Room
Loss of Neural Synchrony
Neuron
Firing
Time

Aging results in more error in phase locking


(poorer synchrony or “jitter”)
Consider as if mild form of auditory “neuropathy”
Jittering a Tone or Speech
(Pichora-Fuller et al., 2007, Hearing Research)
Spectrograms for Jittered and Intact
Sentence in Babble
Effect of Simulated Auditory Aging on
Working Memory Span
(Pichora-Fuller, IJA, 2008; Brown & Pichora-Fuller, Canadian Acoustics, 2000)
Inter- & Intra-individual Differences
Pichora-Fuller, Phonak, 2007

INTER: Individuals differ in WM capacities
Fred
Mary

INTRA: Allocation of capacity resources to processing vs
storage varies with task demands
Bottom-up processing
Top-down processing
Fred in quiet
WMS = 6
Fred in noise
WMS = 4
Fred in more noise
WMS = 2
WM
HL
Age
Less
More
Word Auditory Recognition & Recall Measure
(Smith, Pichora-Fuller, Wilson, & Alexander, Ear & Hearing, 2016)
Word
WEST
FLAG
Recognition
√
RISK
GRACE
Judgment
X, Flat
√
√
Recall
West, √
Flat,
√
√
√
√
√
√
Risk, √
Grace, √
√
BAR
X, Car
SHOVE
√
X
X
Card, X
Shove, √
X
WHAT
MOON
√
√
√
√
What, √
Moon, √
CALF
THAT
√
√
√
√
Rice, √
That, √
√
√
WARRM Study - Participants
N = 48 per group
YN
ONH
OHL
Word Repetition vs. Recall

Repetition



Effect of group
No effect of setsize or task
Recall



Effect of setsize x group
Effect of setsize x task
(serial position)
(
Effects of Modality & Linguistic Level on Recall
(Pattison, Pichora-Fuller & Smith, AAS, 2016)
Young Adults with
Normal Audiograms
Assessment Issue



Should audiologists test cognition?
Should others (e.g., neuropsychologists) test hearing?
How could information be shared across
professions?
Hearing Loss Can Impair Performance on
Any Task Using Auditory Stimuli

Weinstein & Amsel (1986)
 N=30
institutionalized elders with senile dementia
 10 of 30 reclassified to less severe category of
dementia when retested with amplification
 (83% had hearing loss > 25 dB HL, significantly higher
than comparison sample w/o dementia)
MoCA:
Montreal Cognitive
Assessment
www.mocatest.org
Visuo-spatial/executive
Naming
Memory
Attention
Language
Abstraction
Repetition
Orientation
MoCA Repeat & Recall (Dupuis et al., 2015)
Good Hearing
80
All Participants
25
80
70
70
20
60
15
50
% Recalled
60
40
10
30
5
20
0
50
40
30
20
10
0
10
Face
Velvet
Church
Daisy
0
Red
Face
Good Hearing – Hearing Loss
No difference on final word, but the
HL group remembers fewer of the
earlier words in the list, even if they
correctly repeated the words twice on
the learning trials.
Velvet
Church
Daisy
Red
Hearing Loss
80
70
% Recalled
Group Difference
30
60
50
40
30
20
10
0
Face
Velvet
Church
Daisy
Red
When Does Cognitive Aging Start?
1.5
Synonym Vocabulary
Pattern Comparison (Speed)
Raven's (Reasoning)
Recall (Memory)
84
Z-Score
0.5
0.0
50
-0.5
-1.0
Percentile of Population
1.0
16
-1.5
10
20
30
40
50
60
70
80
90
Chronological Age
Salthouse (2004) Current Directions in Psychological Science
Cognitive Aging

Gains:
Knowledge is preserved
and context is helpful
Losses:
Processing
 Working memory
 Slowing
 Attention/Inhibition
Benefit from Context (dB SNR)
7.0
6.0
2-3 dB
5.0
4.0
3.0
2.0
1.0
0.0
Younger
Intact
Younger
Jitter
Older Intact
HIGH
LOW
Context, Intelligibility & Brain Activation
(Obleser, Wise, Dresner & Scott, 2006)
High vs. low predictability at
intermediate signal quality for
younger adults listening to
distorted (noise-vocoded) SPIN
sentences
Activation to HIGH-CONTEXT >
LOW-CONTEXT speech
Various areas activated including the
left dorsolateral prefrontal cortex
(working memory & semantic
processing)
Cognitive Neuroscience of Aging





Same performance achieved with different processing
More widespread activation ~ brain reorganization
 Young brain activity more lateralized
 Old brain activity more distributed
Deterioration or compensation?
HAROLD: Hemispheric asymmetry reduction in older adults
(Cabeza, 2002)
PASA: Posterior-anterior shift in aging
(Davis, Dennis, Daselaar, Fleck & Cabeza, 2008)
Compensation
(Grady, 2012, Nature Reviews Neuroscience, 13, 491-505)
low
high
Physiological Measures
Autonomic (Para)sympathetic Nervous System:
• Pupilometry
• Skin Response
• Cardiac Response
Central Nervous System:
• Brain imaging (fMRI)
• EEG (P3 and late positive potentials; alpha)
Autonomic (Para)sympathetic Nervous
System Measures

Pupil dilation (Kramer)



peak pupil dilation indexes momentary load
resting pupil diameter before/after stimulus presentation indexes
state of engagement
Pupilometry research on cognitive processing demands during
listening sensitive to:







speech intelligibility (Zekveld, Kramer, Festen et al. 2010),
type of background noise (Koelewijn, Zekveld, Festen et al. 2012),
syntactic complexity (Piquado, Isaacowitz, Wingfield 2010),
auditory stimulus characteristics (Kramer, Lorens, Coninx et al. 2013),
degraded spectral resolution (Winn, Edwards & Litovsky et al. 2015),
cognitive abilities (Zekveld, Kramer & Festen 2011) and
divided (vs. focused) attention (Koelewijn, Shinn-Cunningham, Zekveld
et al. 2014).
Autonomic (Para)sympathetic Nervous
System Measures

Skin conductance (Mackersie)
electrical activity on the skin surface mediated by the sympathetic
nervous system
 used to infer automatic attention (orienting), effort, motivation, and
emotional reactivity
 increase in skin conductance with increasing listening task demands
for some speech repetition tasks suggests potential role in the
evaluation of listening effort


Cardiac responses

Heart Rate Variability (Mackersie)
over time (e.g., standard deviations of inter-beat intervals) or
 frequency domains (spectral analysis of inter-beat interval variation)
 reduction in HRV with increased listening task demand observed for
several HRV measures -- may be useful as an index of listening effort

Motivation & Fatigue
Motivation – Cardiac Sympathetic Activity

PEP: Cardiac pre-ejection period (Richter)
 time
interval between the beginning of the excitation of the
left heart ventricle and the opening of the aortic valve
 a direct indicator of myocardial contraction force—the
stronger the heart contracts, the shorter is the PEP
 changes in PEP reflect changes in myocardial sympathetic
activity
 index used in motivation research
Motivation - Neuroeconomics

Brain Imaging (Eckert)





role of cingulo-opercular and fronto-parietal brain areas in adaptive
control during speech and language processing
increased frontal brain activity when listening tasks are challenging
activity reflects a decision-making process about the expected value
of working to optimize performance given the potential value
realized from the task
activity reflects how important success on a task is to a person (i.e.,
how the person evaluates success importance for a task)
in monkeys, the firing rate of noradrenergic coeruleus neurons in the
brain increased and was correlated with both pupil dilation and effort
related to the energization of behavior (Varazzani et al. 2015)
Fatigue (Stress)


fatigue may hinge on motivation and the control and
management of goals insofar as expending effort can
be fatiguing if goals are externally imposed, but not
when activities are self-initiated and meaningful
(Hockey 2013; Hornsby et al.)
understanding the role of motivation and arousal in the
choices made by listeners about how and when they
engage (or not) in effortful listening takes us beyond the
simple assumption that effort will go up as difficulty or
demand for cognitive capacity goes up
Relevance to AR?
3D: Effort ~ Demands and Motivation
T0 -T1: demand constant, increasing motivation;
e.g., noise level is constant but the topic of
conversation becomes a highly interesting story;
T1-T2-T3: motivation constant, increasing
demand, resulting in increased effort; e.g., story
continues to be highly interesting but noise
increases as more people arrive at the party;
Over the course of an activity,
Demand ~ level of background noise
Motivation ~ person’s evaluation of the
importance of success in performing the activity.
T3-T4: demand constant, motivation drops,
resulting in decreased effort; e.g., noise remains
steady but the highly interesting story finishes
and the conversation becomes less interesting.
What would you do differently in AR?

Evaluation of demands on capacity – cost
 HOW
to reduce demands
 HOW to increase capacity

Evaluation of success importance – benefit
 WHEN
to quit
 WHEN to persist

Stress ~ balance of demands given capacity