Transcript Stein and Meredith 1993 - North Dakota State University

Audiovisual Multisensory Facilitation: A Fresh
Look at Neural Coactivation and Inverse
Effectiveness.
Lynnette Leone
North Dakota State University
“However, the primary somatic, visual and auditory
cortices are not interconnected, and each projects
to very restricted and entirely separate fields
chiefly in their immediate vicinity…” (Jones and
Powell, 1970)
“Such integration is as critical for making sense of
the inputs the brain receives from different
modalities as it is for interpreting multiple inputs
from any single modality…for the brain to integrate
them [these inputs] the different senses must
ultimately have access to the same neurons.”
(Stein and Meredith 1993)
Outline
1. Multisensory Integration
2. Redundant Signals Effect
3. Inverse Effectiveness
4. The Current Study
5. Future Directions
Multisensory integration
Definition: (MI) refers to the process by which
convergence of information from two or more
individual sensory systems onto particular neurons
results in a neuronal response that is qualitatively
and quantitatively different than the responses of
these neurons to individual signals (Calvert, 2001).
Multisensory integration
McGurk Effect
Visual Rabbit
Multisensory integration
Facilitation – more likely to occur when
two things happen at the same time
and / or in the same place.
Suppression – more likely to occur if
two event occur at widely different
times and / or places.
Redundant Signals Effect (RSE)
Definition: the modulation of reaction
time to pairings of sensory stimuli
presented simultaneously over one or
more sensory channels.
- facilitative MI
- not exclusively multisensory
Redundant Signals Effect (RSE)
Separate Activation vs. Neural Coactivation
- RACE models
– Miller’s Inequality
P (RT < t|A and V) ≤ [ P (RT < t|A) + P (RT <
t|V)] - [P (RT < t|A) ‫ ٭‬P (RT < t|V)]
Redundant Signals Effect (RSE)
Some studies:
Miller, J. Cognitive Psychology (1982)
Experiment 1
• Subjects: 74, undergrads rt handed
•
•
•
•
•
Visual Stimuli: asterisk (‫)٭‬
Auditory Stim: 780 Hz tone (150ms)
Task: Simple RT
3 conditions: A, V, AV
Randomly Interleaved
Redundant Signals Effect (RSE)
• Results
- mean RT
V = 412 ms, A = 409 ms, AV = 326 ms
- violations of inequality
occurred across a range of reaction
times (250 – 350 ms)
Miller, J. Cognitive Psychology, (1982)
Redundant Signals Effect (RSE)
Molholm et al Cognitive Brain Research (2002)
•
•
•
•
•
•
•
Subjects: 12 (5 female, 11 RH), 23.8±2.7 yr
Visual Stimuli:
60 ms, 1.2 deg disc
Auditory Stim: 1000 Hz, 60 ms, 75 dB SPL
Task: Simple RT, right index finger button press
3 conditions: A, V, AV
Randomly Interleaved
ISI: 750-3500 ms
Redundant Signals Effect (RSE)
Molholm, S. et al Cognitive Brain Research (2002)
Redundant Signals Effect (RSE)
Spatial and temporal constraints
- Miller, (1986)
- Stein et al, (1996)
- Frassinetti et al, (2002)
Inverse Effectiveness
Rule: combinations of weaker stimuli lead to
greater facilitative effects.
Studies:
- Stein and Meredith, 1993
- Diederich and Colonius, 2004
- Holmes, 2007
Current Study
A. How might changes in the processing
time of one system influence the timecourse of the RSE?
- increasing stimulus contrast leads to
decreases in reaction time.
(Harworth and Levi, 1978, Murray and Plainis, 2003 and
Vassilev, Mihaylova and Bonnet, 2002)
Current Study
B. Hypothesis –
If neural coactivation is indeed
responsible for the RSE, then changes in
signal processing time in one modality
will necessarily change the facilitative
effects obtained when those signals are
paired with signals from another
modality.
Method
Pretest Procedure:
- Single-interval forced choice signal detection
paradigm
Response
Signal
Yes
No
Yes
No
Hits
False alarms
Misses
Correct
rejections
Method
Pretest Procedure:
- 15 Blocks; each block contained 24 unisensory
visual (2 x 12 contrast intensities, spatial
frequency 1 c/d) and 24 unisensory auditory (2 x
12 levels of dB attenuation), and 2 catch trials.
- d’ calculated for responses to individual stimuli
(d’ = Z(h) – Z (FA); nonlinear (sigmoid) leastsquares regression interpolated stimulus
intensities yielding criterion performance (d’ = 2).
Visual stimulus (Gabor patch = 1 c/d; duration = 100ms;
view distance = 114 cm, 2.25◦ eccentricity from fixation)
Auditory Stimulus
(1000 Hz pure tone
duration = 100ms)
Subjects: 7 (4 Male,
7 RH); 23 – 55 yrs
Method
Experimental Procedure:
- Single-interval signal detection paradigm;
- Each block: randomly interleaved trials of
unisensory visual stimuli, unisensory auditory stimuli,
catch trials, and audiovisual multisensory
combinations at stimulus onset asynchronies
ranging from -100 to +200 ms.
- Task: Subjects instructed to respond as quickly and
accurately as possible when they perceived a
stimulus. Calculated d’ for unisensory stimuli every 2
blocks and adjusted intensity settings to ensure
criterion response accuracy (d’ = 2).
Data Analysis
Experiment 1: Results
Experiment 2:
Paradigm replicated experiment 1 except that visual
stimulus contrast was increased (3x).
Experiment 3:
Paradigm replicated experiment 1 except that auditory
stimulus volume increased (3x).
Experiment 4:
Paradigm replicated experiment 1 except that both visual
stimulus contrast and auditory stimulus volume were
increased (3x).
Conclusions
What the heck?
Attentional effects:
- endogenous attention
- exogenous attention
What’s next
Experiment 5
- examines attentional contributions using a
variation of a Posner cuing paradigm to separate
the effects of attention from those of neuronal
interaction.
Experiment 6
- examines inverse effectiveness
gradient.
Future Directions
• Dark adaptation
- rods dominate our vision in the
dark
- rods operate on the order of
100ms slower than cones
• ERP
Acknowledgements:
Dr. Mark McCourt
Dr. Wolfgang Teder - SäleJärvi
Tech Support:
Dan Gu, Brian Pasieka
All of the subjects (most of whom are in this
room)