Transcript questions and suggestions - AAAC emotion

Multimodal sensory
integration:
questions and suggestions
GERG • Santorini, 18-21 June, 2004
Outline
• Models of multimodal integration (MI)
• Functions and factors of MI
• A cognitive neuroscience approach: The superior colliculus
• Recognition of emotion: How many levels of analysis are
needed in one modality? The case of visual integration of low
spatial frequencies of emotional facial expression.
• What about the time in multimodal integration models?
• Conclusions
GERG • Santorini, 18-21 June, 2004
1
Some models of integration
(1) Direct Identification Model (DI)
The input signals are directly
transmitted to the bimodal
classifier (Klatt, 1979).
(2) Separated Identification
Model (SI)
The visual and the auditory input
are separately identified through
two parallel identification
processes (McGurk & McDonald,
1976).
GERG • Santorini, 18-21 June, 2004
2
Some models of integration (cont’d)
(3) Dominant Modality Recoding Model (RD)
A dominant modality drives the
perception of other modalities, this
dominant modality would be different
according to the context or the task
(identification or localization).
(4) Motor Space Recoding Model (MR)
Basics of the model: The two
modalities are projected upon a
common motor space where they
are integrated before final
categorization.
(4bis) Model of Motor Dynamics
GERG • Santorini, 18-21 June, 2004
3
Factors of integration
Two major factors are relevant for multimodal
integration whether it is spatially (spatial
occurrence in event localization, e.g. ventriloquy)
or through recognition (McGurk effect).
Two functions:
Two major processes:
RecognitionIdentification
(what system)
Temporal aspects
Localization
(where system)
GERG • Santorini, 18-21 June, 2004
Spatial aspects
4
Boundaries of multimodal
integration
Localization:
Recognition:
Thurlow et al. 1973
- 10° maximum of divergence
- 200 ms of asynchrony
Illusion of collision:
- 40° of divergence
- ~ 400 ms of asynchrony
-Lewald et al. 2001
- 3°
- 100 ms
- similar to limits of the
McGurk effect (Watanabe,
2001)
GERG • Santorini, 18-21 June, 2004
5
The superior colliculus: An
example of spatial integration
Receptive field: 55% of neurons are a receptive field related to
multimodal integration (visual-auditory-somesthesic) in this
nucleus: the responses of these neurons are higher when a
multimodal stimulation occurs than when there is a unimodal
stimulation.
GERG • Santorini, 18-21 June, 2004
6
Adapted from Emanuelle Reynaud
Propreties of multimodal neurons in the superior
colliculus (Stein et al., 1993;1998)
The spatial rule:
The responses of neurons are higher when multimodal spatial stimuli occur
compared to unimodal stimulus or the sum of unimodal stimuli.
When the spatial occurrence of stimuli are disparate these neurons do not
discharge or show a decrease of spontaneous activity.
The temporal rule:
Apparently time is less important in the generation of responses of the multimodal
neurons than spatial occurrences. The amplitude of the increase of response
decreases with the increase of asynchrony. The maximum of responses is related
to the overlap of pattern activity through the time (binding problem).
The rule of inverse efficacity:
When two stimuli are spatially near and temporally synchronized the response of
multimodal neurons is superior to the maximum of the unimodal response.
The less the response is high for the unimodal stimuli, the higher the multimodal
response is (% of gain).
These rules are likely to be also useful in the identification of multimodal
integrative areas at the cortical level (e.g., the Superior Temporal Sulcus (STS)
or the parietal lobe).
GERG • Santorini, 18-21 June, 2004
7
Functions of multimodal integration
related to emotion and attention
The process of multimodal integration could maximize the detection of
events and their identification. These two processes are relevant to
modulate attentional processes and could thus orient the ressources of
organism (or ECAs …) on specific events or objects.
People better recognize the stimuli when two or more sensory channels
are excited (visual+auditory) and their reaction time decreases with the
increase of channels excited (when the stimuli are congruent).
Multimodal level
(e.g. STS, parietal lobe)
Sensory information
(e.g. auditory unimodal cortex)
GERG • Santorini, 18-21 June, 2004
Sensory information
(e.g. visual unimodal cortex)
8
Recognition of emotion
In a virtual environment it is necessary to detect the emotional signals
at more than one level for a unimodal stimulation, for example: spatial
frequency in the visual domain: analyses of low frequencies and high
frequencies with two different and partially independent processes (cf
Vuilleumier et al., 2003)?
Some brain areas (e.g.
amygdala) respond
differentially to emotional
faces at low frequencies than
high frequencies.
GERG • Santorini, 18-21 June, 2004
9
Recognition of emotion
Concept of weighting of channel in relationship to emotional
state:
• For example disgust is more recognized in the visual channel
than the auditory channel;
• Fear could be more recognized in the auditory channel than the
visual channel (often confused with surprise in the visual
channel)
• How can we weight the channels before recognizing the
emotional states? Different levels of analyses (cf example of low
and high frequencies in the visual channel).
GERG • Santorini, 18-21 June, 2004
10
Recognition of emotion
Integration: which model could help to detect more precisely and more
rapidly the emotional state of people? (two simplified examples)
Sensory
information
(e.g. auditory modality)
Time 2
Feature
extraction
Local
Recognition
Decision - identification
Sensory
information
Feature
extraction
Sensory
information
Feature
extraction
(e.g. visual modality)
(e.g. auditive modality)
Local
Recognition
Fusion:
Pattern recognition
Sensory
information
(e.g. visual modality
Dynamic aspects:
temporal channel
and analysis (?)
weighting from time
1 to time 2
Decision - identification
Feature
extraction
GERG • Santorini, 18-21 June, 2004
Time 2
11
Conclusions - questions
• To what extent can the functional analysis of the brain help us
to enrich the models of multimodal integration as much in
recognition as in production? Collaborations?
• At which moment should integration be realized? On several
occasions? To what extent should successive integrations mutually
influence each other?
• Are categorization and emotional labelling relevant to determine
the recognition and the production algorithms? Should there not be
an attempt to categorize (detect) the cognitive processes underlying
the emotional processes (unfolding with time) in the different
modalities (cf emotional facial expressions described in the
appraisal processes rather than the discrete or dimensional models).
• What is the status of time in the multimodal integration process?
GERG • Santorini, 18-21 June, 2004
12