Transcript Maurizio Gentilucci - Embodied language II

Gesture and speech are controlled by the same system
Maurizio Gentilucci
Dipartimento di Neuroscienze, Università di Parma, Italy
RTM (Rete Multidisciplinare Tecnologica), IIT (Istituto Italiano
di Tecnologia) and Università di Parma
GESTURE: GESTICULATION (DEICTIC, MOTORIC,
REPRESENTATIONAL), ICONIC, SYMBOLIC, SIGN.
(Kendon 2004, McNeill 2000)
Gesture and speech belong to
two distinct communication
systems (Krauss e Hadar 1999).
Gesture functions as an auxiliary
support when verbal expression
is temporally disrupted or word
retrieval is difficult.
Gesture and speech belong to the
same communication system
(Goldin-Meadow 1999, McNeil
1992).
They are linked to the same
thought processes even if the
expression modality differs.
The link between gesture and speech (and in general language) supporting the view
that gesture and speech are controlled by a same system can be the result of the
activity of a system evolved from a class of neurons recorded in monkey premotor
area F5 (Rizzolatti et al. 1988).
Monkey grasps with the mouth
Monkey grasps with the right hand
Monkey grasps with the left hand
This system (the dual hand-mouth command system) can
be involved in transferring aspects of the control (and
meaning) of manual actions to the control of mouth
postures and vice versa aspects of the control (and
meaning) of mouth actions to the control of hand
postures.
The role of understanding the meaning of hand and
mouth gestures can be carried out by a system evolved
from the class of the mirror neurons (Gallese et al., 1996;
Ferrari et al., 2003)
Grasp of a small and large object and
simultaneous opening of the mouth.
Grasp of the small and large object with the mouth and
simultaneous opening of the fingers
From an evolutionary point of view, this circuit of
commands to both hand and mouth might have been
used by humans to transfer an initial arm gesture
communication system from hand to mouth
(Armstrong, Stokoe and Wilkox 1995, Gentilucci and
Corballis 2006). It can also be used to couple gestures
to speech (Gentilucci et al. 2008).
A system transferring aspects of the grasp control to
speech seems to be active in modern human behavior
Grasp of a small and large object and
simultaneous pronunciation of syllables.
THE CONTROL OF THE GRASP WITH THE HAND
AND THE MOUTH AFFECTS THE CONTROL OF
POSTURES OF THE MOUTH AND THE HAND.
THE CONTROL OF THE GRASP WITH THE HAND
AFFECTS THE PRONUNCIATION OF PHONEMIC
UNITS
Effects of hand postures, mouth postures, foot postures, and
vocalizations on the kinematics of reach to grasp of differently sized
objects executed with hand and mouth
Can assumed postures of one effector (mouth or hand) and specifically
configurations mimicking the interaction with objects of different size affect the
control of the simultaneous grasp with another distal effector (hand or mouth)? An
affirmative response to the question can support the idea that representations of the
interactions of a distal effector with objects are sufficient to affect the control of
the grasp of another distal effector. In addition, it excludes the possibility about a
simple temporal coupling between the movements of the two effectors. Second, is
the reciprocal interaction between distal effectors restricted to hand and mouth or
can be it extended to foot postures?
Experiment 1: reaching with the mouth and grasping with the
mouth after assuming the posture of power grasp, precision
grasp and relaxed hand.
Experiment 2: reaching with the mouth and grasping with the
mouth after assuming the posture of extended, flexed and
relaxed fingers
Experiment 3: reaching-grasping with the hand after opening.
closing and relaxing the mouth
Experiment 4: reaching-grasping with the hand after
extending, flexing, relaxing the toes.
We examined the possibility that speech can affect grasp by
considering that vocalizations require particular postures of the
internal mouth (Leoni and Maturi 2002). The open vowels, such
as /a/, are related to large internal mouth apertures, whereas
closed vowels, such as /i/, are related to small internal mouth
apertures. If external mouth postures affect the control of grasp
(experiment 3), it is possible that even specific internal mouth
postures assumed for pronunciation of vowels affect the control
of grasp.
Experiment 5: reaching to grasp the objects while producing
different vocalizations, i.e. /a/, /ɔ/ and /i/.
THE CONTROL OF POSTURES OF
HAND AND MOUTH RATHER THAN
TOES, AFFECTS THE CONTROL OF
GRASP WITH MOUTH AND HAND.
VOCALIZATIONS (OPEN AND
CLOSED VOWELS) AFFECT THE
CONTROL OF HAND GRASP
We found that deictic gestures (HERE, THERE, Chieffi
et al. 2009) as well as communicative gestures
(CIAO,NO, STOP,Barbieri et al. 2009; Bernardis and
Gentilucci, 2006; Gentilucci et al. 2006) reciprocally
interact with the words of the same meaning.
We tested the hypothesis whether a system relating hand
gestures to internal mouth postures used to vocalize
exists and it can be considered precursor of a system
relating gestures to words.
Effects of the production of representational gestures
LARGE and SMALL (unimanual and bimanual
gestures) on
1) Production of vowels (“A”, “I”; experiment 1)
2) Pronunciation of words (“GRÀNDE”, large,
“PÌCCOLO”, small) (experiment 2)
3)Pronunciation of pseudo-words (“SCRÀNTA”,
“SBÌCCARA”, experiment 3)
Production of the vowels “A” and “I”
Pronunciation of the words “GRÀNDE” (large)
and “PÌCCOLO” (small)
Pronunciation of the pseudo-words “SCRÀNTA” and
“SBÌCCARA”
A system coupling unimanual gestures to vocalization exists
and may be considered precursor of a more distributed
system relating gestures to words.
This system, on the basis of word meaning categorized the
gestures in two classes (LARGE and SMALL) creating a
size representation independent of effector. This influenced
the pronunciation of the vowels included in the words. The
gesture LARGE (bimanual condition) induced an increase in
formant 1 (F1). The increase in F1 corresponds to an
increase in internal mouth aperture. Conversely, the words
influenced the hand aperture.
Pronouncing pseudo-words was responsible for a size
representation not yet independent of the gesturing effectors.
fMRI (Willems et al. 2007, Xu et al. 2009) and rTMS (Gentilucci et al.
2006) experiments suggest that il circuit formed by Pars opercularis,
Pars triangularis, and Pars orbicolaris of Inferior Frontal Gyrus is
involved in the integration between gesture and speech.
Thank you! and…