Transcript Lucia Riggio - Embodied language II

How language incorporates affordances
Lucia Riggio
Department of Neuroscience
University of Parma
Cambridge 3 September 2013
Affordances
 Affordances refer to properties of an object or to specific
parts of it (typically the handle) that trigger components
of actions such as reaching or grasping
 Visual perception of objects activates the very system
responsible for their actual manipulation, that
transforms object pragmatic features into appropriate
motor programs for their interaction
Handle graspability
Visual presentation of manipulable objects in which the most important feature relevant
for action is violated (e.g., a cup with a broken handle) prevents motor activation from
occurring.
[ Buccino et al 2009 ]
Object graspability
Dynamic responses (i.e. the grip
force) to a set of spheres with
increasing size were no longer
correlated with the apparent
weight of the stimuli, if the
experimenter referred to them as
planets, rather than graspable
solids
[ Taylor & Zwaan 2010]
Language
Embodied theories predict that nouns of graspable objects
should also activate affordances since understanding of
language is achieved by recruiting the same sensorimotor
systems activated when we experience the action or
object a word refers to.
Verbs
 Presentation of action verbs associated with different effectors results in somatotopic
activation of motor areas (e.g. Pulvermüller et al 2001; Hauk et al 2004; Tettamanti et
al 2005)
 Modulation in the MEP for a muscle of a given effector associated with the verb
expressed in a sentence (e.g. Buccino et al 2005)
 Modulation of RTs when the effector used to respond is also involved in the action
expressed by the linguistic material (e.g. Buccino et al 2005; Boulenger et al 2006;
Sato et al 2008; Dalla Volta et al 2009) or when the response requires the execution
of a movement in the same direction of the action described by the sentence (e.g.,
Glenberg & Kaschak 2002)
Verbs : interference and facilitation
 Interference effects within 200-250 ms after the
presentation of action-related word: slowing–down of the
RTs and decrease of the MEPs when either the same
effector as that expressed by the action verb or sentence
(e.g. Sato et al 2008; Buccino et al 2005) or the same
directional movement are involved (de Vega et al 2013).
 Facilitation effects in the matching conditions for longer
time intervals (e.g. Glenberg & Kaschak 2002; Zwaan &
Taylor 2006; de Vega et al 2013).
Nouns
Is the early involvement of the motor system, observed
during the understanding of action-related sentences
and verbs, also present for nouns?
Nouns: Materials
Hand-related
nouns
Foot-related
Abstract
nouns
nouns
Forbici (scissors)
Pedale (pedal)
Tattica (tactics)
Tazzina (cup)
Pantofola (slipper)
Superbia (arrogance)
Spazzola (brush)
Pedana (footboard)
Ritegno (reluctance)
Forchetta (fork)
Pattini (skate)
Elogio (praise)
Pettine (comb)
Gradino (step)
Deferenza (deference)
Matita (pencil)
Mocassino (moccasin)
Gelosia (jealousy)
Accendino (lighter)
Scalinata (staircase)
Lusinga (allurement)
Martello (hammer)
Ciabatta (slipper)
Inganno (deceit)
[Marino et al 2013]
Nouns: behavioural experiment
Nouns: behavioural experiment
Nouns: TMS experiment
Nouns: TMS experiment
*
Nouns: conclusions
 Both experiments indicate the early involvement of the left motor
cortex in the representation of nouns of graspable objects, in line
with a causal role of the motor system in language understanding.
This modulation is comparable to that previously found with verbs
or action related sentences (e.g. Hauk & Pulvermüller 2004;
Buccino et al 2005; Boulenger et al 2006; Sato et al 2008).
 The differential pattern of results between the two hemispheres is
consistent with the dominance of the left hemisphere for language
and tool processing.
Nouns, as their external referents, elicit motor information.
Stable and variable affordances
 Stable affordances: stable aspects of the interaction with objects,
which can be included in an object representation stored in memory
 Variable affordances: variable aspects, which change depending on
the way the object is presented
 Canonical affordances: variable aspects associated with a canonical
interaction
The comprehension of words referring to a particular object activates only
stable and canonical affordances .
[Borghi & Riggio 2009]
Objects and nouns in space

The activation of motor information associated with
visual objects is spatially constrained (e.g. Costantini et al
2010; 2011)
Is motor information elicited by nouns spatially constrained?
Experiment 1: objects
Stimuli were presented in a 3-D room created by means a 3-D Studio MaxTM
Reach-to-grasp response device
Task: Participants had to signal whether an object, presented either
within or outside their reachable space, was natural or manufactured
by making reach-to-precision or reach-to-power grasp responses.
[Ferri et al 2011]
Experiment 2: nouns
Task: Participants had to signal whether the object (near or far) was the same (congruent)
as that indicated by the word, or different, producing reach-to-precision or reach-topower grasp responses.
Results
Experiment 1: objects
An action compatibility effect between the
response and the grip evoked by the object,
only in the reaching space and only for
artefacts
Experiment 2: nouns
An action compatibility effect between the
response and the grip evoked by the noun
that was not modulated by the spatial
position of the visual object
 Variable features such as distance and location are not integrated
in the motor representation of the objects elicited by language.
Conclusions
 Action information relative to the size of an object is spatially
constrained for visual objects but not for their nouns.
 Space information elicited by a seen object is different from
that elicited by nouns.
Converging results
Examples of stimuli
Task: Right-handed participants had to decide whether the paired objects
would typically be used together.
Results: Decision times faster when stimuli were in the correct locations for a
right-hand action (active member on the right) only when objects were
presented.
[ Yoon et al 2010 ]
Combinations of nouns and verbs
Do concrete verbs and nouns combined in single
sentences have a different weight on the modulation
of the motor system?
Sensorimotor specificity of verbs and nouns
Material and Procedure
Sensible and non-sensible sentences formed by concrete verbs and nouns were presented
in a sensibility judgment task. Verbs could express an action that could be performed in a
specific way (low degrees of freedom) or in very many different ways (high degrees of
freedom). Nouns referred to graspable and non-graspable objects.
VLDoF
VHDoF
Sensible
Non-sensible
To squeeze the orange
To water the handle
To recycle the bottle
To consult the onion
To water the flowerbed
To squeeze the sunset
To consult the archives
To recycle the herd
Noun of graspable object
Noun of non-grasp. object
[Marino et al 2012]
Results
 With Verbs with LDoF performance is independent on the noun type: activation
of specific features of objects on which the action is usually performed
 With Verbs with HDoF performance is affected by the noun type
In short
 The motor system is modulated by concrete nouns in a way
comparable to concrete verbs
 Nouns seem to activate only stable and canonical affordances.
Distance and location, related to the reaching of the object, seem
not to be represented.
 When nouns are combined with verbs the sensorimotor
specificity of both words could have a role in the motor
activation.