Transcript A NEC Demonstration_5

Collaborative Jazz Improvisation
as an example of Cognitive and
Network Extension
David Mott, John Ibbotson, IBM
12th May 2009
Structure
• Introduction
– extended cognition
– jazz playing
•
•
•
•
Jazz as collaborative extended cognition
Jazz as an networkable activity
Experiments
Relevance to ITA
Introduction
What is Cognitive Extension?
• Coupling with external artefacts
– counting on fingers
• Socially extended
– language to share thoughts, refine concepts, execute
plans
• Sense of self
– affected by interaction with others
– confidence as an improver of performance
[The Extended Mind, Clark and Chalmers]
Examples that convince me of
Cognitive Extension
• Driving a car:
– the car “drives itself”
– I have the same “feeling of vulnerability” when
I turn that I have with my own body
• Playing jazz music
– (on a good day) the piano “plays itself”
• Consider the jazz example …
Leaving artefacts in the
environment
• augment memory
– memory == existence of an entity in a place
– keys in same place
– (plant triggers in memory)
• reminders/requests to do things
– notes, paper trails
• for self and others
Using tools
• notepads
• calculators
• hoey [cognition in the wild]
Is jazz cognitive and capable of
cognitive extension?
Jazz group
• instruments are tools
• need to collaborate
• need to problem solve
Use of patterns
• Some aspects of performance is realtime
– no time to construct totally new melody lines
– people use patterns previously learnt
– but not necessarily exact note-for-note, so
cognition is still required
• Other aspects are less realtime
– who is doing next solo
– this can be done by problem solving
Is playing jazz NEC?
•
Is it Cognitive?
–
–
•
Is it network?
–
–
–
–
•
out of tune means you cant play
good key action leads to better playing
good sound quality leads to better playing
there is an intimate connection between tool and mind
Do the tools interact?
–
•
must synchronise, respond, and lead other members of the group (different roles for same
player at different times in an ad hoc way)
If you cant hear, you cant play
If the “groove” is good, you play better (mismatch of different player’s rhythms can destroy
playing)
If the “cross-musician feeling” is good you play better (eg take up of others improvisation and
rhythmic ideas)
Does the tool (the piano) assist?
–
–
–
–
•
a complex problem solving activity involving sequencing, tempo, structure (you only have
abstract chords for guidance), in real time
Improvisation requires more than just muscle memory
bad matching (eg guitar and piano in same “tonal register”) can destroy playing
Therefore playing jazz is definitely NEC!
–
–
The sum is significantly greater than the whole, and the network and tools are essential
(I am talking here about quality of playing - achieving a “jazz feel”, not basic note work)
Do jazz performers leave artefacts?
• Does one performer leave something for
later use?
– to another performer
– to himself
– to the “environment”
• e.g. start a song with a certain rhythm,
tempo?
– really setting up a pattern
Instruments as tools
• Does the particular layout of the
instrument enhance the cognitive
extension?
• Most jazz tools have a visual component
(keyboard “lays out” chords spatially but
different transpositions have different
patterns)
Do performers “play” other
instruments (use other tools)
• Does piano player “play” the drums?
– piano player does play rhythmically
– if the piano is “heard” as drums then this is
like playing a drum?
– eg the passing drum rhythm to drummer
• Does piano player “play” the bass?
– e.g. if bass is lost then piano can play the
bass line to resynch
Jazz performance - Tactical and
Strategic
• Strategic
– Book the hall and the group
– Advertise
• Tactical
– perform a piece
Can jazz be network extended?
Utility of the Network
• To achieve the quality of face to face?
– when communication MUST be distributed, with
reduced communication, can the network rise to the
same quality as face to face?
• To improve over face-to-face?
– Are there situations where the network can enhance
performance eg
•
•
•
•
training
direction
better awareness of what others are doing
better awareness of holistic view of performance
What does NEC give you?
• People playing jazz together play better
than on their own
• Other players contribute to creativity
• But group must also be disciplined for this
to happen
– some “maverick” behaviour (strange chords,
rhythms, phrases) may add to creativity, but
too much is disruptive
Confidence
• Confidence is key to quality of playing and
integration
• Easy to lose confidence if group not
working together, not synchronised
• Can the players help increase confidence
• Can the network help increase confidence
• Might a bad network reduce confidence?
• Rhythmic stability seems very important
Rhythmic collaboration (1)
• Ending a song
• Actions
– looked at the drummer and played a rhythm
– taken over by drummer
– subsequently mutual ending
• Needed
– visual communication
– music communication
– shared schema/pattern
Rhythmic collaboration (2)
• Guitar lost synchronisation
– Save the Last Dance for Me has unusual 5-bar
phrase lengths
– Guitarist changed chord too early
– Pianist emphasised new chord at the right place and
guitar resynchronised
• Needed
– music communication (both ends)
– musical “validation” ability
– shared memory of song
Schemas
• Schemas are typical patterns/stories we
expect to happen
– “eating at restaurant”
– “dodgy accountants”
– enemy movements?
• Musical Schemas
– 4 bar phrases
– “Surprise Symphony” [play example]
– jazz structure: tune, solo, tune
What is communicated?
• Organisational
–
–
–
–
who is to solo next
when a solo is finished
timing of piece
when to complete a piece
• Leadership
– leadership passes between the players, semi ad hoc
• Musical Patterns
–
–
–
–
Tempo, eg change to double time
Style, eg “bebop”, “latin”
Phrases, specific phrases may be echoed, or modified
Harmony, eg player may lay down a chord substitution
and this is taken up by others
Communication of Information
•
Rhythmic Metre (beat), the gross timing of the piece
•
Rhythmic Style, the gross “type” of piece, eg Latin, Swing, Rock
•
Rhythmic Confidence/Stability, whether the drummer is reliable
•
Harmonic Progression, roots
 sync the “tapping of feet”
 selection of accompaniment pattern
 personal confidence
 need to “lead the rhythm”?
 sparseness of accompaniment
 sync the place in the sequence
 more space to play richer “higher voicings”
• Harmonic Progression
 “where I am”???
Communication of Information
• Rhythmic Metre (beat), the gross timing of the piece
 sync the “tapping of feet”
• Rhythmic Style, the gross “type” of piece, eg Latin, Swing, Rock
 selection of accompaniment pattern
• Rhythmic Confidence/Stability, whether the drummer is reliable
 personal confidence
 need to “lead the rhythm”?
 sparseness of accompaniment
• ?? “easier to play if the piano sound is good” ??
Other’s tools
• Is it possible to extend the mind into others
instruments?
• In a genuinely collaborative situation can
this achieve a deeper common
understanding?
• Would need intimate feedback between
other tool and ones brain, via network
Key collaborative performance
aspects
• Synchronised rhythm
– style (accents, patterns) varies (swing, latin)
– key aspect for a good performance
– key for confidence
• Harmonic structure
– where are we in the chord sequence?
• Organisational structure
– when do we end and change solos?
Creating jazz NEC
How do we create a Jazz NEC?
• MIDI instruments:
– keyboard
– midi drums
– midi guitar (or use keyboard)
• MIDI is a stream of digitised events that can be
communicated, instrumented and recorded
• Setup a MIDI “group” across the network with
remote players, and play!
– connect via sensor fabric
– build OpenSim module to communicate MIDI?
NEC over the Sensor Fabric
Synth
Synth
Sensor
Client
Sensor
Client
Fabric
Node
Synth
Sensor
Client
Dashboard
Data
Plugins
Control
Control
Experiments
How do we evaluate a Jazz NEC?
• Evaluation is “quality of jazz feel”
– must be done by other musicians (or keen listeners)
subjective feel
– BUT I believe you will get a good degree of
agreement (you can tell when a band is “together” or
“tight”)
• Could you analyse the MIDI stream
mathematically?
– Not sure, you might pick up “gross defects” (mismatch
of rhythm, complete collapse of group), but for the
“jazz feeling” I suspect this is elusive
What can be measured?
• Degree of “music playing itself”
– subjective judgement by player
• post hoc listening and recall (there I was playing well, there I got
lost)
• additional effector (foot button?) pressed to indicate “im now in the
flow”
• Emotional State
– Emotive-like devices, measuring emotion, mood, physiological
factors
• Group review by players
– post hoc listening and recall
– discussion of what was going on, issues (I couldn’t hear the
guitar, the drums were lagging, I felt the group was not together)
• …
Experiments
• Effect of network quality
– can artificially reduce MIDI signals between players
• reduce volume so cant hear drummer
• simulate bandwidth reduction by dropping packets
• Effect of face-to-face presence
– playing over headphones vs being in visual contact
• Effect of injecting information
– put drum “double time rhythm” onto the network to
see if it is taken up by others
• If things go wrong, can the group recover?
QOI
• Quality of Information measures could be
applied to performance?
Measuring vs changing
• Could just measure a performance
• Could introduce a change and see the
effects
• It may be premature to experiment with
the effects of introduced change
Simple intervention – play to click
• Rising to f-f:
– prioritise click over full drums, given reduced
bandwidth
– agent to approximate drums and send to players
• Improving f-f
–
–
–
–
already used in real performances
would click give confidence to new players
intervene with double time click?
auto quantise drums – would this increase
confidence?
BeatFinder
• Locate “beats” in music
– 1, 2, 3, 4, 1, 2, 3, 4
– a complex task, lets assume we can do it
• Show coloured screen for “in time”, “early”
and “late”?
• Detect individual accuracy of beats
• Detect collaborative accuracy of beats
• Work out and display where in the music
we are
Displaying place in tune
• use beat tracker to calculate place in tune
– jazz navigational structure usually fairly
simple
– would need to visualise place in context?
• would this improve confidence in new
player?
• or take away the skills learning?
Experiments with beat finder
• Surely groups already synchronise their playing?
– NO, not if nervous or multiple versions of rhythm and
not clear where beat is
• Possible experiments:
–
–
–
–
For individual practise
Dashboard view of collaborative performance
See other players beats (all, or selected)
One player (drummer, trainer) pushes a
synchronisation pulse to other players
• What effects would this have?
Using network for training
• transcription/copying is very important for
learning
• could a trainer play examples to copy?
– in the context of a networked performance
– note-for-note copy or approximate patterns
• could a trainer give real-time cues?
The relevance to ITA
What is the point of jazz in the ITA?
• An example of extended mind
– well understood, easy to relate to (for some)
– an articulation of the abstract cognitive
concepts
• NATO uses it as an example of C2
– ????
• Realistic test bed of some complex
experiments
• Stimulation of new ideas
Metaphors
• (Lakoff) Cognitive metaphors bridge from
concrete to abstract
– LIFE IS A JOURNEY
– TIME FLOWS FORWARDS
– EMOTIONS GO UP AND DOWN
• (Saslow) Musical Metaphors
– MUSIC IS A PATH
– MUSIC IS A FORCE
The FORCE Metaphor
MUSICAL EXAMPLE
Military TERMS
DEFENCE, SEIZE
Jazz as extended mind
• coupling with external artefacts
– finger positions on keyboard to calculate harmonic
progressions
• socially extended
– other people as memory of tunes, styles, rhythms,
harmonic progressions
– synchronisations of rhythms
• sense of self
– jazz playing in social context affects sense of self