Transcript Face-to-face conversation as the basic form of communication

Role of Visual Information
Referential Communication Results
•
Multiple 2-person
referential communication
tasks
• E.g., Find nearest doctor on
map
• E.g., Build trash cart
•
Common results:
• Voice speeds solutions
compared to typing
• Faster times
• More turns
• More words
• Visual channel doesn't help
(in a talking head set-up)
70
60
Face to face
Voice
50
40
Writing
Typing
30
20
10
0
Solution time
35
30
Voice
Voice+video
Writing
Writing+video
Typing
Typing+video
25
20
15
10
5
0
Solution time
(fm Chapanis, 1972)
"Talking-heads" video doesn't improve referential
communication
•Tacit
assumption in using video
• Since face-to-face communication is
successful, communication media
that
are more like face-to-face are better
•Most
of the content is in the words
•Gestures
may be pre-verbal, rather than
illustration
•For
emotion, video and audio channel
can be redundant
•Rich
media are useful for handling
ambiguous and conflictful topics
• E.g., Images change lie-detection,
but help liar over the lie-detector
https://www.youtube.com/watch?v=0kd2SO1_kSA
Seeing your partner doesn't
improve ability to communicate
about objects in the world:
Physical Co-presence Provides
Visual Cues for Grounding
Type of Visual Information
•
•
•
Side-by-Side
Collaboration
Head-Oriented Task-Oriented
Video Systems Video Systems
Others' heads and facial expressions
Yes
Partial
No
Others' bodies and actions
Yes
Partial
Partial
Task objects
Yes
No
Partial
Work Environment
Yes
No
Partial
Physical co-presence: People are present at the same time in the
same place
Provides several types of visual information that can be used to
ground utterances.
Task-oriented video systems vs. “talking heads” video:
– Advantages: View of others’ hands and of task objects allows monitoring of partner’s
attention and comprehension; shared view of objects allows for efficient reference.
– Disadvantages: No feedback from others’ facial expressions
Bicycle Studies: Method
•
•
Bicycle repair task
25 pairs of participants
– “Helper” provides instructions
– “Worker” repairs the bike
•
Communications Media
– Side-by-Side
– Audio/Video: head-mounted camera showing workers’ hands and
field of view
– Audio-Only
•
•
Within-subjects design
Dependent measures
– Task completion times, observer ratings of work and communication
quality, surveys, content-coded transcripts
In the Video Condition Pairs Used Visual Space
•
•
Grounding: Used visual space to disambiguate
reference, establish common ground, & identify task
elements
Situational awareness: To know when worker was
ready for next instruction
– Example(219_s_v.mov)
•
But technical flaws limited its utility
Bicycle Study: Key Findings
•
Performance is best with full
physical co-presence (Side-bySide)
Communication with video
system was more efficient than
audio-only.
– Workers use deictic pronouns for
task objects/locations when they
think the helper can see them
•
The head-mounted video system
used in this study did not
adequately support shared visual
space.
Mean Task Completion Time (Minutes)
•
14
12
10
8
6
4
2
0
Side-by-Side
Video
Media Condition
Audio
Exploring the Role of Shared Visual Information
•
What features of physical space influence its
value?
– Fidelity of views
– Hypotheses: Delay, rotation & host of other factors
that make views dissimilar will degrade collaborative
performance
•
When is shared visual context most important?
– Visual complexity
– Hypothesis: When task is complex enough that
language itself is insufficient to efficiently describe
events
New Research Paradigm to Test Those
Questions
•
Stylized referential communication task
– To increase control
– To systematically vary task attributes
•
Construct artificial shared visual environment
– Allows independent manipulation of features of a shared
space that co-occur in the real world => identify which are
important
Cooperative Jigsaw Puzzle Task
•
•
Helper has picture of target and gives instructions to
worker, who moves pieces to match target
Subjects communicate via audio & shared computer
screens
Work area
Staging area
Shared view
Target
Manipulations
Task visual complexity
Simple
Complex
Primary colors
Tartan plaids
Static colors
Changing colors
Pieces abutted
Pieces overlapped
Visual fidelity
– None: Audio only
– Partial
• Shared screen with a 3-second delay
• Shared screen with rotation
• Shared screen with a small view port
– Immediate: Full shared screens with no delay & no rotation
Summary of Multiple Experiments
•
Task performance
– Shared visual space improved task performance
(speed & accuracy) in all experiments
– Improved performance most for visually complex
tasks
•
Shifted conversational strategies
– Shared visual space improved improved
efficiency of reference (e.g., words/reference)
– Lack of shared visual space forced many
workarounds
Experimental Manipulations
Fidelity of the Visual Space
• Immediate
• Delayed (3 seconds)
• None
•
Other studies
– Rotation of the spatial perspectives
– Discontinuous, “push to see” images
Immediate condition
Visual difficulty:
• Static vs. Dynamic Tasks
•
Other studies
– Spatially easy vs. difficult puzzles
– Easy versus difficult to name objects
– Same vs. different visual perspective
No SVS condition
How Does SVS Change Communication
How are pairs communicating differently when they have a shared
visual space?
Communication is more efficient with a shared visual space
•
•
•
•
•
Helper uses actions to assess worker’s comprehension
Both helper and worker use more efficient referring expressions
and diexus
Helper can precisely time interruptions and corrections
SVS facilitates an awareness of the task state
Without shared visual space
•
Worker becomes responsible for updating helper on state of the
task
Shared Visual Space Is More Important in
Changing Environments
Shared visual space improved performance
Effect of Shared Visual Space on
Puzzle Completion Time
•
100
3 second delay led to significant decrease in the
value of the shared space
80
Seconds
Immediate significantly faster than delay and no
SVS (p<.0001)
60
40
20
0
Immediate
Delayed
None
Fidelity of Shared Visual Space
Task Com pletion Tim e by Shared
Visual Space and Color Drift
Seconds
150
Drift
100
50
Stable
0
Immediate
Delayed
None
Fidelity of Shared Visual Space
Shared visual space was more important when the
objects were changing (i.e., hard to describe)
•
Immediate affected significantly less than delay
(p=.05) and no SVS (p=.0002)
Shared visual space is less important when words
can easily describe the objects and environment
Rate of Word Production:
Workers Increase Their Speech Rate
Word Rate by Shared Visual Space and
Speaker
The fidelity of the SVS influenced Workers more than the
Helpers (F(2,110) = 10.80, p < .0001)
5
4.5
Words Rate per Trial (nLog)
Pairs increased speech rate when fidelity decreased (all p<.02)
4
3.5
Helper
The pairs adjusted their use of language to accommodate for
lack of shared space
3
2.5
•
Workers increased their speech rate to compensate
2
1.5
The accommodation was insufficient in comparison to when
the pairs made use of the shared visual space
1
0.5
Worker
0
Immediate Delayed
None
Fidelity of Shared Visual Space
Conversation
Issuing Acknowledgements
Workers took a more active role in ensuring messages were understood when
there was no shared visual space (i.e., when the helper could see them).
Immediate SVS
No SVS
H: The, the right hand, the top right hand
corner of the blue block touches the bottom
left hand corner of the first orange block.
W: [Positioned piece correctly]
W: Like that?
H: Yeah.
H: All right that's good.
H: And that's gonna be on top of the red one
but only the right side of the red is going to
be showing.
W: [Positioned piece correctly]
H: You know what I mean?
W: OK, so it's like...
H: Oh, like, put it on the left side of the red.
W: ...side of it and you see half of the red
block.
H: Right, of the red, Yeah.
W: OK.
Acknowledgements of Understanding: Pairs Use SVS to
Monitor Comprehension
Acknow ledgem ents of
Understanding by Shared Visual
Space and Speaker
Acknowledgements of
Understanding per Trial
6
Worker
5
4
Pairs were most explicit in stating their
understanding when they had no shared visual
space
Workers were more explicit in stating their
understanding when there was no shared visual
space available
Typically the Helper gave directives and the
Worker moved pieces
3
•
2
Helper
•
•
1
0
Immediate
Delayed
None
Fidelity of Shared Visual Space
Used SVS to monitor understanding
Reserved language for breakdowns
Without the SVS the pairs substitute language to
confirm understanding
What About Feelings of Connection?
(Garau, Slater, Bee, Sasse, 2001)
•
•
•
Study comparing f/f, avatar w/ realistic gaze, avatar w/
random gaze & audio communication
Negotiation task – avoid a scandal
DVs
– Face-to-face feel, Involvement, co-presence, partner
evaluation:
Yee et al (2007) Meta-Analysis
Comparison of 25
experiments
– People interact via text or
voice
– Presence of avatar
• None
• Unrealistic (cartoon)
• Photorealistic
– Outcomes
• Performance
• Subjective evaluations
of experience or
partner
•
•
Performance &
subjective evaluations
improved with avatar
Realism only influence
subjective evaluations
0.16
Correlation (r) btw avatar type and
outcome
•
0.14
0.14
Performance
0.13
Subjective
0.11
0.12
0.1
0.08
0.06
0.04
0.02
0
-0.02
No avatar vs
avatar
Cartoon vs
photorealistic avatar
-0.04
-0.06
-0.04
Avatar type
Yee, N., Bailenson, J., & Rickertsen, K. (2007). A meta-analysis of the impact of the inclusion
and realism of human-like faces on user experiences in interfaces CHI '07 Proceedings of the
SIGCHI conference on Human factors in computing systems (pp. 1-10). New Yor: ACM
Apple Facetime
https://www.youtube.com/watch?v=iu1jHtf_oUc