Transcript Write an interview question

Playtesting Workshops:
Playtesting To Refine
Judeth Oden Choi, Jodi Forlizzi, Michael Christel, MacKenzie Bates, Rachel
Moeller, Jessica Hammer
Carnegie Mellon University
1
Recon Activity
•
Walk around the room and read other
teams’ recipes.
•
Leave 3 “That’s cool!” comments on sticky notes.
•
Leave 3 “I’m curious or confused” questions on
sticky notes.
2
Refine Is For You If:
•
You are working on a prototype
•
(lo-fi, hi-fi, a mod of an existing game, a single
mechanic, or a whole experience)
and
•
You know your player experience and/or design goals
3
Why Playtest to Refine?
What you know:
• You have a set of game/experience features you want to use
• You know the desired affect you want your experience to have.
What you want to learn:
• How does this set of game features affect the experience?
• How can I improve the experience?
• How can I improve my goals?
4
Recipes & Sketches
You already selected ingredients for your
prototype that connect to your player
experience goals.
5
Why Recipes?
A prototype is a hypothesis.
•
You are hypothesizing that your design
(your set of ingredients) will create a certain experience. You can
test this hypothesis through playtesting.
Use your recipe to formulate questions.
•
How do your ingredients affect the experience?
Read the sticky notes on your recipe.
•
Select (or write) three questions about your prototype. What are
you curious about? Where do the uncertainties lie?
6
Methods in Refine
• Observe
to capture behavior
• Think-Aloud
to capture thoughts, process
• Interview
to capture motivations, feelings, attitudes
• Survey
to capture structured data
7
Observation
8
Observation Tips
Look for ACTIONS
• Vocalization
• Gestures
• Reactions
Look for CHANGE
• Shifts in posture
• Change of facial expression
• Shifts in focus (or eye contact)
9
Notetaking Tips
• Situate notes within game
• Use time codes, level, or event
• Withhold Interpretation during test
• Do NOT Interrupt. If you must help the
player, make a note of it.
10
Let me repeat
DO NOT INTERRUPT
…don’t congratulate
…don’t help
…don’t explain the UI
…don’t apologize
…unless the player is really, really stuck.
(If they do get stuck, make a note of it. It’s really
valuable information!)
11
Recording Tips
• Use video and audio to capture behavior.
• Use video or in-game tools to capture interface.
• Use photos to document moments. Consider framing.
• Give players space. Consider their comfort.
• Get permission.
• Review and notate immediately after playtest
12
Uno Playtest
Player experience goal:
competitive social interaction.
Modified the mechanics to create 4 prototypes
How do the mechanics affect the player experience? Can
you observe a difference?
13
Think Aloud
http://www.doctordisruption.com/design/design-methods-27-think-aloud-protocol/
14
Think Aloud
15
Interviews
Photo by Heinrich-Böll-Stiftung
16
Ask good questions
Questions should be:
1.Important
2.Accurate
3.Answerable
17
Ask Important Questions
Important questions relate to your player experience goal.
1. Did you like the music?
1. Uninformative
2. What feelings do you
associate with the music?
2. Important if your goal is to
iterate the music
3. What parts of the game made
you feel _(confident)__?
3. Important to help assess current
success status of game
4. Which feelings hadn’t the
4. Important … but difficult to test
greatest impact on your ability
to learn _(fractions)_?
18
Ask Accurate Questions
Accurate questions address the player's relationship with
the goal.
1.
2.
3.
4.
Is this a good game?
Is the game fun?
Why is this game fun?
I noticed you laughed during
x? What was going through
your mind? How did you feel?
5. I noticed it took several tries to
get through x? How did it feel
to encounter that challenge?
1.
2.
3.
4.
What does good mean?
How do you identify fun?
Will the player even know?
Measurable way of identifying
player enjoyment, but only
looking for positives
5. Measurable way of looking for
player enjoyment and struggles
19
Ask Answerable Questions
Answerable questions capture the aspect of your game
you're focusing on.
1.
How much did you like the game?
1.
Assumes they liked it
2.
Did you like the game?
2.
Suggests yes/no answers
3.
What did you think of the game?
3.
Open-ended, good for exploring
4.
What was your favorite moment in
the game?
4.
Concrete examples can get at deeper
thoughts
5.
How would you describe this game to
someone who had never played it
5.
before?
Teach-back technique helps players be
more precise and accurate themselves
20
Writing Interview Questions
Refer back to the questions
you generated at the beginning
of the workshop.
Write an interview question that you might ask a
player after a playtest of your prototype.
21
Surveys
22
Surveys
Benefit: easy to automate, pre-structured data
Challenge: writing good questions is hard!
Flaw: GIGO (Garbage In, Garbage Out)
Strategies
•Iterate questions over the semester
•Try out questions on friends and advisors
•Include multiple-choice and open questions
•Keep it short
23
Likert Scales
•Degree of agreement
•Can also measure frequency, importance, quality, likelihood
•Use 5 or 7 categories
•Clearly and accurately labeled
•Low to high, arranged left to right
•Equal number of positive and negative
24
Survey Bias:
Leading
(1) How fast was car A going when it turned
right?
(2) Did you see a stop sign?
35% answered yes to (2)
(x) How fast was car A going when it ran the stop
sign?
(y) Did you see a stop sign?
53% answered yes to (y)
25
Survey Bias:
Anchoring
26
Survey Bias:
Anchoring
27
Survey Bias: Anchoring
28
Survey Bias: Anchoring
https://www.youtube.com/watch?v=HefjkqKCVpo
29
Validated Survey Questions
Pre-existing surveys:
• Immersive Experience Questionnaire (IEQ)
• Player Experience of Need Satisfaction (PENS)
• Game Experience Questionnaire (GEQ)
Use validated survey questions ONLY IF they align with
your questions.
30
Writing Survey Questions
Writing accurate and answerable questions:
Writing important and relevant questions:
31
Writing Survey Questions
Phrase one of your questions as a
survey question.
Will you use…
Multiple choice
● Likert scale
● Open response
●
Share your question!
32
Planning a Playtest To Refine
33
Step 1: Why are you testing?
1. Identify your goals.
2. Identify your questions.
•
How does this set of game features affect the
player experience?
•
How can I improve the experience?
•
How can I improve my goals?
34
Step 2: Select Methods
1. Observation
•
•
•
•
Notetaking
Photography
Video & audio recording
In-game measures
2. Think-Aloud Protocol
3. Interview
• Ask important, accurate & answerable questions
4. Surveys
•
•
Iterate, iterate iterate
Reduce bias
35
Step 3: Find Players
1. Do you have access to your target
audience?
•
If not, what’s the next best thing?
1. Consider your playtesters comfort
•
•
•
What is their comfort-level with the technology?
With the the type of experience?
With being observed?
36
Step 4: Select a Setting
In the wild or the lab?
37
Step 5: Set-up
1. How many team members do you
need? Who fills what role?
2. Where will you place the experimenters in relation
to the players?
3. On-Boarding
•
•
•
Write an introduction script.
Prepare written instructions.
Create mock-ups of unfinished interfaces.
4. How long is each playtest? How many rounds can
you schedule?
38
Why playtest?
Each prototype is a hypothesis. Keep testing and
iterating on your hypothesis.
39
Check Out
Contact us:
[email protected]
http://playtestingworkshops.com
40