Transcript Player Usability: 5 examples - Game Analytics Resources v. Anders

Digital storytelling,
online behavior
User
behavior,
data mining
Communication
in games
Digital/
Social
Media
Computer
Science
Behavioral
economics
Persuasion,
value,
learning
User
Information
Science
Game
Industry
Development,
game
economics
HCI
Play experience,
design

MSc. In Natural Sciences


PhD in Computer Science


Game piracy, behavioral economics, co-creation – more game telemetry data mining
Assistant Prof., Department of Communication, Aalborg University


Play experience, biometrics, game data mining, game development
RA/project lead, Department of Informatics, Copenhagen Business School


Empirical evaluation of games, HCI, user testing, game telemetry
Post doc. At the Center for Computer Games Research, IT University
Copenhagen


Large-scale trends and evolutions in time/space, Geographic Information Systems
Yet more game data mining, more game development, more innovation
Co-Founder & Lead Game Analyst, GameAnalytics

Tools and consulting on application of game telemetry to development

90% applied research

10% theory (play experience, play personas)

Collaboration with industry – real needs

Collaboration with international colleagues
 1 single-authored publication ...

Focus: How users interact with IDE
applications and each other + the business
side

Game User Research – answering e.g.:
 Who are the users of interactive digital entertainment
products?
 What do they do and where, with whom and why?
 How do we develop products for different users?

Multi-disciplinary ”field”
 Researchers from CS, HCI, communication, design, media,
psychology, AI, art, economics, development ...

Emergent field – lack of established theory

Exponential growth in research publications

Backed by a growing industry where users are
central

Four main lines of investigation in GUR:
 Usability: Can the user operate the controls?
 Playability: Is the user having a good experience?
 Behavior: What is the user doing while playing?
 Development: Integrating GUR in business practices

Why interesting?
 New field of research
 Emerging methodologies + theories
New field
 Plenty of tough problems
 Collaboration
 Broad relevance
Multi-disciplinary
 Multi-disciplinary
 Affects millions of people
 Industry interest
 Latest technologies
Impact
Digital storytelling,
online behavior
User
behavior,
data mining
Digital/
Social
Media
Computer
Science
Behavioral
economics
User
Information
Science
Communication
in games
Persuasion,
value,
learning
Game
industry
Development,
game
economics
HCI
Play experience,
design
Patterns in
play behavior
Play personas
Understanding
games and players
Spatial user
behavior
Behavior
correlations with
PX/PsyPys/design
Improving
development &
testing

Metrics = Business Intelligence [BI]

BI is derived from computer-based methods for
identifying, extracting and analyzing business data
 for strategic or operational purposes
 Across market-, geographic- and temporal distance

Supports decision making (Decision Support
Systems)

Quantitative measures about any aspect of games
 Players: gameplay, customers, monetization,
 Production: team size, pipeline, milestones, markets
 Technical performance: servers, infrastructure
 Any other relevant quantitative measure (e.g.
management)

Analysis of game metrics = game analytics

[No accepted definition (working on a standard)]

Metrics are measures, e.g.:
 Average playtime per player
 Number of ”Swords of Mayhem +5” sold
Players
 Daily Active Users
 % server uptime/stability
 Avg. network latency
 Bugs reported/bugs resolved /day
 Customer support call avg. length
Performance
Process

Big data: populations not samples
 Understanding all players
 Research/development out of the lab and into the real
world

Big depth: Detailed recording of all aspects of
play
 Includes communication, navigation, cross-games ...
 Combining GUR data sources for in-depth research

Behavioral telemetry inform what players are
doing, only by inference why

Finding the right features to track is not
obvious

Managing the allure of numbers

Game data mining = data mining of game metrics

Gartner Group:
“the process of discovering meaningful new
correlations, patterns and trends by sifting through
large amounts of data stored in repositories, using
pattern recognition technologies as well as statistical
and mathematical techniques”

Common approaches in game data mining:

Description
 Characterization






Discrimination
Classification
Estimation
Prediction
Clustering
Association

Simple description of patterns in data

Accomplished using Explorative Data Analysis

Example: how rapidly does the ”warrior” class
advance through levels?

Answers many questions from designers and
producers

Drill down/across

Using a large number of known values to predict possible
future values


How many players will an MMORPG have in 3 months?
When will a F2P break the 1 million player threshold?
When will people stop playing?


One of the most widely used data mining methods in
game analytics
 Persistent world games
 MMOs
 F2P

Orders data into classes, but the class
labels are unknown (unsupervised)

Groups formed according to internal
similarity vs. across-group dissimilarity

Subjective element

Problems applying
algorithms to game metrics

Goal: Using gameplay (behavior) metrics to classify the
behavior of users

Uses:
1.
Comparing behavior with design intent
2.
Optimization of game design
3.
Debugging of playing experience
4.
Adaptation: Real-time dynamic adaptation to player type

Tomb Raider: Underworld (2008)
 AAA-level commercial title
 Data from 1.5 million users via Square Enix
 Hundreds of variables

Metrics should fit purpose
 Selected variables fitting key game mechanics
 Jumping, completion time, causes of death …

Analysis:
 Clustering algorithms (PCA, k-means)
 Self-Organizing Map (unsupervised)

Revealed a 4 distinct behaviors (94% users)

Players use the entire design space

Behaviors translated into design terms



8.68% (Veterans): Very few death events (environment).
Fast completion times. Generally perform very well in the
game.
22.12% (Solvers): Die rarely, very rarely use the help system.
Slow completion. Slow pace of play.

46.18% (Pacifists): Largest group of players, dies from
enemies. Fast completion time, minimal help requests. Good
navigation skills, not experienced with FPS-elements in TRU.

16.56% (Runners): Die often (enemies, environment), uses
the help system, very fast completion time
Towards big data:
 1st study: 1365 players
 2nd study: 30,000 players
 3rd study: 203,000 players
 4th study (in prep): 1.6 million players
 5th study (in prep): across games

From dozens to hundreds of variables

Can we predict when people stop playing?

Use: uncovering design problems; engagement

Approach
 TRU: 7 levels + prologue
 10,000 randomly selected players
 7 groups of metrics (400+ variables)
 Training data: lvl 1

Simple logistic regression best fit: 77.3% (base: 39)

Decision trees (prediction)

Use: predicting player behavior; transparent models – ideal for
communicating across stakeholders

Level-2 rewards
 Rewards > 10
▪ Level-3 playtime
▪ -> playtime > 43 minutes : 4
▪ -> playtime < 43 minutes : 7

Rewards < 10 : 2

Lvl 2 rewards and playtime lvl 3 predictors of quitting

Do paladins always have names like ”Healbot”?

Do Warlocks always have names like
”Ûberslayer?”

Are mages always called ”Gandalf”?

Are there any kind of
?

7,938,335 WOW characters (5 years logging)

Name, Race, Class, Playtime, Guild, Server
Type, Domain, etc. ...
3,803,819
unique names
(a surprising lot)
More diverse than real-world names - despite naming constrictions
Looks like naming is important to players – only unique feature you have
RP-characters most diverse (83% unique – rest ~58%)
Class
Race
”Pretty”
”Bestial”


”pretty” races named differently than ”bestial” races
Not due to differences in m/f character ratios
=
Gnomes and dwarfs named as ”bestial” races?
=


Names on US servers different from EU servers
Except for RP realms (larger overlap btw. EU/US)

What is the chance that ”Gimli” will be a dwarf?

Estimated conditional propabilities of a given
class/race/server type given a particular character
name

Class and Race best predictors, but server type and
faction also hints at naming decisions

Some names are very good predictors, others are
not -> so yes, Gimli will likely be a dwarf
1000 most common names
128,058
(not a lot, but still 100* bigger than any other study)
38 coding categories found
Some names multiple categories/hard to classify (e.g. ”Raziel”)

Regular vanilla real-world names most common (Sara, Mia,
etc.) [186]

Mythology – notably Greek [164]
 Anubis, Odin, Ares, Loki, Nemesis

Popular culture – games, cartoons, film ... [174]
 Naruto, Sakura, Tidus, Valeria, Revan, Zelda

Fantasy literature (Tolkien rules supreme) [39]
 Earendil, Sonea, Morgoth, Aragorn

A lot of names in breach of ToU

697 of 1000 names categorized
Rest: Nouns, verbs of unspecified nature

Semantic nature to categorize:

 ”Negative”: Nightmare, Sin, Fear, Requiem
 ”Positive”: Hope, Love, Pure
 ”Neutral”: Who, Moonlight, Magic, Snow

Names with negative semantic meaning 6
times more common than positive semantic
 Are gamers depressed?
 Or do ”dark” names just sound cooler?

Lots of ”why”´s unanswered:

Why are certain names more popular than others?

Why do the Mage class exhibit a greater variety of
names than other classes?

Why do some players pick names of characters from
the same game they are playing?

Need to talk to the players ...

Demand in the IDE industry
 Unique openness to research-industry
collaboration

Attractive research challenges
 Complex, mixed-methods, multi-
disciplinary, big data

Better methods and algorithms (all forms of metrics)

Correlating behavior, PX and design
 Spatial game analytics

User profiling: behavior, personality, motivations ...

Decoding and predicting behavior

Maturing development practices (from joint warehousing
to GUR)

”Guerrilla metrics”-methods


6+ games
5+ game ”types”


Same patterns?
90%+ prediction

Power law:
 When the frequency of an event varies as a power of some attribute of
that event (session length)

Does all playtime behavior follow a power law?

Blog.gameanalytics.com

andersdrachen.wordpress.com
 [slide deck available here]

IGDA GUR SIG – LinkedIn group, 350+ members

The GUR SIG Mendeley Library – mixed industry/research

GDC archives – industry SOTA

Research publications – ACM, IEEE, Springer digital libraries
+ new book on game telemetry out 2012