Transcript Game Development Telemetry

Georg Zoeller,
BioWare
Game Development
Telemetry
Georg Zoeller,
Lead Technical Designer,
BioWare Austin
About
Georg Zoeller
[email protected]
http://www.linkedin.com/in/georgzoeller
• 7 player characters with unique content and quests.
• 4 male, 3 female player appearances.
• 10 dedicated story testers.
• 9 male and 1 female.
Guess which appearances rarely got used?
Once upon a time….
…in the ‘Jade Empire’ QA department
Dragon Age: Origins
•
•
•
•
•
•
•
•
•
•
~800,000 words of dialog voiced by
144 voice actors in English alone.
180+ area layouts.
3 classes, 3 races, 2 gender options.
10 companion characters.
6 unique, hour long Origin Stories.
300+ abilities
300,000+ lines of script code.
3,000 Cutscenes, 5,500 animations, 20,000 models
New IP, World, Rules, Engine.
Full Path coverage is hard to achieve, and even harder to verify.
QA resources are finite and sometimes volatile.
Planning and managing for full path coverage is very time
intensive.
The Problem.
Telemetry n. The science and technology of automatic measurement
and transmission of data by wire, radio, or other means
from remote sources, as from space vehicles, to
receiving stations for recording and analysis.
Game Development Telemetry –
… automatic measurement and transmission of data from
game executable, build pipeline and development tools
for recording, analysis and workflow improvement!
Definition.
…not from space, yet
Customer facing
Developer facing

Understand how customers
experience the game.

Understand how developers use
tools and pipeline.

Identify and react to issues post
launch.

Identify and react to pipeline
and workflow issues .

Understand the market, learn for
future games.

Measure stability, productivity
and efficiency of development.

Already quite popular.

Is Business Intelligence.
Same idea, different goals.
2nd Generation homegrown telemetry system.
Serving Dragon Age: Origins, Mass Effect2, SWTOR
On Dragon Age: Origins alone:
•
•
•
•
1,100,000+ games tracked on 1,141 different machines.
Tracked any game of DA:O in the entire EA network.
250 GB of data gathered in ~38,000,000 datapoints
50,000+ bugs, 4597 unique users on 582 machines.
Let’s have a look…
All your data….
Introducing ‘SkyNet’
• Know who is running which build, language, executable.
• Drill down into any running game…
Chronological / spatial event visualization.
Aggregated spatial visualization (hmp)
• Raw data is great…
... if your day starts with SELECT socks, pants FROM closet;
• Engineers demand different detail than managers.
• Executive summaries are … graphs.
• Best solution we found: Drill-down
Data Visualization
Why presentation makes or breaks your system.
• Report view provides the next level of detail.
• Aggregates information for leads and intermediate managers.
• Reports are nothing more than SQL statements.
• But can be exported and subscribed to.
• Allow further drilldown…
• This view is interesting to programmers, not so much managers.
• Incoming callstacks are timestamped, sanitized and hashed.
• Tells us the rough# of unique crashes in the system
•Freshly introduced crashes file an ‘investigation task’ to
engineering QA.
• Who can retrieve the full details by drilling down…

Business Intelligence – Great for evaluation outsourcer performance
Now let’s use the telemetry data to improve other workflow...
Industry Typical Bugtracker interface
• Lots of fields
• Some mandatory
• Many freeform.
Result:
• Fields stay empty
• Random values, Typos
• Database pollution.
Smart Bugs
When bugs learned to listen…
Telemetry enabled interface.
•
•
•
•
•
•
•
•
•
Retrieves fields from the server.
No typos in database fields.
Fewer mandatory fields.
Links to session history, RunInfo, etc.
Location aware bugs.
Suggests who should get the bug.
Warns of duplicates based on text, build and location.
Vacation Tracker Aware.
Smart Categories.
But wait, there is more…
Better?.
• 95% adoption rate for new interface after 3 months.
• Significant Bug count increase (especially non QA).
• Significant Bug quality increase.
• Bugs always have platform and build #.
• Hardware information in each bug.
• Much cleaner bug database, spatial bug information.
• Spatial information == great for finding dupes.
• Increased overall QA capacity by drafting everyone to help.
• Much improved reporting…
Results.
Achievements work!
•
•
•
•
They influence player behavior through reward.
Teach about deeper aspects of the game.
Yield valuable telemetry for developers.
Inspire some gamers to go to great length...
… and many developers are gamers, so:
•
•
•
•
Yes, they work for (some) developers too.
Can be used to influence how testers play.
Yield valuable balance telemetry for design.
Inspire some developers to play the game. A lot.
Achievements
For developers?

Doubled hours of playtime
for a month after
introduction

Some people stayed
‘addicted’ and had to get
them all.

Increased bugs filed by
both QA and engineers.

Reward desirable
behavior…

Got people competing and
excited about the game.
Morale boosters.

Taught users about more
obscure game features.
Results.
•
Telemetry systems don’t mix well!
• Different goals, different needs.
• Different technical requirements
•
Start small, don’t over-engineer.
•
Be Non Invasive.
•
Reuse test hooks
•
Make it dead simple to add events.
•
Create an API and they will come…
Getting Started
Practical tips and pitfalls.
PS3 Clients
• C# Server
• Receives Packets (udp)
• Tracks clients.
360 Clients
PC Clients
SkyNet
Server
Automated
test bots.
• Mangles packets into DB.
ASP.Net WebApp
DB
Web
Frontend
SkyNet API
BioWare
Users
External
Contractors
EA Users
Tools and
Sensors
Our Tech.

UDP vs TCP?
◦ Lightweight, non blocking.
◦ We use one way communication.
◦ Telemetry data may get lost.
Server software

◦ C# is RAD.
◦ Runtime-Compiled ‘packet handlers’
◦ Dynamically routes events into tables by type.

Protocol
◦ Character delimited string.
◦ Only defined in packet handlers.
◦ No support for fragmentation.
More Tech Talk.

Game Client / Server

Toolset
◦
◦
◦
◦
Callstacks, Warnings, etc.
Game Events, Movement, etc.
Savegames, Screenshots
Performance Timings
◦ Startup time, Usage patterns.
◦ Callstacks, Warnings, etc.




Wikis / Sharepoint.
Bugtracker
Landesk
IP enabled coffee maker.

Twitter
◦ Goes well with timeline of PR
Useful data sources.
Usage patterns.
One database that indexes information from any available
datasource in the company.
•
•
Makes it searchable and viewable.
And retains historical information so we can understand,
optimize and learn.
•
Push information back into the game (e.g. warn users when
playing a broken build).
•
•
Something like this…
Endgame
Where do we go from here?
Questions, Comments, Feedback?
If you have further questions -or end up implementing your
own development telemetry system, I’d like to hear from
you – especially if you have ideas for additional data
sources.
Georg Zoeller
[email protected]
Thank you.
For attending this presentation.