Transcript Slide 1

The Asset pipeline for Just Cause 2: Lessons Learned
Mathias Westerdahl - System Architect, Tools
Overview
2
• Background
• Just Cause 1
• Just Cause 2
• Current pipeline
•Build system and compilers
•Central code
• Take aways
•Q&A
The Asset pipeline for Just Cause 2: Lessons Learned
What is Just Cause?
• Large open world
• Sand box game play
• Dynamic game play
• No Levels, continuous streaming
• Multiplatform
• Very flexible editing and testing of content
• Needs a fast and flexible pipeline
The Asset pipeline for Just Cause 2: Lessons Learned
3
JustEdit
4
• Our main editor is called JustEdit
• Edits the terrain, missions, locations, roads, etc.
• Each mission/location contains entities
• An entity contains game objects or other entities
• This highly flexible system led to very many assets
to export (~1500 locations and ~4000 entities)
• Runs the game
• No turnaround times for testing
The Asset pipeline for Just Cause 2: Lessons Learned
Workflow Overview for Just Cause 1
JustEdit
5
Maya
Terrain
Locations
Missions
Particles
Models
Physics
Animations
Skeletons
Perforce
Physics
Game
The Asset pipeline for Just Cause 2: Lessons Learned
Damage
Editor
Workflow Overview for Just Cause 1
JustEdit
Many manual steps
Game ready data in
version control
6
Maya
Terrain
Locations
Missions
Particles
Game ready data in
version control
Models
Physics
Animations
Skeletons
Perforce
Physics
Damage
Editor
Editing of game
ready data!!!
Game
The Asset pipeline for Just Cause 2: Lessons Learned
Lessons learned from Just Cause 1
• Remove as many manual steps as possible
• Put export steps in separate tools (i.e. compilers)
•
Decrease code complexity
•
Increase robustness
•
Easier to read and debug
• Introduce intermediate data formats
• Tools code broke easily
The Asset pipeline for Just Cause 2: Lessons Learned
7
Actions Taken after Just Cause 1
• We decided on creating our own content build system
• Compilers in C++, derived from previous export steps
•
Took existing code and put into use directly
• The Project Compiler was the heart of the system
•
Triggered all other compilers
•
Dependency checking was implemented
separately by each compiler (time stamps)
• Users can right click on any file and compile that
dependency tree
•
The editor and other tools use this functionality
The Asset pipeline for Just Cause 2: Lessons Learned
8
Actions Taken after Just Cause 1
• Auto Builders were setup
• Code
•
All compilers, exporters, plug-ins, editor
• Content
•
Full nightly builds for all platforms
•
Builds ISO’s
• Cruise Control + NAnt was used to create the auto builders
•
CCTray is a system tray icon tool
•
Alerts users when a build fails
The Asset pipeline for Just Cause 2: Lessons Learned
9
Just Cause 2 – New Tools
• A lot of new tools were ordered by the content creators
• Detailed design docs were written
• Each tool was assigned a “customer”
•
The programmers and the customer kept close communication
•
The customer was responsible to test the tool and approve it
• Tools as plug-ins to JustEdit
• WTL was used for GUI in editors/plug-ins
The Asset pipeline for Just Cause 2: Lessons Learned
10
Workflow Overview for Just Cause 2
Plug-ins
JustEdit
Vehicles
Characters
Cutscenes
Maya
Terrain
Locations
Missions
Particles
11
MotionBuilder
Models
Physics
Skeleto
ns
Animations
Cutscenes
Constraints
Perforce
Sync
Auto Builder
Compilers
Compile
The Asset pipeline for Just Cause 2: Lessons Learned
Game
Lessons Learned from Just Cause 2 – New Tools
• 10 new tools at start of project!!
•
Code not up in the 90%-100% complete where it should be
• Flaws in workflows
•
Not all tools were used in the extent that was intended
•
Content that was easy to edit was not very suitable in
game with respect to performance and memory
• Design docs were accepted “as-is”
•
They were essentially a wish list from content creators
The Asset pipeline for Just Cause 2: Lessons Learned
12
Lessons Learned from Just Cause 2 – New Tools
• Customers
•
They were not part of the dev-team, communication suffered
•
Often planned full time on other tasks
•
Unclear responsibilities
• Plug-ins
•
Unstable C++ interfaces
•
Should have gone with network communications
•
Too complex
• GUI: WTL and C++
•
WTL is too bare bone and C++ slows down iterations
The Asset pipeline for Just Cause 2: Lessons Learned
13
Lessons Learned from Just Cause 2 – Compilers
• Compilers in C++
•
Using C++ for scripting isn’t always very convenient
•
Long build times, making it awkward to tweak
•
Some compilers included all game code
• Very platform specific
•
Win32 heavy with DirectX amongst other things
•
Couldn’t run on auto builders with no desktop
The Asset pipeline for Just Cause 2: Lessons Learned
14
Lessons Learned from Just Cause 2 – Compilers
• Programmers implemented things slightly differently
•
Causing anomalies in overall behavior
•
Breaking dependency checking
•
Hard coded parameters in the compilers
• File formats weren’t stable enough
•
If something went wrong, compiler probably crashed
•
Or worse, the broken data made it into the game
The Asset pipeline for Just Cause 2: Lessons Learned
15
Lessons Learned from Just Cause 2 – Compilers
• Reluctance to work with compilers
•
The pipeline was considered somewhat magic
• Lack of documentation
• The overall flow wasn’t easily overviewed
•
Debugging broken data often done in game code
• Programmers already had all code & data set up
•
This sometimes led to runtime fixes being done rather than
fixing the bugs in the compilers
The Asset pipeline for Just Cause 2: Lessons Learned
16
Lessons Learned from Just Cause 2 – Compilers
• No central code
•
Not using central code meant problems when integrating
• Fixes went missing, you had to find them again
•
Built compilers were version controlled in Perforce
• Not automatically rebuilt and used when needed
• You could still use old buggy code
The Asset pipeline for Just Cause 2: Lessons Learned
17
Lessons Learned from Just Cause 2 – Compilers
• Data formats
•
A lot of different file types (~30)
•
Many of the formats were Xml based
• Mapping “property name” to “value”
•
Formats were more optimal for the editor than for the game
•
Read/write errors weren’t detected
• No version numbers
• No proper error messages could be logged
•
Lot of things to improve!
The Asset pipeline for Just Cause 2: Lessons Learned
18
Lessons Learned from Just Cause 2 – Compilers
• Content build system
•
A bit naïve solution
• Slow dependency checking
• Incomplete dependency tree
• Time stamps are not good enough
•
Due to bugs & anomalies in the behavior, users didn’t trust it to 100%
• “It went wrong! Did I do it in the wrong order?”
• “I’d better resort to my old ways without asking anyone…”
•
No trust meant full clean and full rebuild
The Asset pipeline for Just Cause 2: Lessons Learned
19
Statistics from Just Cause 2
• 3 platforms: Win32, PS3, Xenon
• ~110000 source assets in version control
• ~80000 of those files pass through our compiler pipeline
• Source data amount to ~13GB
• Compiled data amount to
•
~7GB uncompressed
•
~4GB compressed
• A full rebuild took about 4h45m for all 3 platforms
• Building all tools code took ~1h
The Asset pipeline for Just Cause 2: Lessons Learned
20
Current pipeline
21
• What was done to address the retrospective from JC2?
• Fresh goals
•
Rebuild the trust in the pipeline
• Incremental builds, no magic workflows, easy to configure
• 100% accuracy using the dependency checking
•
Simplify the development
• Move common code to a central repository
• Decrease turn around times when creating tools
The Asset pipeline for Just Cause 2: Lessons Learned
Current pipeline
22
• Big decisions
•
We needed a build/deployment system for central code
•
The content build system also needed a major push
•
Needed a simple to use and yet powerful generalized data format
•
Development had to be done alongside JC2
•
Needed to commit to the task to actually get anywhere soon
The Asset pipeline for Just Cause 2: Lessons Learned
Current pipeline - Needy
• Dependency resolver and deployment system
• Written in house using python, very light weight
23
Dependencies.txt
Python-2.5.150.1013.1;buildtool
Needy-4.4.0.204986;buildtool
• Handles “packages”
Buildsystem-6.1.*;buildtool
•
Built libraries (.h + .lib/.so)
•
Built executables
•
Bundles of python scripts (or whatever you want)
havok-55x.277059
•
3rd party libraries / executables
Product.txt
Platform-7.*;buildtool
Debug-8.*;buildtool
• Deploys packages to a central repository
• Fetches packages to local system
Adf-8.*
PRODUCT=animcompiler
VERSION=1.0.0
PLATFORMS="Win32 Linux"
CONFIGURATIONS="Debug Release"
CATEGORY=compilers
The Asset pipeline for Just Cause 2: Lessons Learned
Current pipeline - Needy
• Big change in the coding process
• Code base evolves towards smaller libraries
• Really fast compile times
• Helps keeping track of version conflicts
• Allows for very rapid development and testing
The Asset pipeline for Just Cause 2: Lessons Learned
24
Current pipeline – WAF
•
Light weight build system for code and data
•
Open source (BSD), source is ~80kb
•
Dependency checking
•
Multi core support
•
Easily to maintain and extend
•
Replaces our in house Project Compiler system
•
Python
25
A small snippet from a task definition using waf:
cls = Task.simple_task_type(‘.anim’, ‘${PYTHONC} ${ANIMC} –i ${SRC} –o ${TGT} –p ${PLATFORM}’)
cls.ext_in = (‘.anim’, )
cls.ext_out = (‘.ban’, )
Task.TaskBase.classes[‘.anim’].scan = FindDependenciesFromAnimation
The Asset pipeline for Just Cause 2: Lessons Learned
Current pipeline - ADF
•
Avalanche Data Format
•
Serialization framework
• C/C++, Python support
• Xml <-> Binary support
•
Hash based versioning for binary files
•
Error handling
•
The ability to access data both from Python and C/C++
enables seamless cross language interchange
•
Cross platform support
• E.g. pass data from tools to/from consoles
The Asset pipeline for Just Cause 2: Lessons Learned
26
Current pipeline - ADF
Type definitions:
TM2VertexData = DefStruct( "TM2VertexData",
( "PositionIndex", Array( uint32 ) ),
( "TangentSpaceIndex", Array( uint32 ) ),
( "ColorIndex", Array( uint32 ) ),
( "UVIndices", Array( uint32 ) ) )
TM2Model = DefStruct( "TM2Model",
( "Name", String ),
( "Triangles", Array( uint32 ) ),
( "Vertices", TM2VertexData ),
( "MorphTargets", Array( TM2MorphTarget ) ),
( "Positions", TM2PositionData ),
( "TangentSpaces", Array( float ) ) )
The Asset pipeline for Just Cause 2: Lessons Learned
27
Current pipeline - ADF
28
Read / Write
from Adf import Context
ctx = Context()
ctx.ReadTypeLibrary('mytype.adf')
objs = ctx.ReadInstancesFromXml(‘infile.xml')
objs[0].Name = 'newvalue'
ctx.WriteInstancesToFile( ‘outfile.bin', objs, BIG_ENDIAN )
mytype.py
AdfPack.exe
mytype.h
Infile.xml
mytype.adf
compiler.py
Outfile.bin
The Asset pipeline for Just Cause 2: Lessons Learned
Current pipeline - Compilers
•
C++
• Shared libraries, loaded from Python
• Removed dependencies on old game code
• Replaced in-house code with standard code
• fopen() rather than CFile()
• Decrease complexity, increase portability
•
Scripting and logic in python
• Data processing in Python or C++
• E.g. read/write/process Havok files using C++
The Asset pipeline for Just Cause 2: Lessons Learned
29
Current pipeline - Compilers
•
Python
• Extreme turn around times
• Vast collection of built in modules
• We use these a lot:
• optparse, ctypes, md5, numpy, cStringIO…
• Many of the modules have the backend implemented in C++
•
Most compilers were completely rewritten
• Cut a lot of dependencies
• Code size down to a 1/10th
The Asset pipeline for Just Cause 2: Lessons Learned
30
Current pipeline – Dependency Checking
•
Waf handles the mechanics
•
Scanners find the dependencies
• Waf caches the result
•
MD5 checksums
• Command lines arguments
• Source/Dependency file contents
•
A new compiler path will trigger a compilation
• Very handy when updating packages with Needy
The Asset pipeline for Just Cause 2: Lessons Learned
31
Current pipeline – Dependency Checking
•
A small example of scanning a file for other file references
# Reading a text file containing file references
def FindDependencies(path):
f = open(path, "rb" )
lines = f.readlines()
f.close()
return [ x.strip() for x in lines ]
The Asset pipeline for Just Cause 2: Lessons Learned
32
Current pipeline
33
• Platform independence
•
Possible to run on win32, linux, ps3, xenon etc.
•
Helps keeping the code clean
• Running on linux
•
We wanted to run on linux for the file system performance
•
Run win32 exe’s through WINE
• Wine is a Win32 implementation in linux. It’s not an emulator.
• Intermediate solution before all code was rewritten
The Asset pipeline for Just Cause 2: Lessons Learned
Current pipeline
34
• Incremental auto builder
•
Syncs from version control
•
Only builds the things necessary
• Full auto builder
•
Compares incremental build with full build
• Finds dependency problems
• Find bugs in data formats
The Asset pipeline for Just Cause 2: Lessons Learned
Current pipeline
35
• Change of mindset
•
Keep it simple!
•
Each component should do one thing only
• …and do it well!
The Asset pipeline for Just Cause 2: Lessons Learned
Current pipeline
36
• To sum things up so far
•
We rewrote ~20 compilers (out of ~30) in 6 man weeks
• Using old code as blue print
•
Remaining compilers took longer due to more complex work flows
•
Code size decreased to 1/10th
•
Performance increased in all cases
• 10% to 1000%, 100% was quite common
•
Possible to use compilers in the previous pipeline with little effort
• Early benefits of the new pipeline
The Asset pipeline for Just Cause 2: Lessons Learned
Future improvements
• Reloading of assets in run time (in progress)
•
Virtual File System
• Can load data from hard drive, archives or web server
• Distributed builds
•
Looking into using AFS, a distributed file system
• ssh builder01 modelcompiler.exe -i mymesh1.tm …
• ssh builder02 modelcompiler.exe -i mymesh2.tm …
The Asset pipeline for Just Cause 2: Lessons Learned
37
Take Aways
38
• A few thing we’ve learned so far
• Some are quite trivial but also easy to overlook
The Asset pipeline for Just Cause 2: Lessons Learned
Take Aways
39
• Hardware
•
Defrag the hard drives 
•
Try using other file systems, ext3, ext4, afs.
• Our pipeline is a lot faster on linux than on win32
•
Two separate discs that are designated source/target
•
Ram discs/SSD are getting cheaper
The Asset pipeline for Just Cause 2: Lessons Learned
Take Aways
40
• How to implement the compilers
•
No assumptions on working directory or any other variables
•
Keep the compiler data driven, it’s quite easy
• Project specific options could be read from a file or command line
• Create a simple plugin system
• Extremely easy in python
The Asset pipeline for Just Cause 2: Lessons Learned
Take Aways
41
• Example of loading python files. I.e. creating a plug-in system:
for root, dirs, files in os.walk(‘c:/myapp/plugins’):
if not root in sys.path:
sys.path.append(root)
for filename in files:
if not filename.endswith('.py'):
continue
# We do an import() instead of an execfile() so that
# the loaded code can act as a regular python file
module = __import__(filename.replace('.py', '') )
The Asset pipeline for Just Cause 2: Lessons Learned
Take Aways
42
• How to implement the compilers
•
Write portable code
• E.g. remove DirectX, Win32 API
• ‘-e’ for endianess rather than ‘-p’ for platform
• Beware of case sensitiveness from the start
• Also, beware of ‘\\’ versus ‘/’
The Asset pipeline for Just Cause 2: Lessons Learned
Take Aways
43
• How to organize the data
•
Keep source data separate from compiled data
•
When creating new file formats
• Avoid doing it “by hand”. Create a generalized file type.
• Include a version number
• Store file dependencies easily accessible
• Write the game ready format with efficiency in mind
• Memory, performance
The Asset pipeline for Just Cause 2: Lessons Learned
Take Aways
44
• How to optimize the compilers
•
Write compilers with parallelism in mind
• No “global variables” such as reused temp files on disc
•
Disc cache thrashing
• It comes down to the access patterns
• Do as few reads/writes as possible
• Use tools like Process Explorer or vmstat
The Asset pipeline for Just Cause 2: Lessons Learned
Take Aways
45
• How to optimize the compilers
•
Do reads and writes in memory
•
Lowers the disc usage
•
E.g allocate large buffer and write into it
•
Avoid temp files altogether
import cStringIO
memfile = cStringIO.StringIO()
MultipleWritesToAFile( mydata, memfile)
f = open(options.output, "wb" )
f.write( memfile.getvalue() )
f.close()
memfile.close()
The Asset pipeline for Just Cause 2: Lessons Learned
Take Aways
46
• Dare to do stuff!
•
If the compiler code is unreadable/inefficient
• Will it be faster/more productive to rewrite it?
•
Rewriting will probably introduce new bugs
•
But you will also…
• Probably throw away many old bugs as well
• Increase your understanding of the work flow
• Get a chance to improve design/performance more directly
The Asset pipeline for Just Cause 2: Lessons Learned
Take Aways
47
• Dare to do stuff!
•
Rewriting doesn’t mean you throw the code away
• You will use the code as a blue print of what, or what not, to do
• It’s easy to rewrite something that does the same thing, only better!
The Asset pipeline for Just Cause 2: Lessons Learned
Take Aways
48
• Separate code
•
Use one code base to read/write data in compilers
•
Use another code base for loading the data in game
•
Keeps down code dependencies
•
Keeps the compilers small and simple
•
Less code to maintain in the compiler
• Decreased code size down to 1/10th
•
Less code to compile, faster turnaround times
•
Suitable for file formats that rarely change
The Asset pipeline for Just Cause 2: Lessons Learned
Take Aways
49
• Keep the build working!
•
Only replace old code with new working code
•
Down time is both costly and stressful
• Keep it simple!
The Asset pipeline for Just Cause 2: Lessons Learned
Links
50
• Python – http://www.python.org/
• waf – http://code.google.com/p/waf
• wine – http://winehq.org
• jqplot – http://www.jqplot.com/
• protobuf – http://code.google.com/p/protobuf/
• Download slides and source code on our site!
The Asset pipeline for Just Cause 2: Lessons Learned
Q&A
51
www.avalanchestudios.se/presentations
[email protected]
Download slides and source code on our site!
The Asset pipeline for Just Cause 2: Lessons Learned
52
The Asset pipeline for Just Cause 2: Lessons Learned