Transcript Game Audio Presentat..

Audio for Games
Scott Selfon
Xbox Advanced Technology Group
Microsoft Confidential
Overview
The roles of audio in games
Audio challenges on a game console
Score, Sound Effects, and Dialog
Challenges
Typical Solutions
Content creation / implementation process
Audio integration and end game
Case studies on Xbox
Leading edge / future of game audio
Q&A
Microsoft Confidential
Game Audio Has Come a Long
Way…
Fidelity
Tinny mono speaker  5.1 home theater systems
1 channel/4 bits  256 channels, 24 bits
FM oscillators  48 kHz wavetable synthesis
Audio processing power
Off-line effects processing  real-time DSP effects
4-channel mixing  Realtime Dolby Digital encode
Realtime 3D positioning
Production pipeline
Low level chip programming  Separate sound
designer/composer and audio programmer
Microsoft Confidential
The Roles of Audio in Games
Audio is 50% of the experience
Support (improve?) the visuals
Provide information about what we don’t
see
Provide emotional support for gameplay
Rewards/encouragement
The best audio implementations are rarely
noticed by the gamer
1995 MIT study
Game reviews and audio
Microsoft Confidential
Audio Challenges on a Game
Console: Aesthetic
Competing with other ‘living room’
experiences
But much less linear
Events may be skipped, repeat, play in
unpredictable order
Players expect same kind of realism as they
get from non-interactive media
Microsoft Confidential
Audio Challenges on a Game
Console: Technical
Finite Resources
Storage space (not as much of an issue
currently)
Memory (actual playing sound data, audio
library size, etc.)
Processing power (realtime effects, mixing,
etc.)
Interactivity
Avoiding repetition is perhaps most important
Secondary goal: compelling non-linear audio
Need solutions that don’t require a lot of
development time
Microsoft Confidential
Audio Challenges on a Game
Console: Organizational
“Post-Production” mentality towards audio –
scheduled late
Often appropriate in film, television
Often viewed in budgeting as expense rather
than investment
Tradition & expectations for game audio - hasn’t been
always been of highest quality (see previous slide)
No dedicated audio programming resource
Often up to composer/sound designer to understand and
determine what’s possible and feasible
Cross-platform audio solutions
Simplest-to-implement programmatic solutions used
Microsoft Confidential
Audio Role Comparison
Film/TV/Radio
Games
Sound Designer
Composer/arranger(s)
Orchestrator(s)
Recording Engineer(s)
Mixing/Mastering
Engineer(s)
Music Supervisor
Music Editor
SFX Editor
ADR/Dialog Editor
Foley Artist(s)
Casting (for voices)
Sound Designer/Composer
(maybe 2 separate people)
Audio programmer (part
time)
Microsoft Confidential
Hiring a Content Creator
Generally ‘buy-out’ (work for hire)
Composer paid straight fee (per platform?)
Developer/publisher generally owns all rights
to music
May or may not include cost of recording,
musicians, mixing, etc.
May negotiate for publishing
Per copy sold royalty (rare)
Paid for TV/radio spots
Royalty for thematic use in sequels, spinoffs
Opportunity for game soundtrack CD
Microsoft Confidential
Budgeting for Audio
Musical underscore
Live, synth, or mix? And how much?
How does it start and stop? (linear, looping,
branching)
Licensing existing music
Sound Effects / Ambience
Live recordings or sample libraries?
Creation of new modeled sounds
Sub-contracting
Often essential for large projects
Microsoft Confidential
Laying out an Attack Plan
Look at all ways system can make noise
Work with technical document to
determine what’s preferable/do-able
E.g. MIDI or streamed (CD/DVD/HD) music
Real-time effects (reverb, filtering, etc.)
What other game resources are being used
Maximize audio capabilities while
minimizing impact on rest of game
Often different for different parts of game
Microsoft Confidential
Tools, Tools, Tools…
Evaluating for technical and aesthetic criteria
Make authoring and programming pipelines relatively
independent
Maintain creative flexibility
Linear audio
Authoring: Same as for other media
Programming: Cross-platform and platform specific
options
Non-linear audio = generally proprietary
PC: DirectMusic or ‘in-house’ engines
Consoles: Platform-specific or ‘in-house’ solutions
Microsoft Confidential
Underscore Challenges
Film
~90 minutes, often <60 minutes of score
Linear, unchanging storyline
Game
10+ hours of gameplay
Gameplay events may be quite variable (in
order, duration, etc.)
Don’t need “music” everywhere (but at least
ambience)
Score: Often linear tracks several minutes long
(may loop)
Microsoft Confidential
Typical Underscore Solutions
for Consoles
Create several long linear tracks
May loop
May be played ‘soundtrack’ style (no interaction with
gameplay)
Save purely musical score for key one-time game
points
Use ambience instead to set mood
Avoids repetition of recognizable thematic material
Advanced: Non-linear audio
Alternate versions with looping and branching
Various intensity levels of music that crossfade based on
game state
Microsoft Confidential
Licensed Music
Not appropriate for all games
Often difficult to make interactive/dynamic
Disconnects score from moment-to-moment
gameplay
Can be effective in some situations
Name recognition/cross-marketing potential
Can set titles up for user soundtrack support
Typical uses
Existing brands/licenses (esp. movies and TV)
Heavily used by racing games
Microsoft Confidential
Composing a Game Score
Spotting session
Where can music support action?
Compose cues for linear sections of game
(for instance, cutscenes)
Need final cut for accurate timings (hah hah)
Record and mix
Pre-rendered: Synth, live players, mixture
Real-time rendered: MIDI, individual pads, etc.
Implement playback in game
Triggers to start, stop, transition, etc.
Edit to match ‘final’ animations
Microsoft Confidential
Sound Effects Challenges
No pre-existing soundtrack to work with
Same sound event may be heard
hundreds of times
Variation
Environmental interaction
Tying FX to game physics and AI
Sheer number of sound elements to create
Combinations and permutations
Everything that can make a noise should
Microsoft Confidential
Typical Sound Effects Solutions
Variation
Implement basic pitch randomization
Several versions of each wave
Break a sound into components and record
versions of each of those
Tying effects to game physics / AI
Implementation varies per-title
Often vast majority of dev time spent here
Advanced: Dynamic processed sounds
Environmental effects (reverb)
Filtering (force of impacts, radioizing, etc.)
Microsoft Confidential
Creating Ambience and Sound
Effects
Spotting session (if game playable), or
look at artwork (environments, objects)
List everything that can make noise, prioritize
Object interaction matrix
Often start with organic sounds
Sound effects library ($$$, can be generic)
Create own samples (free, unique)
Sound processing (pre-rendered or
dynamic)
Reverb, filtering, chorus, effects
Microsoft Confidential
Dialog Challenges
Casting – producer may want involvement
Directing / Coaching
Stitching (first | and | 10 | from the | 25)
Dialog repetition is easily recognized
Localization
Re-recording voice talent
Engine integration
Grammatical differences
Recreating pre-rendered processing
Microsoft Confidential
Typical Dialog Solutions
Triggers for dialog (just like music)
Variation for common lines (often too few)
Stitching – coach the talent
Localization
Solutions vary per-title
Very careful asset management
Sometimes source assets can just be swapped
Advanced: Sophisticated dialog engine
Variation tracking to avoid repetition
People don’t always have to say anything
Microsoft Confidential
Prepping for Dialog
Content creator works with producer,
game designer, and/or writer to:
List and describe characters
Write each character’s scripts
Cast person for each character
Real voice talent makes a difference
Many actors can do multiple roles
Microsoft Confidential
Dialog Production
Record all dialog
Encourage improvisation
Coaching for creating convincing stitchable
dialog
Cut up dialog into separate files
Clean up for stitching – sound designer for
sports title can spend majority of time here
Post-recording processing
Document everything for localization
Microsoft Confidential
Squeezing it all into 5 bytes
Best to determine resources up front
Memory and CPU available
Disk space and bandwidth available
Inevitably, not enough room
Asset swapping (programming time)
Compression
Use of streaming where possible
Cut assets
Microsoft Confidential
And the Handoff…
Source assets given to game
Typically high fidelity + compressed versions delivered
Freelance composer/sound designer: often end of work
Programmer integrates audio assets into title
Often tasked with mixing (either with or without content
creator feedback)
Implement runtime sound processing (reverb, effects,
filtering, etc.) unless engine natively supports
Testers confirm sounds are in place
Asset/trigger lists, scripts, and/or manual triggers
Variation testing?
Microsoft Confidential
After the Fact: More
Challenges
Archive, archive, archive
Create a recording of typical gameplay for
reference
Disaster always strikes at the 11th hour
Keep an organized collection of any sounds
created (spreadsheet, database)
Localization
Voice talent recasting, re-recording
Recreate processing (unless real-time used)
Releasing a CD
Microsoft Confidential
Audio Case Study:
Project Gotham Racing
Stereo ambience, with some 3D positioned
foreground ambient elements
Player’s car
Stereo engine ambience
Engine sound varies with RPM, velocity, physics, etc.
Tire sound varies with surface (sidewalk, stairs, street,
etc.)
3D positioning on other cars
Engines (varies with RPM, velocity, physics)
Car horns
Cornering, tire squeals, collisions, etc. (all with some
variation)
Dynamic sounds
Tire squeals (randomized between several versions and
pitches)
Collisions (depend on speed, material involved, etc.)
Microsoft Confidential
Project Gotham Racing: Music
Game disc-supplied licensed songs (WMA)
User Soundtrack support
Allows for personalized driving music mixes
Music uniquely integrated into game
Use of ‘car radio’ or ‘CD’ playback
‘Radio’: Location-specific DJ interstitials
Microsoft Confidential
Audio Case Study: Splinter Cell
Detailed ambiences, foley, sound effects
Audio as a gameplay element (stealth)
Stealth: Noise tied to game AI
Footsteps, interaction with objects, weapons
Audio as an informational tool
Interactive musical score
Tells player when enemies alerted
Use of musically appropriate boundaries for
transitions
Microsoft Confidential
Audio Case Study: Halo
4,311 triggerable events (soundtags)
11,629 source waves (soundfiles)
2.5 GB uncompressed audio
22 kHz mono and 44 kHz stereo
80 minutes of source music
20+ hours of gameplay
12 voice talents, 7,735 lines of dialog
…all in 2-MB memory footprint
Microsoft Confidential
Halo: Audio Tools Used
Pre-rendered linear music/FX = traditional
standard audio post production tools
Pro Tools, Studio Vision, Peak, Digital
Performer
Dynamic audio = custom tool/engine,
randomization/permutation for each event
Pick one or more waves to play
Random variation in pitch
Weighting for how often a sound will be used
Microsoft Confidential
Halo Audio Engine: Dialog
Simulation of actual speech patterns
Close connection to game AI
Large number of events a character can
respond to, multiple phrases per action
Characters don’t always speak
Tracking speech
Multiple voice talents for multiple characters
Character only says one thing at a time
Positioning (3D versus locked-to-speaker)
Repetition avoidance
Microsoft Confidential
Halo Audio Engine: Sound Effects
Variations in all sound effects
Weighting (frequency for multiple versions of sound)
Randomized pitch control
Variations in ambience
Various loops of different lengths
Layered 3D-positioned detail sounds (birds, crickets,
machinery hum, etc.)
“Cascading” sound effects
If many instances of an effect are playing, play a single
more complex sound
Response to game environment
Specific conditions/events/game physics
Environment interaction (reverb)
3D spatial positioning versus static
Microsoft Confidential
Halo Audio Engine: Music
Typically linear or looping
If looped, alternate versions with varying
lengths
Dynamic execution (“spotted” with
programmer)
Varying transition types based on situation
Branching
Crossfades
Game event tags
Appropriate absence of music
Silence is golden - not scored wall to wall
Timeouts
Microsoft Confidential
Halo 5.1 Usage
Sound effects typically 3D-positioned (4 channels)
Some ambience 3D-positioned as well
“Quad” music and ambience
Doubled stereo (fronts 100% / rears 50%)
Crossfades based on location
Center channel
Cortana (“first person” dialog)
Radio communication
LFE channel
Used to emphasize more massive sounds
Heartbeat (low health)
Explosions
Microsoft Confidential
Halo Environmental Reverb
Usage
Reverb settings changed based on
environment
Indoors versus outside
Confined spaces versus large spaces
Caves versus hallways
Sound interactions with reverb
Especially 3D-positioned sound effects
Some ambient elements also
Microsoft Confidential
Halo Implementation
(1)
2-MB in-memory footprint
22-kHz mono ADPCM for 3D sounds
44-kHz stereo ADPCM for music + ambience
ADPCM = ~3.5:1 compression
Use of Xbox hard disk
Copy all sound data to drive at start of level
(faster to read than DVD)
Pull small chunks into memory rather than
entire banks (pseudo-”streaming”)
Drive persistence = faster load next time
Microsoft Confidential
Halo Implementation
(2)
Sound during load screens
Small looped sample
Multiple instances played with pitch shifting
and filtering as load progresses
“Dynamic” – speed and actual sound vary with
read time
Microsoft Confidential
Leading Edge Audio Challenges
Voice integration (esp. for online titles)
Chat and gameplay implications
Manipulation (masking, radioizing, etc.)
Voice recognition
Command and control
Further multichannel audio usage
Fuller 5.1 implementations
Dynamic audio mixing
Physical modeling
Still too expensive (CPU, programming effort)
for most games
Microsoft Confidential
Wrap Up:
The State of Game Audio
Lots of independent composers/sound designers
Many coming from outside game industry
G.A.N.G. (www.audiogang.com)
Still typical for developers to reinvent audio
engine with each title (newer tools try to solve)
Xbox efforts
Xbox Registered Developer and Registered Content
Creator Programs
Tools and dedicated support for audio content creators
and programmers
Game Designer/Producer education
Microsoft Confidential
Q&A
[email protected]
Resources
G.A.N.G. (www.audiogang.com)
Game Audio Pro e-mail list (Yahoo)
DirectMusic (www.microsoft.com/directx)
2003 Game Developer Conference lectures
Microsoft Confidential