Transcript Q - Web3D 2011

Remote-Rendering related
research at Hasselt University (B)
dr. Peter Quax
Expertise Center for Digital Media
Multimedia and Communication Technology group
Remote Rendering
for mobile devices
• Results of the national project
‘architectures for mobile
community content creation’
– ’03/’04
• Create a virtual city guide for use on PDAs
and PC
– 3D recognizable
visualization
– User-generated multimedia
content
– Community features
– Communication
Remote Rendering
for mobile devices
• Issues
– Only a single device capable of
rendering accelerated 3D graphics
(Dell Axim x51v)
– Scenes could not be rendered in
software (large textures)
– Deployment on a 3G network
(WiFi did not cover the entire city)
• Axim X51v included Intel
Graphics accelerator
– No 3D engines available, create
custom render engine based on
OpenGL ES 1.0
– Local rendering provided very
good user experience (formal
testing), but devices too expensive
to deploy on massive scale
Remote Rendering
for mobile devices
• Solution for low-end devices
– Create a custom remote rendering
system to enable hybrid deployment
(local and remote rendering)
– Rendering is separated from the
application logic for scalability
purposes (remote render is just
another user to the system)
– Use off-the-shelf PC hardware to
render 25 viewports at once (in
QCIF), segment the individual views
and encode them in parallel
– Limit user freedom (like shooter on
rails) to guarantee Quality of
Experience
– H.263 QCIF transmission at 15fps on
early UMTS networks was feasible
– Conclusion : works fine, but rapidly
outdated by local rendering
solutions on mobile devices
Remote Rendering and
Interactivity Requirements
• One of the most challenging contexts for interactive applications : multiplayer games
• Number of studies available (see NetGames series of workshops), show
that most demanding genre is First Person Shooter
– Most popular genre among hard-core gamers
– High number of interactions per time frame
• Experiment objective : determine whether there is an indication for a
boundary above which players become aware of network delay
– Objective (in terms of score)
– Subjective (in terms of experience)
• Experiment in context of access network
delay, but results also apply to
remote rendering of these games
Remote Rendering and
Interactivity Requirements
• Do players perform worse when their network connection to
the server is delayed (vs no impairment) ?
– Objective measurement
35
25
mean score
15
Yes !
5
-5
-15
-25
0
1
2
3
4
5
6
7
8
9
player
impaired
overall
unimpaired
10
11
12
13
14
Remote Rendering and
Interactivity Requirements
• Do players notice when their connection to the game server
(simulating the access network) is delayed ?
– Subjective measurement
10
9
network rating
8
Yes !
7
6
5
4
3
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
scenario number
impaired players
unimpaired players
all players
16
17
18
19
20
Remote Rendering and
Interactivity Requirements
• Is there a threshold that can be
defined ?
45
40
% occurance as best rated
– Below 60 ms, players are less likely to
feel an impact (subjective and
objective)
– Subjective and objective
measurements match
– If the access network has a delay of
60ms, there is little hope for a remote
rendering solution to be viable under
these conditions (will only get worse)
– Conclusion : look at other use cases
(in gaming context) for Remote
Rendering besides First-person
shooter games
35
30
25
20
15
10
5
0
20
40
60
delay (ms)
80
100
Remote Rendering for
Massive Multiplayer Games
• Architecture for Large-scale Virtual Interactive
Communities (ALVIC)
– Design a scalable client/server based solution for
MMOGs (main focus on MMORPG genre)
– Emphasis on practical issues
• Deployment (firewall issues, delay experience)
• Economical considerations (cost of deployment and
maintenance)
• Manageability (moderation, security, cheating)
– Suitable for deployment on cloud platforms
– As generically applicable as possible
Remote Rendering for
Massive Multiplayer Games
• ALVIC-NG uses a layered
approach
– Ensure scalability of each layer
– Additional entities compared to
traditional setups : proxy layer
– Goal
• Reduce the number of connections
between clients and server(s)
• Perform caching where possible
• Be extendable towards Remote
Rendering for low-end devices
(media players, dedicated set top
boxes,…)
Remote Rendering for
Massive Multiplayer Games
• What is unique about the
proposed architecture ?
– No sharding/instancing
(like WoW)
– Dynamic scalability
• Servers assignment to spatial
subdivision scheme is not fixed
(like in Second Life)
• Can scale from single to hundreds of servers without service
interruption
– Responsibilities are shared between clients (mostly
static) and servers (everything dynamic)
Remote Rendering for
Massive Multiplayer Games
• Applicability of Remote Rendering
– Client is relatively ‘dumb’, mainly
concerned with visualization and local
interaction handling
– Proxies perform boundary testing and
act as message gateways
• Easy to add another shell of remote
rendering servers to the architectural
design
– Transparent transition between local and
remote rendering cases (e.g. from PC to
mobile device or low-end console)
– Decoupling of game logic and remote
rendering infrastructure
– Conclusion : more generic solution for an
entire class of games
In-network adaptation of
Remote Rendering streams
• Growing interest in so-called Spectator Mode
– OnLive, Xbox Live,...
• Generate stream once, serve large number of viewers
• What about heterogenous contexts ?
– Multitude of devices (high-end PCs, thin client consoles, tablets,
smartphones,…)
– Multitude of access networks (100Mbit cable, 3G HSDPA, EDGE)
• Create a reusable infrastructure for multiple purposes
– Not limited to Remote Rendering (generic video streaming-compliant)
• No adaptations needed to remote rendering system
– Can be applied to off-the-shelf remote rendering solutions or video
services
In-network adaptation of
Remote Rendering streams
• Reduce complexity required and load factor on remote
rendering setup
– Generate a single high-quality streams for all viewers
– Intelligence required on two levels :
• Network intelligence : bandwidth availability, channel quality, transit
delay,…
• Application intelligence : window size, user attention, capabilities of
device,…
In-network adaptation of
Remote Rendering streams
• Adapt streams on-the-fly using an overlay network of intelligent
proxies (NIProxy)
• Place the adaptation nodes at the edge of the network (integrated
at head-end of access network)
• Proxy exchanges
parameters (network &
application) with
client
• Transcode video on
demand, without server
intervention (or use
scalable video codecs)
Efficient delivery of
omni-directional video
• Omni-directional or panoramic video
allows exploration by the user in
recorded video sequences
– Move orientation of the camera after
capturing
– Zoom in/out and maintain quality level
– Like street view, but video-based
• Distribution of these sequences is subject to
optimization
– Not really required for low quality sequences (e.g. 720p
or less)
– Sequences captured by our hardware
are (typically) above 8000x1900 pixels
• Cannot be streamed at once to a client
(Set Top Box or Tablet device)
Efficient delivery of
omni-directional video
• Borrow ideas from remote
rendering to optimize the flow
– Perform pre-processing steps on the server to generate compliant
streams
– Do the precise customization for each user on the device (fat client) vs
the server (thin client)
– Retain interactivity (camera movement) as is typical in RR setups
• Recompositing the scene for each client at server-side is not very
scalable, so 2 alternatives :
1.
2.
Add meta-data to the stream and customize streams by
manipulating the compressed bitstream (using H.264 slices)
Cut the generated stream into small segments that are individually
decodable, have client request only those required
Efficient delivery of
omni-directional video
Original sequence
• Pre-processing steps for scalable delivery
using segments
Output files
Parallel encoding
and HLSsegmentation
hardware (multicore servers)
One file per quality level, per minute, per segment
Efficient delivery of
omni-directional video
Original sequence
• Pre-processing steps for scalable delivery
using H.264 slices
Output file
Encoding
hardware
One file per quality level
Efficient delivery of
omni-directional video
Cluster of custom servers
3 streams: one for each resolution
• Delivery to end-user
over standard protocols
• Easily scalable back-end
• Server does not perform full
rendering sequence, but
composes custom viewport
using slices (defined in the
compressed domain) or
delivers segments on
demand
Viewport cropping
20
Efficient delivery of
omni-directional video
• End-user application
– Based on (IP) Set Top Boxes
– Second Screen applications (besides primary TV broadcast)
on tablets (android/iPad)
– Web platform (WebGL with HTML5 video features)
• Currently in testing phase, will be rolled out
on testbed in short term (EOY)
More information ?
•
•
•
•
•
"Adapting a Large Scale Networked Virtual Environment for Display on a PDA". Tom Jehaes,
Peter Quax, Wim Lamotte, Proc. of ACE 2005, Jun. 2005.
"On the applicability of Remote Rendering of Networked Virtual Environments on Mobile
Devices" . Peter Quax, Bjorn Geuns, Tom Jehaes, Gert Vansichem, Wim Lamotte, Proceedings
of the International Conference on Systems and Network Communications, IEEE, Oct. 2006.
"Objective and Subjective Evaluation of the Influence of Small Amounts of Delay and Jitter on
a Recent First Person Shooter Game". Peter Quax, Patrick Monsieurs, Wim Lamotte, Danny De
Vleeschauwer, Natalie Degrande, Proc. of NETGAMES2004, ISBN 1-58113-942-X, Aug. 2004.
“Effective and Resource-Efficient Multimedia Communication Using the NIProxy “. Maarten
Wijnants and Wim Lamotte. Proceedings of the 8th IEEE International Conference on
Networks (ICN 2009)
"ALVIC-NG : State Management and Immersive Communication for Massively Multiplayer
Online Games and Communities" . Peter Quax, Bart Cornelissen, Jeroen Dierckx, Gert
Vansichem, Wim Lamotte. Journal on Multimedia Tools and Applications, Special Issue on
Massively Multiplayer Online Games. Volume 45, Issue 1 (2009), Page 109. Springer.
http://www.edm.uhasselt.be
[email protected]