Transcript Virtual Reality: History

Curs 7
Arhitecturi SW pentru RV
Networked VE
Virtual Reality: History
Sistemele de RV sunt complexe
Dezvoltarea unei aplicații complete de RV este o
activitate care necesită manipularea mai multor
tehnologii și rezolvarea sarcinilor in anumite limite
de timp.
• Pentru dezvoltarea unui sistem complet, o medie de timp
satisfacatoare este de 12 luni.
• Teamwork-ul joaca un rol fundamental.
Virtual Reality: History
Complexitatea sistemelor de RV crește
• Cresc cunoștințele noastre cu privire la mecanismele de percepție
umană
• Progresele tehnologice sunt lente dar constante: performanțele
generale ale sistemului sunt în creștere și blocajele sunt eliminate
• Complexitatea tot mai mare a sistemelor rezultate
Virtual Reality: History
Componente multiple
Collision
Detection
Real-Time
Physics
Real-Time
Graphics
Tracking
Sound
Haptics
Motion
Control
Network
 Un sistem modern de VR este o colecție de mai multe module de înaltă
performanță
 Fiecare componentă are un domeniu de cercetare propriu, și necesită
tehnologii dedicate, instrumente și abordări specifice.
Virtual Reality: History
Grafica 3D
"Interactivitatea" necesită un “load” foarte
mare poligoane;
Multe tehnici și algoritmi există (sau sunt
dezvoltate) pentru manipularea optimă a
bazelor de date foarte mari cu date
geometrice;
Virtual Reality: History
Collision Detection
It used to be a “big thing” in VR.
• Collision detection algorithms searches
for virtual objects interferences
•Algorithms are almost optimal – very
mature research field
• There is little point in collision detection
without physic simulation
• Revived a bit lately: GPU-based collision
detection
Virtual Reality: History
Physics simulation
Interaction between virtual objects
need to be simulated with virtual
physics
• Objects in a VE should react to stimulus
(gravity, external forces, collisions)
• Compound objects need to be considered,
and mechanical effects of joints between
rigid bodies simulated.
• Well known simulation models, but again
the hard task is simulation in real-time.
Opportunities & challenges:
• Parallel computing and dedicated coprocessors.
• Real-time simulation of deformable
objects.
Virtual Reality: History
Positional sound



Sound sources can be:
– Sampled: playback of a pre-existing files (with additional, post
processing effects)
– Synthesized: sound is generated on-line
Audio sources usually needs to be positioned in 3D space (and they
move around)
Even when the sources are static the listener is moving
Virtual Reality: History
Interaction Devices Management
Users may interact with a system
using various types of devices and
technology
 Interaction can be 3DOF, 6DOF,
force enabled, thought enabled
and more…
 New forms of interactions are
emerging such as using portable
BCI devices
 Complex devices require
dedicated APIs and access
paradigms

Virtual Reality: History
Haptic feedback
• Haptic-enabled manipulaton involves
the user’s body perception-action
feedback loop: still a vastly
unexplored research field with many
technological challenge.
• Advanced software techniques and
hardware devices need to be
developed to achieve realistic haptic
rendering of rigid and soft objects
• Haptic interfaces are usually custom
objects and depends on the
applications
Virtual Reality: History
Inertia feedback
In some VR systems the user
is placed on a moving platform
Stewart platform: six independently
actuated legs, where the lengths of
the legs are changed to position and
orient the platform.
Virtual Reality: History
Networking
“No man is an island” communication between remote VR nodes can be
very important.
 Interleaved data streams: audio, video, geometry transformations, haptics
and more.

Virtual Reality: History
Developing VR apps: the big picture
• VR is really about interaction. From a software point of
view, interaction is a time-critical activity
• Consequence: any VR application is a collection of high
performance, low latency, dedicated tasks
• Tasks are not independent. There are cross-tasks
implications that should always be considered
Virtual Reality: History
What is the “state” of a VR application?
• In theory: a collection of information about the virtual world
• In practice: a collection of data structures and variables
3D meshes
Forces
Positions
VR World
“state”
Velocities
Accelerations
Orientations
Virtual Reality: History
The state evolves over time…

…advancing in a discrete sequence of steps and
following the simulation computation.
t0
t1
t2
t3
Virtual Reality: History
Why does the state keep evolving?

Mainly to accommodate the effects of two things:
– External events (mainly due to user input/tracking);
– Internal events (timers, network events)
•Button_A_pressed!
•Joy_1_MOVED!
•MOUSE_OVER!
VR World
“state”
•Time_01!
•Time_02!
•Time_03!
Virtual Reality: History
Q: How frequently does the state evolve?

A: Fast enough to keep up with the fastest activity taking place
inside the simulation.
Simulation
step
t0

t1
t2
t3
Consequence: the internal update rate of the simulation depends
on the VR system configuration and the tasks performed
Virtual Reality: History
Loops in a typical VR application
~10Hz
~ 60 Hz
100-200 Hz
>1kHz
~30Hz
Virtual Reality: History
The graphics loop
The state of a VR simulation need to be displayed at regular intervals
(framerate) on a monitor, HMD, CAVE, etc
≈60Hz
State(t)
Update frequency = monitor refresh
Anything more would be wasted
Virtual Reality: History
The collision detection loop
Checking if moving objects collide with each other
???
State(t)
• If collision checking is not “fast-enough” some
collision could occur undetected
Virtual Reality: History
The physical simulation loop


Objects responds to physical laws and interact
Update frequency needs to occur at least at the same
speed as display rate, but stability issues or haptic
rendering often requires much higher speeds
>=100Hz
State(t)
Virtual Reality: History
Real-time physics in practice (I)
• A world is a collection of bodies
• Bodies from different worlds never interact
• Each rigid body has:
• a mass (and moment of inertia)
 needed for motion calculation
 obtainable by composing elementary
solids
• a shape or “geometry”
 needed for collision detection
 obtainable by composing elementary
solids
Note: we are considering rigid-bodies physics
Virtual Reality: History
Real-time physics in practice (II)
• Physics materials are used to describe surface
properties of objects
var material = sim.getMaterial(idx);
material.restitution = 0.1;
material.staticFriction = 0.8;
material.dynamicFriction = 0.5;
material.apply();
Virtual Reality: History
Real-time physics in practice (III)
• Simple objects are connected
using a collection of physical
joints
• Many types of joints are
available
•Sometimes joints can be
broken
Virtual Reality: History
Real-time physics in practice
The physics loop:
1
2
3
1) Apply all the external forces to the scene objects
2) Ask the Physics engine to compute the evolution of
the scene during a certain time interval
3) Get the results back
• Update the scene-graph
• *optionally* : control some external device
Virtual Reality: History
The haptic loop(I)


Haptic refresh rate is much higher than graphics refresh
rate!
Stiffness in haptic rendering is influenced by loop speed
-> collision detection speed
>=500Hz
State(t)
Virtual Reality: History
The haptic loop (II)
Position
(x,y,z)
1
2
3
Force
(fx,fy,fz)
1) Read the device current position*
2) Depending on that position compute a force
3) Ask the device to apply that force
* Reading the device velocity can be useful too
Virtual Reality: History
So, does this mean that physics must run at 500 Hz too?
500
Hz
State(t)
500
Hz?
Physics
Well..., depends:
1) For optimal and more accurate computation and
rendering of forces, yes.
2) To reduce computational workload “virtual coupling” can
in some situations be used.
Virtual Reality: History
The idea behind virtual coupling
500
Hz
Virtual
Coupler
100
Hz
State(t)
100
Hz
Physics
• A virtual spring is attached between the VR object and the
tip of the haptic interface
• The object is considered inamovable between two
consecutive physics loops
• 500Hz computation is only performed to compute (and
apply) the virtual spring elongation forces
Virtual Reality: History
The inertia feedback loop


Some VR applications use also a motion platform
Device control is typically performed at around 100Hz
≈100Hz
State(t)
Virtual Reality: History
Controlling a motion platform
• The platform controller
waits for:
• x, y, z (floating point)
• Roll, Pitch, Yaw
• The system has a high
mechanical inertia
Position commands takes a
while to get executed
Virtual Reality: History
Computing perceived acceleration
• A physic engine can be used
to compute position, velocity
and acceleration of the “head
proxy” of the user.
•Data is passed to a Washout
filter to get “transformed”
Position, Velocity,
Acceleration
Washout
filter
Virtual Reality: History
The data flow
Physics
engine
>=100Hz
Washout
filter
Position, Velocity,
Acceleration
≈ 60Hz
x, y, z
Roll, Pitch, Yaw
Virtual Reality: History
From acceleration to inclination
Washout
filter
•For low-frequency accelerations, the
washout filters tries to use gravity to
simulate forces.
Position, Velocity,
Acceleration
Virtual Reality: History
Network communication loop


Communication between remote VR nodes can be very
important
Update frequency depends on the type of
communication (LAN,WAN), interaction (with or
without haptics) etc…
State(t)
Node 1
10Hz –
500Hz
State(t)
Node 2
Virtual Reality: History
Conclusion: loops are everywhere
~10Hz
~ 60 Hz

100-200 Hz
Requisite I for Integration:
A precise, possibly
deterministic, fine control
on all the loops
>500Hz
~30Hz
Virtual Reality: History
Variables are shared between loops
Graphics
VR World
“state”
Haptics
Physics

Requisite II for Integration : A good framework
takes care of data synchronization
Virtual Reality: History
Same object, multiple representations
Haptic shapes and and
physics shapes
Physics shapes only
Virtual Reality: History
Events: loops can’t model everything
VR_EVENT_
2
VR_EVENT_
1
VR_EVENT_
3
VR_EVENT_
4
~30Hz
Virtual Reality: History
A better general overview
~10Hz
~ 60 Hz
100-200 Hz
>1kHz
~30Hz
~10Hz
~ 60 Hz
100-200 Hz
>1kHz
~30Hz
~10Hz
~ 60 Hz
100-200 Hz
>1kHz
~30Hz

Events and loops are orthogonal concepts: Requisite III
for Integration : a good framework should handle both.
Virtual Reality: History
So, writing VR applications can be hard
• Usually requires excellent C++ skills.
•Hi-performance tools and libraries need to be properly
handled (otherwise performances might suffer).
• Plenty of tools available, but mixing them is not trivial.
Redundancy in files/data structures/code waste a lot of
resources. A LOT of work goes into the framework.
3/29/2016
Software architectures
for modern Virtual
Environments
Virtual
Reality: History
When we discuss networked nodes we
can be referring to:

-

-
Physical connections
the wires between the computers
Logical connections
how the messages flow trough the
nodes
Virtual Reality: History
Nodes communicates over a network
Virtual Reality: History
Fundamentals of Data Transfer (1)
Network bandwidth
 the rate at which the network can deliver data to the destination
host (bits per second, bps)
Network reliability
 a measure of how much data is lost by the network during the
journey from source to destination host
types of data loss:
 dropping: the data does not arrive
 corruption: the content has been changed
Virtual Reality: History
Fundamentals of Data Transfer (2)
Network latency
 the amount of time required to transfer a bit of data from one
point to another
 one of the biggest challenges: impacts directly the realism of the
NVE experience
 we cannot do much to reduce it
origins
 speed-of-light delay
 endpoint computers, operating systems scheduling
 the network itself, routers
Virtual Reality: History
Fundamentals of Data Transfer (3)
Network protocol
 a set of rules that two applications use to communicate
with each other
A protocol defines
 packet formats: understanding what the other endpoint is saying
 packet semantics: what the recipient should do when it receives a
packet
 error behaviour: what to do if (when) something goes wrong
Virtual Reality: History
BROADCAST based architectures
In the ideal large-scale NVE design all
nodes can communicate with others
directly
Advantages:
 Simple to implement
 Players can join at any time
Disadvantages:
 Only supported by LANs
 System management and update a
nightmare
 eventually we cannot scale out
Virtual Reality: History
Client-server architectures
Each player sends packets to other
players via a server
Advantages:
 Works over a WAN
 easy to have reliable communication
 Server can compress multiple
packets into a single packet
Disadvantages:
 Nodes are server-dependent: server
failure brings the system down
 Server slows down the message
delivery
Virtual Reality: History
PEER to PEER architectures

Peer-to-peer: communication goes directly
from the sending player to the receiving
player (or a set of them)
Advantages:
 It works over WAN
 We manage to avoid servers
Disadvantages:
 Nodes need to know network configuration
 Each node has to send same data multiple
times
Virtual Reality: History
Multiplayer Server-Network
Players can locate in the same
place in the NVE, but reside on
different servers
 Real World ≠ Virtual World
 Server-to-server connections
transmit the world state
information
 Each server serves a number of
client players
Virtual Reality: History
Internet Protocol

Low-level protocols used by hosts and routers
– Guides the packets from source to destination host
– Hides the transmission path (phone lines, LANs, WANs, wireless
radios, satellite links...)

Nodes are identified by 4 numbers (192.168.21.12)

Note that addresses can be Public (WAN) or Private (LAN) (more on
this later)
Virtual Reality: History
Internet Protocol



IP is organised in “layers”.
Applications access Transport layer (UDP or TCP APIs)
As we move down the layers data gets more and more
encapsulated
Virtual Reality: History
Traversing Multiple nodes
Virtual Reality: History
IP Transmission Techniques

Unicasting
– Single sender - single receiver

Broadcasting
– Single sender but all nodes in the network are receivers

Multicasting
– Single sender but one or more receivers have joined a multicast
group
Virtual Reality: History
TCP & UDP
Applications rarely use the IP directly but the protocols that are written on top of IP
- Transmission Control Protocol (TCP/IP)
- User Datagram Protocol (UDP/IP)
Transmission Control Protocol (TCP/IP)
User Datagram Protocol (UDP/IP)
Point-to-point connection
Reliable transmission using
acknowledgement and
retransmission
Stream-based data semantics
• Lightweight data transmission






Big overhead
Transmission and receiving NOT
immediate
Slower because of re-transmissions
• Differs from TCP connectionless
transmission
• ‘best-efforts’ delivery
• packet-based data semantics
• Packets are easy to process
• Transmission and receiving immediate
Virtual Reality: History
Selecting an NVE Protocol 1
Multiple protocols can be used in a single system. Which protocol
should I use in my NVE?
Using TCP/IP
 reliable data transmission between two hosts
 packets are delivered in order, error handling
 relatively easy to use


point-to-point limits its use in large-scale NVEs
bandwidth overhead
Virtual Reality: History
Selecting an NVE Protocol 2
Using UDP/IP
 Lightweight
 offers no reliability nor guarantees the order of packets
 packets can be sent to multiple hosts
 more complex services have to be implemented in the application
– serial numbers, timestamps
– recovery of lost packets
– positive acknowledgement scheme
– negative acknowledgement scheme
Virtual Reality: History