Transcript DeveloperWeek

Creating a Truly Immersive VR Experience
Through the Power of Touch
Chris Ullrich | VP UX and Analytics
Immersion Corp.
Overview
• Touch enables presence in virtual reality
• Haptic devices for virtual reality
• Haptic design
• Touch feedback APIs
• Challenges and Opportunities
2
©2017 Immersion Corporation
Part 1:
What is the role of Touch in Virtual Reality?
3
©2017 Immersion Corporation
The Promise of
Gaming and Entertainment with VR
Vision
Audio
IMMERSION
PRESENCE
Touch
The Promise of
Gaming and Entertainment with VR
Vision
Audio
IMMERSION
PRESENCE
Touch
The Digital Entertainment Experience
is Evolving to Full Immersion
360
Silent Films
Binary Arcade
Color
Home Consoles
Animation
CGI
3D
360 Video & VR
PC
Force Feedback
& Rumble
Mobile Games
Virtual Reality
Holodeck Experience
Touch Feedback Hardware for Virtual Reality
7
©2017 Immersion Corporation
Categories of Touch Experience
?
1
No Haptics
2
3
4
Rumble
High Fidelity
Vibrotactile
Kinesthetic
Haptic Technology:
ERM
Haptic Technology:
LRA
Haptic Technology:
Falcon, TouchSense© Force
• No feedback
• Narrow range of effects
• Robust range of effects
• Limits VR to passive viewing
• High action, big event
• Wider range of effects with ability
to control fidelity
• Broken experience, causes
disillusion
• Broader range of action and
interactive games
• Examples: Sony PlayStation and
Xbox One
• Force feedback, proportional to
user input
• Examples: Falcon, Phantom
• Examples: Oculus and Vive
8
©2017 Immersion Corporation
Haptics in Gaming
Rumble is a standard that has been
around for 15+ years without change
Two motors that spin:
1. One large
2. One small
More strength, less speed
Low operating frequency is satisfying
Limited design bandwidth
Haptics in Consumer VR
• Virtual Reality controllers shift from a single
controller to dual motion controllers
• Neither Oculus Touch nor HTC Vive
controllers incorporate rumble
• Instead use single Linear Resonant
Actuators (LRA) per controller with a limited
multi-frequency band
Haptics in VR/Gaming –
Other Technologies
SubPac
Tactical Haptics - Motion Controller
with Reactive Grip Touch Feedback
+ many more on Kickstarter
General Purpose Devices
These devices enable interaction with arbitrary virtual environments
CyberForce
Phantom OMNI
Novint Falcon
Special Purpose: Surgical Trainer
Full Body Haptic Immersion
These devices enable interaction with arbitrary virtual environments
Haptic Workstation (2003)
AxonVR (2017)
Touch Feedback is Essential in
Creating a Fully Immersive Experience
Today
Audio
Vibration
Pressure
Future
VR Headset
Squeeze
Deformation
Direct Neural
Stimulation
We Are
Here
Multitouch
Haptic Sensor Input
Voice
Kinesthetic
Haptic Feedback Output
Friction
Non-Haptic Sensor Input
HRTF Audio
Temperature
Non-Haptic Feedback Output
Part 2:
Haptic Design and Implementation
16
©2017 Immersion Corporation
Haptic Design Principles
• Haptics creates the illusion of
tangible reality.
• Haptics reinforces realism
portrayed in visuals and audio
Touch
• Result: Users find tactile virtual
environments more engaging
and immersive
Visual
Auditory
18
©2017 Immersion Corporation
Model of Haptic Value
Enable basic
interaction with
virtual environments
Changes user’s
ability to function
in virtual
environments
User feels present
in virtual
environments
PRESENCE
PERFORMANCE
USABILITY
IMMERSION
18
©2017 Immersion Corporation
Experiences with Tactile Dimension
Weapons
Ambient Effects
Intuition
Road Effects
Interactivity
Explosions
Real-world
H-2-H Combat
Crashes
Force
Haptic Effect Mappings
Basics of Haptic Effect Design
Clear
Association
Low Latency
Feedback
Short
Duration
21
Tell The Story
With Touch
©2017 Immersion Corporation
Basic Effect Types
Premade
Audio
derived
Parameterized
22
Real-time
Synthesis
©2017 Immersion Corporation
Spatial Effects
Oculus and Vive have 2 controllers, which enable spatialized haptic effects
Occurrences on left side of a character would be mapped to
the left device; those on right would be mapped to right device
If a character’s left hand did something,
they would feel it on the left device
Funneling
Haptic illusion that occurs
between two controllers
Fading between devices can create
a ‘stereo’ sensation localized
between the two controllers
If the avatar is hit on the left side, the
effect would be on the left controller
23
©2017 Immersion Corporation
Practical Implementation of Touch
24
©2017 Immersion Corporation
Rendering Haptic Signals - Considerations
Glitches
And Timing
Synchronization
25
Motor
Dynamics
©2017 Immersion Corporation
Haptics on HTC Vive and Oculus Touch
• Both controllers offer a basic effect function
// duration is in 𝜇s from 0-3999
SteamVR_Controller.Input([index]).TriggerHapticPulse( duration );
// magnitude and frequency are from 0.0f to 1.0f
OVRInput.SetControllerVibration(magnitude, frequency, Controller);
• According to online documentation – SetControllerVibration() is
documented as depreciated for Touch controller (but it still works for me)
• Oculus touch now offers a buffer interface for Touch Controllers
26
©2017 Immersion Corporation
Unity NewtonVR Example - Oculus Touch
public override void TriggerHapticPulse(ushort durationMicroSec = 500,
NVRButtons button = NVRButtons.Touchpad)
{
StartCoroutine(DoHapticPulse(durationMicroSec));
}
private IEnumerator DoHapticPulse(ushort durationMicroSec)
{
OVRInput.SetControllerVibration(0.2f, 0.2f, Controller);
float endTime = Time.time + ((float)durationMicroSec / 1000000);
do
{
yield return null;
} while (Time.time < endTime);
OVRInput.SetControllerVibration(0, 0, Controller);
}
• Plays effects in a
separate thread
• Use ‘yield return’
to play effects
longer than one
Unity frame
https://github.com/TomorrowTodayLabs/NewtonVR
27
©2017 Immersion Corporation
HTC Vive Effect - SteamVR
//length is how long the vibration should go for
//strength is vibration strength from 0-1
IEnumerator LongVibration(float length, float strength) {
for(float i = 0; i < length; i += Time.deltaTime) {
SteamVR_Controller.Input([index]).TriggerHapticPulse(
(ushort)Mathf.Lerp(0, 3999, strength));
• Same thread as
render loop
• 4ms chunks
of signal
yield return null;
}
• Example effect is
parameterized
}
28
©2017 Immersion Corporation
Vive Effect - SteamVR
// vibrationCount is how many vibrations
// vibrationLength is how long each vibration should go for
// gapLength is how long to wait between vibrations
// strength is vibration strength from 0-1
IEnumerator LongVibration(int vibrationCount, float vibrationLength,
float gapLength, float strength)
{
strength = Mathf.Clamp01(strength);
for(int i = 0; i < vibrationCount; i++) {
if(i != 0) yield return new WaitForSeconds(gapLength); yield
return
• Pulsed haptic
effect example
• Separate thread
StartCoroutine(LongVibration(vibrationLength, strength));
}
}
https://steamcommunity.com/app/358720/discussions/0/405693392914144440
29
©2017 Immersion Corporation
OVRHapticsClip
• Buffer oriented API for Touch controllers
• Each buffer element is 3.125ms long
and 8bits
• Max buffer is 800ms
• Buffers can preempt or mix with existing
haptic effects
v
• Convenience method for creating a
haptic effect from an audio clip
30
©2017 Immersion Corporation
Using the OVR API
OVRHapticsClip hapticClip = new OVRHapticsClip(myAudioClip)
OVRHaptics.RightChannel.Preempt(hapticClip);
‘OVRHapticsClip reads in an audio clip, downsamples the audio data to a sequence of
bytes with the expected sample rate and amplitude range, and feeds that data into the
clip’s internal amplitude buffer.’
• Audio buffers can create interesting haptic effects
 Just using the sound effect is not usually the best haptic experience
• Can create a library of effects or synthesize audio buffers
 Ramps, pops, etc. from audio tools or sources
https://developer3.oculus.com/documentation/game-engines/latest/concepts/unity-ovrhaptics/
31
©2017 Immersion Corporation
Pseudo Code for ‘Physics’ Driven Haptics
• Create an effect as a function of momentum difference
• Experience:
 High relative momentum collisions are strong
// generic collision handler
OnCollisionEvent( Object _avatar, Object _objectB ) {
// Calculate relative momentum vector length
float relP = length(_avatar.mass*_avatar.velocity- _objectB.mass *_objectB.velocity);
// scale and clip for haptic effect
float magnitude = SCALE*clip( relP, 0.0, 100.0);
// Play collision vibration effect
LongVibration( 0.5f, magnitude);
}
32
©2017 Immersion Corporation
Gold Standard for Realism:
Proxy Object Algorithm
• Well known algorithm for continuous contact
 Haptic effect strength is proportional to proxy distance
• Works good for kinesthetic devices but
adaptable for vibration haptics
• Hard to do for kinematic avatars (e.g. hands)
• Users experience objects as ‘solid obstacles’
Figure 1. Relationship between proxy position
and that of the haptic device (SensAble, 2005)
Haptic Rendering: Foundations, Algorithms and Applications
https://www.amazon.com/Haptic-Rendering-Foundations-Algorithms-Applications/dp/1568813325
33
©2017 Immersion Corporation
Part 3:
Challenges and Opportunities
34
©2017 Immersion Corporation
Enabling Touch in VR
Experience design is enabled with a creative workflow and consistent playback.
Create
Distribute
Playback
Purposefully
designed
Encoded once to be
playable everywhere according to
the content creator’s design intent
Standard playback format for
a consistent experience on
different VR systems
35
©2017 Immersion Corporation
Visual Authoring Tools ~ 2005
Microsoft Visual Force Factory
Immersion I-Force Studio
36
©2017 Immersion Corporation
Visual Authoring Tools ~ 2016
Haptic design should be built as a part of the experience, not an afterthought
Visual track
Audio track
Haptic track
37
©2017 Immersion Corporation
Opportunity: Haptic Content Creation
Challenge
Expectation
Opportunity
• Designing haptic effects in
interactive environments is still
too complex
• Producers want a consistent
experience across different
haptic hardware
• There is an opportunity to
enable the ecosystem to
integrate haptics into VR
experiences
• Triggered effects
• Animation driven effects
• Synthesized effects
• VR content creation tools such
as Unity and Unreal Engine
have low level haptic support
38
©2017 Immersion Corporation
Enabling Presence
• There are limits to what can be done with vibrating controllers
 Haptics need to be highly dynamic and consistent with virtual environments
 Haptic controller transparency strongly affected by choice of motor
o
High definition actuators significantly better
• Next gen controller form factors
 Inspired by force feedback controllers of the past but with greater transparency
 Vibrotactile-Kinesthetic metaphors provide an extremely rich (and cost effective) display for
consumer virtual reality
Conclusions
40
©2017 Immersion Corporation
Summary
• Haptic devices are more available than ever before, particularly for
consumer virtual reality
• There are good metaphors to leverage haptic feedback for presence in VR
• Creating and managing meaningful haptic effects requires some effort:
low level API’s make this possible but difficult
• There is no good solution for consumer haptic effect design
41
©2017 Immersion Corporation
Touch Matters
Touch is the bridge between physical and digital reality
…touch adds
interaction and control
…touch helps
suspend disbelief
42
…touch provides
a sense of presence
©2017 Immersion Corporation
Thank You
43
©2017 Immersion Corporation