Transcript PPT - the UNC Department of Computer Science

Transport and Rendering Challenges of
Multi-Stream, 3D Tele-Immersion Data
Herman Towles,
Sang-Uok Kum, Travis Sparks, Sudipta Sinha, Scott
Larsen and Nathan Beddes
Department of Computer Science
University of North Carolina at Chapel Hill
October 28, 2003
1
‘’Portal’ to Another Place
Static, High-Resolution 3D Scene
UNC - January 2000
Head-Tracked, Passive Stereo Projector Display
2
Static Scene Acquisition
• DeltaSphere™ Scanning
Laser Rangefinder
– Time of Flight
– Resolution: ~3K x 2K for 150o
azimuth and 90o elevation
– Digital Camera mounted at
identical COP for color image
• Output
– Range Image
– Color Camera Image
Images courtesy of 3rd Tech, Inc.
3
NTII Collaborators
• UNC-CH – Computer Graphics
– Henry Fuchs et al.
• UPenn – Computer Vision
– Kostas Daniilidas et al.
• Brown University – Collaborative Graphics
– Andy van Dam et al.
• Advanced Network & Services
– Jaron Lanier & Amela Sadagic
Goal: Develop a Live 3D ‘Portal’
4
3D Tele-Immersion – Phase 1
• 3D Real-Time Reconstruction
– Correlation-based Stereo Algorithm
– 1-3 fps at 320 x 240 (quad-PC/550)
– 1 cubic meter working volume
– Up to 5 streams of depth images
• 3D Tracked Stereo Display
– Life-size with gaze awareness
– 3-PC Rendering Cluster
– Composite Scene
» Live 3D Remote Collaborator
» Static, office background
» Collaborative Graphics
5
Standard Videoconferencing Model
Camera
Network
Monitor
• 1-Camera to 1-Monitor Paradigm
• Single Stream of 2D Image Data
View-Dependent Data !
6
3D Tele-Conferencing Model
3D Camera
3D Camera
Network
3D Rendering
3D Camera
User Tracker
• N-Camera to 1-Monitor
• Multiple Streams of 3D Image Data
– Depth Image (16-bit 1/z) and RGB Image (24-bit)
View-Independent Data w/ View-Dependent
Rendering !
7
3D Tele-Immersion- Phase 2
3D Camera
Acquisition
Acquisition
Network
Reconstruction
Reconstruction
3D Camera
Network
3D Rendering
• Higher Quality
– Improve Resolution, Frame
Rate and Display
• Larger Reconstruction
Volume
– 30-60 color cameras
– Acquire full environment
Terascale Computing System (Lemieux)
at Pittsburgh Supercomputing Center (PSC)
8
2000-2002 Results
• Show video
9
Outline
• Introduction
• Data Transport Challenges & Solutions
• Rendering Challenges & Solutions
• Conclusions/Futures
10
Data Transport Challenges
• Reliable Transport
– Low Latency
– Synchronized Arrival of Multi-Streams
• Large Bandwidth Requirements
No. Streams &
Resolution
2D Bandwidth
2Hz@24bpp,
Mbps
3D Bandwidth
2Hz@40bpp,
Mbps
5@320x240
18.4
30.8
5@640x480
73.8
122.8
15@640x480
221.4
368.4
11
Compression Opportunities
• Color Image Compression
– Spatial Compression (JPEG)
– Temporal Compression (MPEG)
• Depth Image
– 16-bit 1/z image
• Elimination of Redundant Data
– Overlapping FOVs from Multiple Cameras
– Reduces BW and Rendering Requirements !!
12
Overlapping Depth Images
13
Redundant Data Elimination
• Algorithm: Compare Every Depth
Stream
• Pros
– Best compression
– Eliminates all redundant Data
• Cons
– Jeopardizes Real-Time Performance
– Not scalable with number of 3D cameras
– Maximizes network BW requirements between
‘3D Cameras’
14
Group-Based, Differential Stream
Compression
• Two Stream Differential Comparison
– Maintains real-time performance
- Scales with number of cameras
- Static, Disjoint Stream Groups
- Geometric coherency metric
• One Reference Stream per Group
• Main Reference Stream
– Stream most closely aligned with remote viewpoint
– Uncompressed
15
Grouping & Differential
Encoding
1
2
3
Group
1
4
5
6
Group
2
7
8
9
Group
3
16
Algorithm Performance
• Using Synthetic Camera Arrays
– 13 and 22 3D-camera arrays
– Real-world 3D data
• 5X Compression Ratio
• 50-60% of Best Possible Point
Reduction
• 10z Performance at 640x480
17
Multi-Stream Transport
• Reliable is Good, so TCP/IP
WRONG!
• Latency
• Unsynchronized Arrival of Streams
– Synchronized at Cameras
– Must be re-synchronize at renderer
• TCP Flows Competing for BW
18
Frame Arrival Time Variation
19
Throughput Variation
20
CP-RUDP
• Reliable UDP (RUDP)
• Coordination Protocol (CP)
• Characteristics
– A TCP-friendly UDP protocol
– Aggregately acts like N TCP flows
– Providing BW coordination and synchronization of multiple
flows
– Prevents send rates from exceeding network capacity
Flows that are ahead are given less BW
Flows that are behind are given more BW
21
Future CP-RUDP Testbed
• Future Network Router Functionality
• Emulated today on fast FreeBSD PC
3D Camera
3D Camera
CP-RUDP
Software
Router
CP-RUDP
Software
Router
Internet2
3D Rendering
3D Camera
User Tracker
22
Outline
• Introduction
• Data Transport Challenges & Solutions
• Rendering Challenges & Solutions
• Conclusions/Futures
23
Rendering Challenges
• High Performance, Interactive
Architecture
• Quality Surface Representation
• Scalable Performance
24
High Performance, Interactive
Architecture
• Frame Re-synchronization
– ‘N’ 3D-camera Streams
– Timestamps on each data frame
– Buffering -> More Latency BAD!
• Multi-threaded Design for Interactivity
– 30Hz Refresh Rate Desirable
– <10 Hz Data Update Rate
• 3-PC Rendering Cluster (Linux)
– Left & right eye rendering nodes
– 3rd Node – Network Aggregation Point
25
Rendering Architecture
26
Quality Surface Representation
• Depth Images (1/z + RGB per pixel)
• Real-Time Triangulation
– Performance Limiting
– Artifacts due to noisy outliers. Most Disturbing!
• Real-Time Massive Point Cloud Rendering
– Display List vs. VAR
– GL_POINT vs. GL_POINT_SPRINT
– Screen-Oriented vs. Object-Oriented Splats
27
Fixed-Size Splats
28
Screen vs. Object Space Splats
29
Scalable Performance
• 2x Gain w/3-PC Rendering Cluster
• Ride the Wave (Moore’s Law & Beyond)
Not Enough!
• Divide and Conquer
– Eliminate Data Replication Requirement
– Match # Primitives to Performance
• Depth Compositing
– UNC PixelFlow, Stanford/Intel Lightning-2 or HP Sepia
– DVI output – RGB and Z
30
Conclusions/Futures
• Much work remains !
– Performance and Fidelity
• Investigating Temporal Compression of 3D
Streams
• Testing CP-RUDP Network Protocol
• New Voxel-Based Solution
– Elminates Redundant Data
– Normals and Confidence Value per Voxel
– GPU Programs for Object-oriented Quad Splats with
Projective Texture
31
Research Sponsors
• National Science Foundation
– Grant IIS-0121293 Junku Yuh, Program Director
• DARPA
– 3-D Tele-Immersion over NGI
• Advanced Network & Services, Armonk, NY
– National Tele-Immersion Initiative (NTII), 1998-2000
32
Thank You
UNC - January 2000
33
Thank You
UNC - January 2000
34
Positive Technology Trends
• Low-cost image sensors
• Vigorous research into new light field and
image-based scene reconstruction algorithms
• Incredible performance in commodity PCs and
3D graphics
• Ease of Developing Custom Hardware
• New Displays – HiRes LCD and Plasma panels,
Tiled-Projectors, OLEDs, Auto-stereoscopic
• Increasing network bandwidth to the
workplace and home
35
Renderer Block Diagram
36
VIRTUE: VIRtual Team User
Environment
• Information Society Technologies (IST)
Programme of the European Commission
http://bs.hhi.de/projects/VIRTUE.htm
37
VIRTUE Partners
• Heinrich-Hertz Institut (now Fraunhofer
member)
– Disparity Estimator, Real-time Components
• University of Delft
– 3D Algorithms, Image-Based Rendering
• TNO Human Factors
• Sony U.K.
– System design, Compositor, Rendering, MPEG-2
• Harriot Watt University
– 3D Video Processing
• British Telecom
– Project Prime, Video-based Head Tracking
38
VIRTUE Technology
• Realistic Wide View Synthesis for Dynamic
Scenes
• Object-based Segmentation
• Disparity Engine runs 4x4 grid @ videorate
• Facial feature (eye) tracking
• Generating View-dependent Novel View
• 2D Display (with motion parallax) rendered
with virtual background
Demonstrated at IBC – Fall 2002
39
HP Labs ‘Coliseum’ System
Image-based Visual Hull
40
HP Labs ‘Coliseum’ System
• Personal Immersive Conferencing
• Based on ‘Visual Hull’ research of Matusik,
McMillan et al. at MIT
– Space carving algorithm
– Foreground/Background
• 5 camera (1394) array, Now runs on 1 PC
– 224 x 224 @ 10 Hz
• 300 x 300 display window
– 3D Volume Composited with 3D VRML scene
Demonstrated at ITP – Dec 2002
41
Credits
• Colleagues at
– UPenn: Kostas Daniilidis (Co-PI), Nikhil Kelshikar, Xenofon
Zabulis
– PSC: John Urbanic, Kathy Benninger
– Advanced Networks & Services: Jaron Lanier, Amela Sadagic
– UNC: Henry Fuchs (PI), Greg Welch, Ketan Mayer-Patel
• UNC RAs
– Ruigang Yang, Wei-Chao Chen, Sang-Uok Kum, Sudipta Sinha,
Scott Larsen, Vivek Sawant, Travis Sparks, David Ott, and
Gabe Su
42
Our Vision: ‘Xtreme Tele-Presence
‘Office of the Future’
Andrei State 1998
43
Our Vision: ‘Xtreme Challenges
Distributed Graphics
Presentation
•
•
•
•
• Collaborative Datasets
• User Interfaces
3D Stereo Display
View Dependent
Front Projection
Spatialized Audio
3D Scene Acquisition
• Cameras
• 3D Reconstruction
Networking
• QoS
• BW-Latency-Jitter
‘Office of the Future’
Andrei State 1998
44
Immersive Electronic Books
• New Paradigm for Surgical Training
• Allow Surgeons to Witness & Explore
– Replay a surgical procedure from any novel viewpoint in 3D
– Augmented with annotations & relevant medical metadata
45
Remote Medical Consultation
46