Transcript Goal: seamless mobile multimedia collaboration across distance

Pervasive Pixels
(Columbia University
Dept. of Computer Science)
Henning Schulzrinne (PI)
Steven K. Feiner
Gail Kaiser
John Kender
Kathleen McKeown
Proposed Research
Goal: seamless mobile multimedia
collaboration across distance

Integrate advances across fields
Collaborative work
 Graphical and visual interfaces
 Spoken language understanding and generation
 Vision sensing and understanding
 Networking and security

Contributions

Contextual information management



Harmonizing physical and virtual environments



use workflow to determine display content
multimedia summaries of past and present sessions
map changing virtual information onto physical displays
map layout of physical environment onto virtual space
Network services



clear, flexible interface to common services
authentication and privacy support
infrastructure for persistent large displays
Features of Research
Infrastructure
Large numbers of instrumented multidisplay workspaces
 Networked mobile devices of various
capabilities
 Transparent and automatic adaptability
to changes of place, platform or group
 Support for a wide range of hardware
and software, from commercial to novel

Proposed Research
Infrastructure

Outfit informal areas for collaboration




Stationary setups




Public areas for walk-by interaction
Multiple touch displays, cameras, audio
Portable units
Multiple displays, video cameras, audio
Seminar room, meeting rooms
12 faculty offices
User-based personalization: user location



Triangulation on mobile devices
Visual tracking
Standard methods (e.g., active badge)
Public areas – walk by stations
Multiple touch
displays,
video
projectors
and cameras,
embedded
computers,
speakers and
microphones
Design for Walk-by Collaboration Station
ceiling
network
PC
loudspeaker
proj.
camera
array
microphone
electronic
whiteboard
IR/RF badge
card reader
Public Areas – informal
gatherings
Meeting Room

Remotecontrolled pantilt video
cameras and
projectors,
Omnicam,
conference
table
microphones,
automatic audio
mixer, ceiling
speakers
Faculty Office

Mimio electronic whiteboard, XGA video projector, Ethernet
speaker phone, wall-mounted pan-tilt video camera,
PocketPCs
Seminar room
Omnicam
omnidrectional
audience camera, highresolution DV video
camera, 2 pan-tilt
speaker cameras,
ceiling mounted
microphones,
electronic whiteboard,
XGA high-brightness
video projectors
Functionalities

Conferencing


Interconnection with analog phone






Internet conferencing server to mix IP and PSTN audio
streams
Digital hybrid connects digital or analog sound to existing
telephone system in classroom
Network voice-over-IP interface attached to Nortel
Meridian PBX for 20 simultaneous conversations
Multi-processor servers and IA64 compute and
database server
File storage
Face, speaker and fingerprint recognition
Backup facilities: 2 printers and tape library system
Initial results

HCI: gesture-based user interface for public kiosk



Security: disCFS and WebDAVA secure file systems



disCFS: NFS with credentials instead of authorization
WebDAVA: grant restricted access to resources using HTTP
and Java applets
Web-based collaboration:



mouse replacement for pointing and selecting
uses frontal and side camera
content on all kinds of devices
pass DOM through a series of filters and transformations 
HTML
Ubiquitous multimedia communications infrastructure


being commercialized; I2 demonstration
input into standardization (IETF)
Ubiquitous Computing

Traditionally, focus on closed environments






proprietary protocols
single (trusted) user class
single site (room, lab, home, …)
stand-alone components (“video conferencing”)
PP focuses on whole system and user experience
Pervasive Pixels networking component:

standard protocols:



integration of presence and user context


SIP for media configuration, event notification, instant multimedia
messaging
SLP for service discovery
standardization in the IETF (RPID)
location-based services



user context
user authorization
service location
Mobility in Pervasive Pixels

Terminal mobility


Session mobility



move active sessions to devices
found in the environment
 service discovery
Service mobility


application-layer mobility
complements L3 mobility
move configuration to new
devices
Personal mobility

one user, many devices
Location-based services


Traditionally, focus on
geospatial location (e.g., GPS)
But other aspects as important:




civil location (often more
intuitive)
type of place (home vs. office;
outdoors vs. theatre)
behavioral: distraction, privacy,
appropriateness
Experimenting with lowcomplexity location
mechanisms:




IR/RF active badges with low
installation cost (Ivistar)
BlueTooth location beacons
LAN backtracking and DHCP
swipe cards and i-buttons
8:0:20:ab:d5:d
DHCP
server
CDP + SNMP
8:0:20:ab:d5:d  458/17
DHCP answer:
sta=DC loc=Rm815
lat=38.89868 long=77.03723
458/17  Rm. 815
458/18  Rm. 816
Some initial lessons learned

Usage: remote presence from UKy during sabbatical





Perception: “Multimedia collaboration is a mature field”
Reality: It doesn’t work much better than in 1992



integrating documents, minutes, …
Transition from call-focused to presence-focused


still fails in hard-to-diagnose ways
quality better, but echo, feedback and level issues remain
Integration between synchronous and asynchronous
collaboration


research group meetings
departmental site visit
thesis proposals and defenses
much larger use of asynchronous collaboration (email, bulletin
boards, …)
Working with start-up company: new IP-based departmental
communication system to replace PBX
Columbia SIP servers (CINEMA)
Telephone
switch
Local/long distance
1-212-5551212
Internal
Telephone
Extn: 7040
Single machine
Department
PBX
713x
SIP/PSTN Gateway
Extn: 7134
rtspd: media server
RTSP
sipconf:
Conference server
sipd:
Proxy,
redirect,
registrar
server
RTSP clients
sipum:
Unified
messaging
SQL
database
Web
server
Web based
configuration
SNMP
(Network
Management)
H.323
Extn: 7136
xiaotaow@cs
Quicktime
siph323:
SIP-H.323
translator
NetMeeting
Larger lessons for multimedia
systems research

Software tool support for multimedia communications
lacking




Components designed to be operated by humans



most are applications, not building blocks
cross-platform research media tools are getting very old and
creaky (vic, rat, etc.)
multi-party support very weak (multicast never happened)
IP phones only have HTTP/HTML interface
video projectors just proprietary configuration
Lots of components, but hard to evaluate in real use


still mostly barely demo quality: audio delay, echo, random
failures
people will fall back to good ol’ PSTN quickly
Conclusion



Pervasive Pixels = attempt to integrate
multiple modalities into system, not just
grouping of components
Evaluation in real usage, not just demos
Spread throughout the department, not
just lab