Transcript Augmenting Film and Video Footage with Sensor Data

Augmenting Film and Video
Footage with Sensor Data
N. Su, H. Park, E. Bostrom, J. Burke, M.
Srivastava, D. Estrin
PerCom ’04: March 14-17, 2004
Augmented Recording System
 Wireless
sensor network application for
filmmaking and media production
• Seamless integration
• Mobility
• Increased expressiveness
 Synchronize sensor data with video footage
 Sensor data allows post processing of video
March 10, 2005
Herman Li
2
Why not use image processing?
Image processing techniques can infer position,
motion, light condition…
 For example: Van Helsing
http://www.fxguide.com

Infrared LED Marker
March 10, 2005
Herman Li
3
http://www.fxguide.com
Why not use image processing?
March 10, 2005
Herman Li
4
Why not use image processing?
 Do
not offer fine grain data
 Cannot infer conditions outside of view, or
quantities such as wind speed or
temperature
March 10, 2005
Herman Li
5
Architecture
 Sensor
node
neighbourhood
 Serial port server
 Timecode generator
 Sylph server
middleware
 Jini client
 SQL database
March 10, 2005
Herman Li
6
1.
2.
3.
4.
5.
6.
7.
8.
Sylph server lookup
Sylph translates
query, forward to
serial port server
Serial port server
dispatches
messages to base
stations
Sensors begin
collecting data
Base station
forwards data to
serial port server
Serial port server
interpolates data
Sylph server
announces new
data
Jini client stores
data in database
March 10, 2005
Herman Li
7

Uses CrossBow Mica 1 / Mica
2 motes
• Thermistor, light sensor,
microphone, accelerometer.

Radio range of few hundred
feet, ~10kbps

Runs on PALOS (Power Aware Light-weight
Operating System)
March 10, 2005
Herman Li
http://computer.howstuffworks.com/mote.htm
Sensors node neighbourhood
8
Sensors node neighbourhood

Clustering
• Each base station responsible for a few motes
Base station
• Potentiometer calibration
• Admit closest X sensors
 Neighbours
• Internal frame counter

from 2 to infinity
• Sends data every 13 frames
• Filters redundant values
March 10, 2005
Herman Li
9
Serial port server
Controls base stations via serial connections
 Takes SMPTE timecode and
synchronizes with sensor data
 Combines all sensor data
for a frame and sends as one
packet
 Interpolates missing data

March 10, 2005
Herman Li
10
Serial port server
Time synchronization
• At most 7s per second drift
• One frame error per 4766.7 frames
• Time sync every 2.5 min
 Latency test
• shows delay of at most

3 frames
March 10, 2005
Herman Li
11
Sylph server middleware
 UCLA project,
used in Smart Kindergarten
 Allows queries on sensors
 Defines JINI attributes such as light, period,
command
 JINI query: “READ LIGHT EVERY 30
SECONDS”
 Translated: “SET PERIOD 30 SECONDS”,
“SET COMMAND=STARTSENDING”
March 10, 2005
Herman Li
12
JINI client
Client retrieves sensor data per frame and stores
in MS Access DB
 Provides playback features and data collection
controls

March 10, 2005
Herman Li
13
Evaluations
Deployed 2 base stations, 4 sensors each
 Clustering algorithm took 2.59 min
 Exp1: Graduate light intensity changes

March 10, 2005
Herman Li
14
Evaluations

Exp2: Delay measurement
• Some delay of ~10 frames
• Maximum of 20 frames
March 10, 2005
Herman Li
15
Future work
 Dynamic
control
• Real time on-the-fly adjustment to studio
equipments
 Semantic
indexing of video streams
• Express interest and query high level events
 Continuity Management
• Allows checks for continuity in different footage
March 10, 2005
Herman Li
16
References






Augmenting Film and Video Footage with Sensor Data, Norman
Makoto Su, Heemin Park, Eric Bostrom, Jeff Burke, Mani B.
Srivastava, Deborah Estrin
ARS: http://www.ee.ucla.edu/~hmpark/ars
PALOS:
http://deerhound.ats.ucla.edu:7777/pls/portal/docs/PAGE/CENS_REP
OSITORIES/TECHNOLOGIES/PALOS-TUTORIAL.PDF
http://mmsl.cs.ucla.edu/sylph
http://www.fxguide.com
http://computer.howstuffworks.com/mote.htm
March 10, 2005
Herman Li
17