Transcript On_the_Use_of__Computable_Features_for_Film_Classification

On the Use of Computable
Features for Film
Classification
Zeeshan Rasheed,Yaser Sheikh
Mubarak Shah
IEEE TRANSCATION ON CIRCUITS AND SYSTEMS
FOR VIDEO TECHNOLOGY, JAN 2005
Outline
• Introduction
• Computable video features
–
–
–
–
Average Shot Length
Color Variance
Motion Content
Lighting Key
• Mean Shift Classification
• Results
• Conclusion
Introduction
• Films are a means of expression
– Explicitly, with the delivery of lines by actors
– Implicitly, with the background music, lighting,
camera movements and so on
• Study domain is the movie preview, which
often emphasizes the theme of a film
Introduction
• Maybe a need to extract the “genre” of
scenes
• With scene-level classification, it would
allow a more flexible system of scene
ratings, ex filter and recommendation of
movies
Related Work
• Work by Fischer et al. and Truong et al.
distinguished between newscasts, cartoon,
commercials, sports through decision tree
with examples
• Kobla et.al used DCT coefficients, and
motion vector information of MPEG video
for indexing and retrieval
Computable video features
• Identify four major genres
– Action, Comedy, Horror, and Drama
– because most movies can be classified and
low-level discriminant analysis is most likely to
succeed
• Employ for features
– Average Shot Length, Shot Motion Content,
Lighting Key, Color Variance
Shot Detection and Average Shot Length
• First proposed by Vasconcelos
• Can direct audience’s attention with
controling the tempo of the scene
• Ex. Dramas have larger average length,
whereas action movies shorter shot length
Shot Detection and Average Shot Length
• Detection of shot boundaries using color
histogram intersection in the HSV space
– H: hue (color), 8 bins
– S: saturation, 4 bins
– V: value (brightness), 4 bins
• S(i) represent the intersection of histograms
H and H
of frames i and i-1
i
i 1
Shot Detection and Average Shot Length
min
bin1 bin2
Frame i
bin1
bin2
Frame i-1
Shot Detection and Average Shot Length
min
bin1 bin2
bin1
bin2
When S(i) is less than a fixed threshold
Frame i
Frame i-1
shot boundaries !!
Shot Detection and Average Shot Length
17 shots identified by a human observer
Number of shots detected: 40; Correct: 15;
False positive: 25; False negative: 2
Shot Detection and Average Shot Length
To
improve
accuracy,
an18;
iterative
smoothing
of the 1-D function
Number
of the
shots
detected:
Correct:
16;
isFalse
performed
first2; False negative: 1
positive:
Color Variance
• variance of color has a strong correlational
structure with genres intuitively
– For instance, comedies with a large variety of
bright colors
– whereas horror films with only darker hues
• Employ variance of CIE Luv
– L: luminancy (發光度)
– u,v: chrominancy (色差)
Color Variance
Generalized variance is obtain
ps. All key frames presented in a preview are used to find this feature
Motion Content
• The visual disturbance of a scene can be
represented as the motion content present
• Action films with higher value for such a
measure, and dramatic or romantic movies
with less visual disturbance
Motion Content
Horizontal slice: I(x,t)
Motion Content
Hx, Ht are the partial derivatives of I(x,t)
Lighting Key
• There are numerous ways to illuminate a
scene, one of the common used is Three
Point Lighting
Fill-light:
Keylight:
Backlight:
Secondary illumination
The main
Help
emphasize
sourcethe
of light
source which helps to
on the subject
contour
of the object,
and it isand
soften some of the
itthe
also
source
separates
of greatest
it from a
shadows thrown by the
illumination
dark
background
keylight and backlight
Lighting Key
• High-key lighting:
– An abundance of bright light
– More action, less dramatic
– Ex. Comedy & action movies
• Low-key lighting:
– Ex. Film noir or horror films
Lighting Key
• Many algorithms exist that compute the
position of a light source in a given image
• Unfortunately, assumptions typically made
in existing algorithms are violated, for
example, single light source
• Compute the key of the lighting with
brightness value of pixels
Lighting Key







Lighting Key
• Key frame i with m*n pixels, find the mean 
and standard deviation  of the value
component of the HSV space
• Lighting quantity  i (  ,  )  i   i
– Horror movies with small value
– Comedy movies with large value
Mean Shift Classification
• Mean shift procedure has been shown to
have excellent properties for clustering
and mode-detection with real data
Xi: video features
hi: their bandwidth parameters
Action +
drama
drama
Comedy+
drama
comedy
Action+
comedy
horror
Results
• Conduct 101 film previews obtained from
the Apple website
• The total number of outliers in the final
classification was 17 and 83% genre
classification accurate
Conclusions
• Propose a method to perform genre
classification of previews using low-level
computable features
• Classification is performed using mean
shift clustering in the 4-D feature space of
average shot length, color variance,
motion content, and the lighting key