Transcript Python programming and image matting solutions

NATURAL IMAGE
MATTING
(Implementing Closed-form alpha matte
solution as a python package)
BTech. Project, 2015-16 I Sem
Aniruddh Vyas
CSE Dept, IIT Kanpur
Aim:
 A beginning step to envision python library functions as a goto tool for image manipulations and image enhancement
purposes.
 Discuss natural image matting solutions (alpha matte
extraction) and their implementations and embedding
through Python Image Library.
 Performance comparisons with other image editing
frameworks like MATLAB etc. and how switching to more
broader environment is a step up.
 Pushing the domain for large scale applications of applied
imagery.
Theory (Image Matting):
 Matting refers to the problem of accurate foreground
estimation in images and video. This task of estimation of a
fore- and background layer from a single image is called
Image Matting.
 For estimation from a single color measurement, an accurate
modeling of the scene’s appearance is necessary.
 This problem was mathematically established by Porter and
Duff in 1984. They introduced the alpha channel as the
means to control the linear interpolation of foreground and
background colors.
Theory (Image Matting).. :
 Iz (z = (x, y)) is modeled as a convex combination of
foreground image Fz and background image Bz by using the
alpha matte αz:
Iz = αzFz + (1 − αz)Bz
where αz can be any value in [0,1].
 If αz = 1 or 0, the pixel z is called definite foreground or
definite background, respectively. It is called mixed for all
other scenarios.
Theory (Image Matting).. :
 Assuming it is represented in some 3D color space, thus the
unknown variables are the three dimensional color vectors Fz
and Bz, and the scalar alpha value αz.
 Matte extraction is thus inherently an under-constrained
problem, since 7 unknown variables need to be solved from 3
known values.
 Most matting approaches rely on user guidance and prior
assumptions on image statistics to constrain the problem to
obtain good estimates of the unknown variables.
 Once estimated correctly, the foreground can be seamlessly
composed onto a new background, by simply replacing the
original background B with a new background image B.
Closed-form Solution:
 The closed-form solution referred derives a cost function from
local smoothness assumptions on foreground and background
colors F and B, and analytically eliminates F and B, yielding a
quadratic cost function in α.
 The alpha matte produced is the global optimum of this cost
function, which may be obtained by solving a sparse linear
system.
 This approach computes α directly and without requiring reliable
estimates for F and B, a surprisingly small amount of user input
(such as a sparse set of scribbles) is often sufficient for
extracting a high quality matte.
Python Implementation:
 A module embodying a program of instructions executable by the
machine to perform a method for matting a foreground object F having
an opacity α in an image having a background B, the respective opacity
of selected pixels in the foreground object F and the background B
being constrained by associating a characteristic with said pixels, the
method comprising:
(a) determining weights for all edges of neighboring pixels for the
image;
(b) build a Laplacian matrix L with the weights;
(c) solve the equation α where α=arg min αT Lα s.t. αi=si, ∀i ∈ S, S is the
group of selected pixels, and si is the value indicated by said
characteristic;
 This work addresses basic image manipulation and
processing using the core scientific modules NumPy and
SciPy to achieve same matte results with better performance.
In particular, the submodule scipy.ndimage provides
functions operating on n-dimensional NumPy arrays.
 >>> from scipy import misc
 >>> lena = misc.imread('xyz.png')
 >>> type(xyz)
<type 'numpy.ndarray'>
 >>> xyz.shape, xyz.dtype
((512, 512), dtype('uint8'))
Python Image Ops















Opening and writing to image files
Displaying images
Basic manipulations
Statistical information
Geometrical transformations
Image filtering
Blurring/smoothing
Sharpening
De-noising
Mathematical morphology(erosion, dilation)
Feature extraction
Edge detection
Segmentation
Measuring objects properties: (ndimage.measurements)
We can add Image Matting operation here as a new and totally independent
feature.
Python v MATLAB for image manip
 The concept of Matlab refers to the whole package, including
the IDE. The standard library does not contain as much
generic programming functionality. In fact, processing big
images using the Matlab solver is impossible due to memory
limitations.
 Tasks such as to import images (JPG format), extract
numerical data (pixel brightness), and compare to various
theories, Python had more than enough to offer.
 With its extensive standard Library aimed at programming in
general also contains modules for specific functionalities.
 And with contributions to the development by reporting
issues, helping with documentation, and making small
improvements to the code, additional packages serving
unique functionalities can be easily added.
 The proprietary nature of algorithms also makes it
difficult/impossible for 3rd parties to extend the functionality
of Matlab. Mathworks puts restrictions on code portability.
 Indexing is done with braces rather than brackets, making it
difficult to distinguish it from a function call.
 Python avails build-in library that allows one to call function in a
dynamic library. This can be a very effective means to communicate
with existing code. No compilation required.
 Functions and classes can be defined anywhere. In one file (whether it
is a module or a script) one can design as many functions and classes
as one likes.
 Python was created to be a generic language that is easy to read, while
Matlab started as a matrix manipulation package to which they added a
programming language. Python is more powerful. Because it’s so well
designed, it’s easier than other languages to transform your ideas into
code.
 Image Matting or any other image handling methodology for that matter
requires an interactive environment. It means that the user does not
have to go through the compile-run-debug cycle every time he makes a
change to the code. Instead, a piece of code (a part of an algorithm, for
example) can be repeatedly changed and executed in the same
interpreter. This shortens the development cycle significantly (rapid
prototyping).
 So instead of creating multiple .m files, python, as opposed to MATLAB,
helps to reduce the size of the code considerably and other modules
from library packages like numpy and scipy greatly reduce further
redundant instructions.
Conclusions:
 High quality mattes can be obtained on natural images from a
surprisingly small amount of user input.
 Results of closed form solution for extracting alpha mattes by
minimizing the matting Laplacian subject to the scribbled constraints.
 Python provides as an excellent tool for all heavy operations in image
processing and can be commercially applicable saving time and money.
 Future work entails the need of closed form solvers for the mattes of
image patches that have complex intensity variation.
 It would be interesting to extend this formulation to include additional
assumptions. The goal is to incorporate more sophisticated models of
foreground and background but still obtain high quality results using
simple numerical linear algebra.
 This graph shows the number of lines of code in some
important projects for bioimage informatics/science in
general. As evident, Python has caught up with Matlab and is
in the process of overtaking it.
 The figure below shows matte extracted by using the tool on
a random colored image.
Original Image
Scribbles drawn, white for
foreground and black for
background
(Alpha Matte extracted)
 Furthermore, by looking at the smallest eigenvectors of the
matting Laplacian (not calculated here) can guide the user
where to place scribbles which leads to more accurate alpha
mattes.
 The resulting cost function is similar to cost functions
obtained in spectral methods to image segmentation but with
a novel affinity function that is derived from the formulation of
the matting problem. Experiments demonstrate that these
results are competitive with those obtained by much more
complicated, nonlinear, cost functions.
Some Limitations:
 Implementing matte extraction for multi resolutions and solving
Foreground and Background colors is merely an additional task more
than a limitation but to take it one step at a time its not been made a
part of this project work.
 There are also cases where this method fails to produce an accurate
matte since the method does not make use of global color models for F
and B and at times can’t handle ambiguous situations.
 For-loops in python are interpreted and thus much slower than for-loops
in C or C++. Though, writing a program in C/C++ rather than Python
would often cost a considerable amount more time.
Applications:
 Image matting is one of the key techniques in many image editing and film production
applications, thus has been extensively studied in the literature. Film making is the most
extensive and large scale application. CGI and green screen methods and digital
matting are the most popular methods to add visual effects in a film and are all based
on the enhanced algorithms of matte extraction.
 Python being an appealing alternative, makes it so suitable for all kinds of scientific
computing. Scientists should be able to share algorithms, and to verify the results of
other scientist without having to pay for a certain piece of software. Python is open. This
openness also makes Python easy to extend. Technologies such as GPU computing
and parallel processing are easily made available via Python.
 Interactive segmentation of objects in photos using alpha matting technique has major
scope. This task, also known as 'intelligent scissor' works in particular for semitransparent objects and fuzzy borders.
 Representing images as NumPy arrays is not only computational and resource efficient,
but many other image processing and machine learning libraries use NumPy array
representations as well. Furthermore, by using NumPy’s built-in high-level mathematical
functions, we can quickly perform numerical analysis on an image. SciPy adds further
support for scientific and technical computing. One of the widely used sub-packages of
SciPy is the spatial package which includes a vast amount of distance functions and a
kd-tree implementation.
References:
 Levin, D. Lischinski, and Y. Weiss. A closed-form solution to natural image
matting. IEEETransactions on Pattern Analysis and Machine Intelligence,
30(1):228–242, 2007.
 Standard python library documentation
Thank You !