Transcript USING THE S60 MOBILE PHONE PLATFORM FOR REMOTE

THESIS COMPETITION
USING THE S60 MOBILE PHONE
PLATFORM FOR REMOTE
MONITORING WITH IMAGE
PROCESSING AND CAMERA
CONTROL
By
Kittinut Khumkiatkul
Content
1. Introduction
2. Local server module
3. Mobile Client module
4. Experiment and results
5. Conclusion
1.1 Background
The surveillance system is to transmit the event (signal) from the
local to remote location for the monitoring purpose. To effectively
monitor the event, there are two importance points to be
concerned:
1. Ability to automatically classify the object in the frame :
In practical use, manual classification is susceptible for the human
error. The image processing is needed to help the human for
classification task
2. Accessibility of the system beyond the internet :
Frequently,
the users stays where the internet is unavailable, so they can not
access the monitoring system. The mobile device is needed to
strengthen system accessibility
1.1 Background (Cont.)
This research is proposed to serve these requirement.
• To do classification work, the local server is installed in the
local site for image processing tasks.
• The accessibility of the system can be increased by using
Symbian mobile client with the API
Network camera
Internet
LAN
Symbian mobile
client
Manual
monitoring
Web browser
Local server
1.2 Concept of study
Main component
1.
Network
camera
2.
Local server
3.
Web server
4.
Web browser
5.
Symbian
Mobile client
1.3 Objective
To enable mobile client communicate with local server
and network camera and do the classification work,
three items have to be done
•
Construct the local server for image processing project
in MATLAB
•
Construct the web server for data transferring
between local components and mobile module
•
Build an API for S60 mobile client to communicate with
the local component via EDGE
2. Local Server module
2.1
General concept
2.2
Network Camera System
2.3
Human Detection Project
2.4
Adaptive background to environment
2. General concept: Target
•
Implement network camera system, comprising of
image flow, image database, web browser and stream
•
Develop the program to detect the object in the image
and classify whether it is the human or not
•
Develop the program to generate alarm when the object
is classified to human
2.1 General concept: Architecture
There are three major
parts in the system
1
Send the image to FTP
Network camera
• Network camera
TCP/IP
• Web server
Web Server
• Local server
4
Store image to web server
and its storage location
TCP/IP
2
Get the image from FTP
Local server
3
Process image
2.2 Network camera system
Setting items
The storage location:
ftp://stweb.ait.ac.th/www
Frequency of taking photo:
Every 2 seconds
Image size:
160 x 120 pixels
Web browser
2.3 Human Detection
Project
Start
1) Acquire Image from web server
2) Pre-process image
No
Is there any
change?
Yes
3) Feature Extraction
4) Analyze the feature and Classify
5) Store image on database and FTP
Finish loop ?
Yes
End
No
1) Image acquisition
•
The Human Detection program on local server acquires
image from web server (via its ftp)
•
Since the image name is time-related name (ISEYYYYMMDD-HHMMSS.jpg), the Existence test method is
used to guess it to acquire image
- The current time: 20080514-103019
- Test to load image name by using “ISE-20080514-103019.jpg”
- Increase 1 sec. if that image does not exist.
•
By using the web server as the image database, the
system is free from setup when the IP of local server
changes
2) Image pre-processing
This process is to adjust the image to the suitable format.
There are two interesting steps
1. Color-integrated segmentation: It is to segment the
object from to background.
2. Morphological operation: It is to improve the quality
of the object shape, compactness and filtering the
small particle objects.
2.1) Color-integrated Segmentation technique
This is the upgrade version of Background subtraction technique (prior
method) by means of YIQ color model to increase the robustness for
lighting and complicated background
Background subtraction
technique
Background
Current image
Color-integrated
segmentation technique
Algorithm comparison: Prior Technique
Changes in luminance and the reflection generate many
disturbances to segmentation. They cause background more
complex since they are the part of value in R, G and B
component. The luminance should be separated from color.
R1,G1,
B1
Start
Background – The current
image
Convert to binary image
End
RGB model
R1 = R1actual + LR1
G1 = G1actual + LG1
B1 = B1actual + LB1
Background
subtraction technique
YIQ model
Y1 = Y1actual + LY1
I1 = I1actual
Q1 = Q1actual
Lighting effect on
that component
Y=luminance
I, Q = mixture of color
Algorithm comparison: Upgraded Technique
Color-integrated
segmentation technique
Start
Convert to gray scale
image
Compare background to
the current image pixel
The
Background
subtraction
technique
yes
Background – The current
image
yes
Convert to binary image
End
The current image Background
yes
That pixel is background:
Set it to zero
BG is Significantly
higher than CI
no
CI is Significantly
higher than BG
no
no
Same color (same I
and Q: YIQ)
2.2) Morphological operation
The object in segmented image is full with the damages, holes
and irrelevant objects. Therefore, this process is used to
improve the object quality for the further process
Morphological operation
(Dilation and Erosion)
• Dilate to repair the
damage parts
• Filter the irrelevant
objects
• Fill the holes
Segmented image
Pre-processed image
3) Feature extraction
The unique differences between human and non-human,
called Feature, are used to classify the object. Three features
that have the consistency for all condition (gender, race, etc.)
are selected
•
Geometrical feature
•
Object Skeleton
•
Skin pixel extraction
3.1) Geometrical feature
Shape
and to
sizethe
of research
the objectassumption
are extracted
as the
Regarding
thatand
the used
position
of
one
of is
criteria
of classification.
The geometric
are
camera
in range
0.5 - 2 meters,
there are grids
2 unique
applied
to count
pixel,
by line to define the width
characteristics
of the
human
as passed
the following:
and height of the object
• Human size is always larger than 12%: The experiment show
Shape: ratio between
that the
Size: Amount
of theminimum percentage when sample human are in the
the longest line and
white pixels
in
the
frame at 5 meter distance from the camera.
shortest line in both
image
ver. And
hor.
• Human shape is always triangle or vertical rectangular
shape.
The horizontal grid is
used to indicate the
width in a pixel
Width: Amount of
pixels, passed by
horizontal grid
Height: Amount of
pixels, passed by
Vertical grid
The vertical grid is used
to indicate the height in
a pixel
Sample pattern
Triangle
Segmented image
withshape
geometric grid
Vertical rectangular shape
3.2) 4-line Skeleton Recognition method
This feature shows the angle of object axis that human always
has the vertical skeleton due to the direct backbone, but those
of non-human object does not
The 2nd
longest line
The longest line
Preprocessed image
The skeleton of human
The extracted line of skeleton
• The object is applied Zhang-Suen skeletonization method to
extract the skeleton line
• Four longest lines are extracted from the skeleton using
Hough transform and create table of the case of human and
non-human
3.3) Skin pixel extraction
This skin color is the major feature since the skin color
properties is rarely shown in non-human object. To extract the
skin, it is to classify pixel using these process:
The distance equation:
• Create color database (r, g, b) that is collected from the
2
2
 (r p  rhas
gp  5
g s )separated
 (b p  bs ) 2 group (2 for skin and 3 for
sampleDimage
s )  (the
R
G
B
background)
r
,g 
,b 
RGB
RGB
RGB
1.0
BG3
• Each pixel is checked with mean of each group for the
•distance
Pixel is of
thecolor
member
of the group
r
component
Skin1
that has the smallest distance
Skin2
between that pixel to its mean value
BG2
b
b
This pixel is the member of
Skin1 group
1.0
BG1
1.0
2.4 Adaptive background to environment
There are 3 major reasons to change background
•
There is rotating command from mobile phone
•
Time reaches 30 minutes
•
Luminance changes
2.4 Adaptive Background: Lighting changes
There
The
Current
is a change
Background
signal
3. Mobile Client Module
3.1
General concept
3.2
Web Server
3.3
S60 Mobile Application
3.1 General concept: Architecture
There are three major parts in the system
Playback stream
Control camera
S60 Client
HTTPS
HTTPS
View the processed image
http://stweb.ait.ac.
th/~st105000
Web Server
ftp://stweb.ait.ac.th/WW
W
Network
camera
TCP/IP
TCP/IP
Client
browser
Mobile client Module
Local server
Local Server Module
3.2 Web server
1.
2.
•
•
•
Is addressed at http://stweb.ait.ac.th/~105000
Permits mobile phone to communicate with local site
Connect via HTTP and GPRS
Get the image from its database
Issue the camera control command
3. Send SMS automatically to user’s mobile client
3.2 Web server Programs: Main PHP script
The PHP script, server-side script, works on the web server
when it is request from client to do some action such as
redirecting to url, posting message, and others There are 3
main PHP scripts.
1.

Rotating command: Control the direction of the camera movement
by redirecting to the url of camera controlling botton and save
the log to web server
up.php, down.php, right.php, left.php are the code that used for
each direction
2.
Search image: This is used to search the image from its name,
3.
SendSMS: Send the automatically to user’s mobile client
ISE-YYYYMMDD-HHMMSS to allow mobile phone getting the most
recent image without local server
3.2 Web server: SMS Configuration
MOBILE CLIENT
SMS
GSM
PROVIDER
SMS GATEWAY
seachimage.php
or
direction.php
WEB SERVER
LOCAL SERVER
Sendsms.php
3.3 S60 mobile Application
 HTTPClientExample is developed on S60 3rd edition
Maintenance Release platform
 The Carbide.C++ v1.2 is used as IDE
Load the recent
image
Function of API:





Load the recent image
Show the recent image
Playback the stream
Control the camera
Respond to SMS
Show the recent
image
Playback the
stream
Mobile client user
Control the
camera
3.3 S60 Mobile Application: Mobile Interaction
User
UI
Engine
HTTP API
Web Server
Netwrok
camera
PHP
1. Select action
2. Do action
3.
4. HTTP Get
a. Request PHP
c. result
d. result
5. HTTP response
6. Receive data
7. Result
8. Show result *
* In case of camera controlling function, go back to step 4th to get the recent image.
3.3 S60 Mobile Application: Class Interaction
Browser.exe
ClientApp
ClientApp
Document
SMSReceiver
ClientAppUi
ClientEngine
Client
Container
ClientImage
Engine
3.3 S60 Mobile Application: API function
•• Show
the recent
image
Playback
the stream
• Control the camera
Network
camera
rtsp://ipaddress:port/live.sdp
Request RTSP
Redirect to
2 direction url
1
Request direction.php
searchimage.php
url
Receive the
RTSP stream
a
3
b
a
Store
the
Send
thethe
most
Get
imagelog Acquire
rotating
recent
image
command
log
Web server
(http://stweb.ait.ac.th/
~st105000/recentim.jpg)
Mobile client
Local server
Receive image
the
2
4 Receive
data
result log
c
b
SaveChange
as recent
image
background
4.1 Image processing Results
Background
Current image
Pre-processed image
Geometric feature
Skin feature
Skeleton feature
4.1 Image processing: Human classification (1)
START
20%
> 12%
< 12%
Object Size
percentage
Non-Human
H1
NH1-NH8
H1-H5
No match
Skeleton
Case
17%
> 10 %
Next
Skin pixel
percentage
< 10%
Non-Human
Man
4.1 Image processing: Human classification (2)
Next
Triangular or
Vertical rectangular
shape
Object
Shape 1
Other
shapes
Human
Width > 2 and Height
< ½ of image
Others
Object
Shape 2
Man
Non-Human
4.2 S60 Mobile Application: Response to SMS
After detecting the human, local server sends the SMS to
user’s mobile client
Run application
Show image if
SMS match
Standby in
background
The incoming SMS
match to “Detect
Human !”
Receive new
message
5. Research Summary: Completed task
Local
Mobile
Server
Clientmodule
module
Create API for mobile client
Implement
camera sys
 Display thenetwork
recent image

the the
image
to FTP periodically
 Send
Playback
stream

accessed
by web browser
 Be
Control
the camera

controlled
from
mobile client
 Be
Respond
to the
emergency
SMS
Setup
web
serverdetection
and writeproject
script
Program
Human

the image
image flow
database
 Be
Design
chart for
andmobile
client
acquisition system

control command
to
 Redirect
Design segmentation
technique
camera
 network
Create and
collect color database

SMSfeature
to client
 Send
Selectemergency
the consistent
and
design algorithm to extract them
 Design the decision tree for
classification unknown object
Setup video database sys
5. Research summary: Challenge
1. Low communication bandwidth: Owing to the common
frame between stream and image, the smallest size is
chosen as the restricted bandwidth. Moreover, it delays
the program execution in some stage
 Image size – The resolution is reduced, so some image
detail is missing and it makes analysis technique more
complicated
5. Research Summary: Challenge (Cont.)
2. The communication between camera and web server is
not reliable. Sometimes, the image can not be stored
into ftp; moreover, it takes a several seconds for storing
during low connection. The effect was shown acquisition
process
 Image Acquisition – The image acquisition function, trial
& error technique, is needed to wait for the image. It
should be optimized. If the algorithm is too fast search
it has never found the image. If it is to slow, it will
waste the execution time
5. Research Summary: Challenge (Cont.)
3. Variety of the skin and cloth of human and background:
These affect to the segmentation and feature extraction
as the follow detail
 Segmentation – Some cloth that is nearly same as
background becomes the part of background and the
object is able to camouflage to the complex
background. It is essential to use the integrating of
object properties to solve it
 Skin pixel extraction – The cloth that is similar to skin
color can be extracted as the skin and some skin color
can not be extracted as the skin. The color database is
needed to be carefully collected
Thank you