Transcript 開啟檔案

Lips-Control Assistive System
Presenter : Deng-Shing Yang
Advisor: Dr. Shih-Chung Chen
Date: 2012.11.14
1
• Motivation
OUTLINE
• Introduction
• Paper Reviews
• Material and Methods
• Discussion
• Future works
• References
2
MOTIVATION
 Many computers’ input devices are designed for
normal persons, so these devices are unsuitable
for the disabled probably.
 Many researchers develop many auxiliary devices
for the disabled, but these auxiliary devices still
have many defects when the disabled use them in
real life.
 Efficiency and recognition rate are very important
factors of image processing algorithm of the
communication system based on image processing.
3
INTRODUCTION
․Essential property
I. Adaptability
•
•
•
•
Face Size
Face Posture
Background Complexity
Light Intensity Variation
II. Efficiency
• Size of Input Image
• PC Clock
• The Efficiency of Algorithm
4
OBJECTIVES
 We hope to implement a set of lips-control assistive
communication system for the person with cerebral palsy
by LabVIEW. The disabled can use the assistive
communication system without wearing any auxiliary
devices.
 To improve the efficiency and recognition rate of image
processing algorithm.
 To integrate the Human-Machine interface software
system with the hardware system to accomplish the
home appliance control system and the so called
McTin system.
5
SCHEMATIC DIAGRAM OF SYSTEM STRUCTURE
• Schematic Diagram of System Structure
(System
(SystemA)
B)
( Image processing algorithm )
user
USB
Webcam
Image processing algorithm
(face tracking, lips recognition…)
Parallel port
CCD
morse code
PS2/RS232/
USB
McTin
Morse code
algorithm System
utilizing
fuzzy theory
Calling API functions
(keybd_event mouse_event)
6
PAPER REVIEW
Ming-Hsuan Yang, Member, IEEE, David J. Kriegman, Senior Member, IEEE, and Narendra
Ahuja, Fellow, IEEE, “Detecting Faces in Images: A Survey,” IEEE TRANSACTIONS ON
PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 24, NO. 1, JANUARY 2002
7
SOFTWARE SYSTEM ARCHITECTURE
․ Block Diagram
Stage 1
Stage 2
Stage 3
Stage 4
8
FACE DETECTION AND TRACKING ALGORITHM
․ Architecture
RGB Images
Input
Color Space
transformation
Saturation
Convert the
Extracted Skin
Color Area to
a Binary Image
Face Verification
and Tracking
Hue
Convert the Extracted
Non-Skin Color Area to
A Binary Image
Morphology
CLOSE Operation
Lips Template
Input
Morphology
CLOSE Operation
Template
Matching
Morphology
OPEN Operation
Inverse
Binary Image
Coordinates of
Lips Images
AND Operation
No
Morphology
OPEN Operation
Lips Image in the Skin
Color Area
Particles Area
Analysis
Skin Color
Detection
Morphology
Convex Hull
Operation
Reducing
Image Size
Yes
Drawing a
Boundary
Rectangle
Boundary
Rectangle
Cancellation
Coordinates of the
Skin Color Area
Stage 3
9
ALLOCATION AND EXTRACTION OF LIPS
․ Choice
of Multiple Templates of Lips Area
IMAGES
․Target- The wrong allocation must be avoided
․Step1- Calculation of rectangle boundary parameters
R1(x,y)
Face Range
C ri
The Center Coordinate
Cri (x, y)
of Rectangle Boundary
The Center Coordinate of Rectangle Boundary :
 R 2(x )  R 1(x ) R 2(y )  R 1(y ) 
C ri (x , y )  
,

2
2


R2(x,y)
• Step2- Allocation and extraction of lips Images
Lip area:
Length= x ± 30, Wide: y ± 20 (Pixels) ……(1)
Lip’s location in face area:
High= y + 30, Wide= x (Pixels) …………..(2)
10
PROCESSING AND RECOGNITION OF LIPS IMAGES
․ Architecture
Lips Images
Input
RGB MultiThreshold
(dark)
Taking particle
with
maximum area
No
Morphology
CLOSE
Operation
Particles Area
Analysis
Morphology
DILATE
Operation
Greater Than
the Threshold ?
Yes
Logical Signal
“0”Output
Logical Signal
“1”Output
Lips Images
Recognition
Lips Images
Processing
11
Windows API
 What is Windows API?
• The Microsoft Windows application programming interface
(API) provides building blocks used by applications
written for Windows .
• You can provide your application with a graphical user
interface; display graphics and formatted text; and manage
system objects such as memory, files, and processes.
17
Color Space Transformation (RGB
HSL)
 Experimental Results of Face Photo in Lab
․Transformation Formula
1

R  G   R  B  

1 
2
H  cos 
1
 R  G  2  ( R  B ) (G  B ) 2 




3
min( R, G, B)
S  1
( R  G  B)
RG B
L
3
Input Image(RGB)
H: Hue
S: Saturation
L: Luminance
Hue
Saturation
Luminance
13
Skin Color Detection-Binarization of
H S Panel (1)
 Experimental Results of Face Photo in Lab
․Decision Formula
 255
Skin Color j  
0
if H d  H  H u & S d  S  S u
Otherwise
Binary Image and Threshold Values
Range of the Hue Panel
Skin Color j 
Binary Images of the Skin Color
Objects
H d , H u , S d , S u - Threshold Values of the Hue
and the Saturation Panel
Binary Image and Threshold Values
Range of the Saturation Panel
14
SKIN COLOR DETECTIONMORPHOLOGY CLOSE OPERATION
 Target- To fill background with non-skin color area
․ Formula
S R  I  DS ES
1 1 1


E S  D S  1 1 1
1 1 1
 - Erosion Operator
 - Dilation Operator
S R - Skin color image area after ‘CLOSE’ operation
I - Original image
E S , D S  Structure element of the erosion and dilation operation
․ Experimental Results

Original Binary Image
of the Hue Panel

Binary Image
of the Hue Panel
After Dilation Operation
Binary Image
of the Hue Panel
After Erosion Operation
15
SKIN COLOR DETECTIONMORPHOLOGY OPEN OPERATION
 Target- Expanding the Face Area
․ Formula
S R  I  ES  DS
1 1 1


E S  D S  1 1 1
1 1 1
 - Erosion Operator
 - Dilation Operator
S R - Skin color image area after ‘OPEN’ operation
I - Original image
E S , D S  Structure element of the erosion and dilation operation
․ Experimental Results


Binary Image
of the Hue Panel
After Close Operation
Binary Image
of the Hue Panel
After Erosion Operation
Binary Image
of the Hue Panel
After Dilation Operation
16
Skin Color Detection-Binarization of
H S Panel (2)
 Experimental Results of Face Photo in Lab
․ Obtaining the Skin-Color Area
OPEN
CLOSE
Binary Image of the Hue Panel
CLOSE
Binary Image of the Saturation Panel
17
Skin Color Detection-Binarization of
H S Panel (2)
 Experimental Results of Face Photo in Lab
․ Obtaining the Skin-Color Area
inverse
Logical
Difference
open
Binary Image of the
Hue Panel
Binary Image of the
Saturation Panel
Skin Color Area Image
18
SKIN COLOR DETECTIONSKIN COLOR PARTICLES AREA ANALYSIS
 Target- Eliminating the bigger particles
• Algorithm
Ab 0  min( a11 skin color particles area )
Abi  2  Abi 1
Ab 0 The smallest skin color particle area(Pixels)
Abi  i’th area base value after the skin color
particles area analysis
 Experimental Results
2nd Iteration
Binary Image after OPEN Operation
3nd Iteration
10th Iteration
19
SKIN COLOR DETECTIONMORPHOLOGY CONVEX HULL OPERATION
 Experimental Result
Skin Color Area Image After
Particles Area Analysis
The Result After Convex
Hull Operation
20
EXPERIMENTAL RESULTS AND ALGORITHMS VERIFICATION
 Result of Software System
․ The Software System Characteristics
 Lips Image Area Adjustable
 Renewable Lips Template Image
 Display Windows Management
 Images Browser
System Status & Message Display
21
CONCLUSIONS
 We utilize the Open/Close status of lips images successfully
to real time control auxiliary devices for the disabled.
 To decrease the interference of face detection because of
light variation. Therefore, we can detect face more effectively.
 Open/Close status of lips can be detected more stably .
 System complexity and cost can be reduced because of our
new software architecture developed.
 System efficiency can be improved by updating hardware
of computer.
22
FUTURE WORKS
 先把這套程式使它可以正確地執行在Lab VIEW2012
 改良人臉辦識演算法
 增加特徵方向判斷的機制 (To control mouse)
 更準確的定位出嘴唇的位置
 結合更多類型的殘障輔具
 人類臉部表情辨識應用
23
REFERENCES (1)
[1] Ming-Hsuan Yang, Member, IEEE, David J. Kriegman, Senior Member, IEEE, and Narendra Ahuja,
Fellow, IEEE, “Detecting Faces in Images: A Survey,” IEEE TRANSACTIONS ON PATTERN
ANALYSIS AND MACHINE INTELLIGENCE, VOL. 24, NO. 1, JANUARY 2002
[2] Yuemin Li, Jie Chen, Wen Gao, Baocai Yin, “Face Detection: a Survey,” 2004
[3] Vladimir Vezhnevets, Vassili Sazonov, Alla Andreeva, “A survey on pixel-based skin color detection
techniques,” Graphicon-2003, Moscow, Russia, September 2003.
[4] IUT Informatique de Bayonne, Univ. de Pau et des Pays de l'Adour, Bayonne .,“Real Time Tracking for
3D Realistic Lip Animation,” Proceedings of the 18th International Conference on Pattern
Recognition (ICPR'06), 2006
[5] Ching-Hsing Luo, Chung-Min Wu, Shu-Wen Lin, Tsan-Hsun Huang, Cheng-Hong Yang, Ming-Che
Hsieh, Shih-Chung Chen and Chih-Kuo Liang, “Mouth-Controlled Text Input Device with Sliding
Fuzzy Algorithm for Individuals with Disabilities”, IEEE instruement and measurement 2005
(submitted).
[ 6] “ 認識fuzzy-第二版,” 王文俊, 全華科技, 2001
[7] LabVIEWTM PID Control Toolset User Manual
[8] LabVIEWTM Fuzzy Logic for G Toolkit Reference Manual
[9] “LabVIEW & Microsoft 的整合實例(I),” 陸光中/蕭子健, 高立圖書, 2004
[10] LabVIEWTM Using External Code in LabVIEW User Manual
24
REFERENCES (2)
[11] “應用於重度脊髓損傷患者之摩斯碼模糊辨識嘴控輸入系統,” 國立成功大學電機系, 吳崇民, 博士論文, 2004
[12] 吳瑞珍，“人臉特徵自動抽取之演算法設計與應用”，私立元智大學電機工程研究所碩士論文(民91年6月)
[13] 沈韋穎，“即時人臉偵測系統”，國立台灣大學資訊工程學系碩士論文(民92年6月)
[14] 黃泰祥，“具備人臉追蹤與辨識功能的一個智慧型數位監視系統”，中原大學電子工程學系碩士論文(民93年
6月)
[15] http://labview360.com/Default.asp labview討論區
[16] http://msdn.microsoft.com/ msdn
25
THANKS FOR YOUR ATTENTION!
26