Transcript Ⅰ. Current State and Application of Eye Gaze Tracking

Eye Tracking Techniques and
applications
eie426-EyeTracking-200903.ppt
2016/3/28
EIE426-AICV
1
Outline




Part I :Physiology of the EYE
Part II : Mechanism of Eye Movements
Part III :Eye Tracking Technology
Part IV :Eye Tracking Applications
2016/3/28
EIE426-AICV
2
How does our eye work
2016/3/28
EIE426-AICV
3
Part I :Physiology of the EYE
HOW DOES OUR EYE WORK
2016/3/28
EIE426-AICV
4
2016/3/28
EIE426-AICV
5
Part II : Mechanism of Eye Movements
CAN YOU BELIEVE YOUR EYE
2016/3/28
EIE426-AICV
6
Part III :Eye Tracking Technology
HOW EYE TRACKER WORKS
2016/3/28
EIE426-AICV
7
Outline
1.
Current State and Application of Eye Gaze Tracking
(EGT) Technology
2.
Theory and the Classification of EGT Technology
3.
The Framework of EGT system
Ⅰ. Current State and Application of
Eye Gaze Tracking (EGT) Technology
Ⅰ. Current State and Application of Eye Gaze Tracking (EGT) Technology
What is EGT
eye gaze——The line from the fovea through the center of
the pupil is the line of sight (LoS). Usually, we
take the optical axis as line of gaze (LoG). LoG
can be approximate to LoS. In fact, LoS
determines a person’s visual attention.
eye gaze Tracking——By image Processing,if the LoG or LoS
can be estimated, the point of regard (POR) is
computed as the intersection of the LoG (or LoS)
with the object of the scene or space.
 History and Development of EGT

Interests of Visual attention can be traced back to 1897. At that time, it was a kind of
diagnostic research, i.e. the recording of eye movement. Those technologies include
ophthalmometer, Mechanical Method, Electro-Oculography (EOG), Optical-based
Method , electromagnetic Oculography
eye movement initially applied in medical research, such as brain and physiology
analysis. with the development of Electronics, Computer and Image processing
technology, further research focuses on Eye Gaze Tracking Technology .
Search Coil
A picture wearing the EOG apparatus
Ⅰ. Current State and Application of Eye Gaze Tracking (EGT)
Technology

eye tracking techniques：
1.
Direct Observation： ophthalmometer, peep-hole method
2.
Mechanic Method：use level to record eye movement
3.
Optical-based Method ：Reflection Method （mirror or prism）、PupilCorneal reflection、Purkinje Image
4.
Electro-Oculography (EOG): recorded the difference in the skin potential
5.
Electromagnetic Oculography ：The users gaze is estimated from
measuring the voltage induced in the search coil by an external electro-magnetic
field
Except Optical-based Method, the above methods are more or less low accuracy or
high intrusiveness . As a result, Modern Eye Gaze Tracking Techniques are mostly
based on Optical-based Method.
Ⅱ. Theory and the Classification of
EGT Technology
Ⅱ. Theory and the Classification of EGT Technology

Image-Based EGT Technology
1.
Infrared Light : out of visible light; Paired with filter to eliminate the
light of other wavelength；
Eye camera :tracking eye movement and recording eye image
sequence
Image Processing: detecting the visual elements
Further Estimation:。Having pre-processed, the data of eye gaze
can be get from eye-movement model
2.
3.
4.
Ⅱ. Theory and the Classification of EGT Technology
System Framework
Input
Image
acquisition
Image
Processing
Feature
Estimation
Output
Data
Estimation
Eye
Tracking
Calibration
Ⅱ. Theory and the Classification of EGT Technology
Function of Each Module
1
Image acquisition:
2
Image Processing :
3
Feature Estimation:
4
Eye Tracking:
5
Data Estimation:
get bright pupil or dark pupil
Filtering ,Noise reducing ,difference ,thresholding
Estimating the center of Pupil and corneal reflection
Estimation of eye gaze
Algorithm validation ,eye movement data analysis
Theory of EGT Technology


2D to detect the Optical Axis (basic)
2D to detect the Line of Sight (Advanced)
¹â ÖáOA 2D·½·¨
ÊÓÏß Öá LOS 3D·½·¨
Classification of EGT Technology
view of
humanity
intrusive £º Contact lens¡¢ Photo-transmiter¡¢ IR light emitting diodes
non-intrusive £º use image- based Method
view of
application
Diagnostic£º Use the Eye movement characters to Psychology and Neuroscience
(need to distinguish different type of eye movment)
Interactive £º Use Eye Movment to decide Point of Regard ,use for interaction
£¨ need to know the visiual attention£©
Body-Mounted£º goggle¡¢ helmet¡¢ back pack, etc
note : although labeled with inrusive ,but more and more body-Mounted systems
are becoming non-intrusive
View of
appearance
Remote system£º Remote EYE Gaze Tracker (REGT)
Ⅱ. Theory and the Classification of EGT Technology
From View of Humanity
Usually By goggle , helmet …
Head
features：high accuracy
Mounted
Eye Tracker defects：intrusiveness
Usually on table
Remote
features：non-intrusive
Eye Tracker defects：eye and head
relative motion
Ⅱ. Theory and the Classification of EGT Technology
From View of Theory
Purkinje Image
EOG
Limbus-Scalar
IR Tracking
ANN
The Purkinje images are reflections created at different
layers of the eye structure. Eye gaze can be calculated
from these relative positions of these reflections
By placing electrodes around the eye, it is possible
to measure small differences in the skin potential
corresponding to eye movement.
The Limbus is the boundary between the white sclera and the dark ir
of the eye. By placing IR light emitting diodes and photo-transistors,
respectively, above and below the eye. the resulting voltage differenc
is proportional to the angular deviation of the eye.
Training images are taken when the user is looking at
a specific Calibration markers. Use ANN to decide
the eye gaze
Contact Lens
Use a small coil embedded into a contact lens that is tightly fit
over the sclera . The user’s gaze is estimated from measuring the
voltage induced in the search coil by an external electro-magnetic
field.
Pupil-Corneal
reflection
The IR source can generate a glint on corneal and
divide pupil from iris, the difference between can
represent the eye gaze movement
Ⅱ. Theory and the Classification of EGT Technology
From View of Theory
Purkinje Image
EOG
Limbus-Scalar
IR Tracking
ANN
The Purkinje
images
are reflections
created at different
The
Purkinje
images
are reflections
layers of the eye structure. Eye gaze can be calculated
created
different
layers
the eye
from these at
relative
positions
of theseof
reflections
structure. Eye gaze can be
将两对氯化银皮肤表面电极分别置于眼睛左右、上下两侧,
By placing electrodes around the eye, it is possible
calculated
from these relative
就能引起眼球变化方向上的微弱电信号,
经放大后得到眼
to measure small differences in the skin
potential
球运动的位置信息
positions
oftothese
reflections
corresponding
eye movement.
The Limbus is the boundary between the white sclera and the dark ir
The
is the boundary
between
theand
white
sclera
of theLimbus
eye. By placing
IR light emitting
diodes
photo-transistors,
respectively,
above
andthe
below
eye.boundary
the resulting
voltage
and
the dark
iris of
eye.the
This
can
easilydifferenc
is
proportional
to
the
angular
deviation
of
the
eye.
be optically detected and tracked
Training images are taken when the user is looking at
a specific Calibration markers. Use ANN to decide
the eye gaze
Contact Lens
Use a small coil embedded into a contact lens that is tightly fit
over the sclera . The user’s gaze is estimated from measuring the
voltage induced in the search coil by an external electro-magnetic
field.
Pupil-Corneal
reflection
The IR source can generate a glint on corneal and
divide pupil from iris, the difference between can
represent the eye gaze movement
Ⅱ. Theory and the Classification of EGT Technology
From View of Theory
Purkinje Image
EOG
Limbus-Scalar
IR Tracking
ANN
The Purkinje images are reflections created at different
layers of the eye structure. Eye gaze can be calculated
from these relative positions of these reflections
By placing
electrodes
around the
eye, it isthe
possible
By
placing
electrodes
around
to measure small differences in the skin potential
eye,
it is possible
to measure small
corresponding
to eye movement.
differences in the skin potential
将两对氯化银皮肤表面电极分别置于眼睛左右、上下两侧,
The
Limbus is the boundary
between
the white sclera
corresponding
to eye
movement.
就能引起眼球变化方向上的微弱电信号,
经放大后得到眼
and the dark iris of the eye. This boundary can easily
球运动的位置信息
be
optically detected and tracked
Training images are taken when the user is looking at
a specific Calibration markers. Use ANN to decide
the eye gaze
Contact Lens
Use a small coil embedded into a contact lens that is tightly fit
over the sclera . The user’s gaze is estimated from measuring the
voltage induced in the search coil by an external electro-magnetic
field.
Pupil-Corneal
reflection
The IR source can generate a glint on corneal and
divide pupil from iris, the difference between can
represent the eye gaze movement
Ⅱ. Theory and the Classification of EGT Technology
From View of Theory
Purkinje Image
EOG
Limbus-Scalar
IR Tracking
ANN
The Purkinje images are reflections created at different
layers of the eye structure. Eye gaze can be calculated
from these relative positions of these reflections
By placing electrodes around the eye, it is possible
to measure small differences in the skin potential
corresponding to eye movement.
The
Limbusisisthe
the
boundary
between
the white
The Limbus
boundary
between
the white
sclera
sclera
theboundary
eye. By can
placing
and theand
darkthe
iris dark
of theiris
eye.ofThis
easilyIR
be optically
detected
and
tracked
light
emitting
diodes
and
photo-transistors,
respectively,
above and below the eye. the
将两对氯化银皮肤表面电极分别置于眼睛左右、上下两侧,
Training images
aredifference
taken whenis
theproportional
user is looking
resulting
voltage
toat
就能引起眼球变化方向上的微弱电信号,
经放大后得到眼球
a specific Calibration markers. Use ANN to decide
the
angular deviation of the eye.
运动的位置信息
the eye gaze
Contact Lens
Use a small coil embedded into a contact lens that is tightly fit
over the sclera . The user’s gaze is estimated from measuring the
voltage induced in the search coil by an external electro-magnetic
field.
Pupil-Corneal
reflection
The IR source can generate a glint on corneal and
divide pupil from iris, the difference between can
represent the eye gaze movement
Ⅱ. Theory and the Classification of EGT Technology
From View of Theory
Purkinje Image
EOG
Limbus-Scalar
IR Tracking
The Purkinje images are reflections created at different
layers of the eye structure. Eye gaze can be calculated
from these relative positions of these reflections
By placing electrodes around the eye, it is possible
to measure small differences in the skin potential
corresponding to eye movement.
The Limbus is the boundary between the white sclera
and the dark iris of the eye. This boundary can easily
be optically detected and tracked
Contact Lens
Training images
are taken
the user
is looking
Training
images
arewhen
taken
when
the at
a specific Calibration markers. Use ANN to decide
user
is looking at a specific
the eye gaze
Calibration markers. Use ANN to
将两对氯化银皮肤表面电极分别置于眼睛左右、上下两侧,
Use
a small coil embedded into a contact lens that is tightly fit
decide
the eye gaze
就能引起眼球变化方向上的微弱电信号,
over
the sclera . The user’s gaze is estimated经放大后得到眼
from measuring the
Pupil-Corneal
reflection
The IR source can generate a glint on corneal and
divide pupil from iris, the difference between can
represent the eye gaze movement
ANN
voltage
induced in the search coil by an external electro-magnetic
球运动的位置信息
field.
Ⅱ. Theory and the Classification of EGT Technology
From View of Theory
Purkinje Image
EOG
Limbus-Scalar
IR Tracking
The Purkinje images are reflections created at different
layers of the eye structure. Eye gaze can be calculated
from these relative positions of these reflections
By placing electrodes around the eye, it is possible
to measure small differences in the skin potential
corresponding to eye movement.
The Limbus is the boundary between the white sclera
and the dark iris of the eye. This boundary can easily
be optically detected and tracked
Contact Lens
Use
a small
embedded
Training
imagescoil
are taken
when theinto
user a
is contact
looking at
lens
that
is
tightly
fit
over
the
sclera
. The
a specific Calibration markers. Use ANN to decide
the eye gaze
user’s
gaze is estimated from measuring
将两对氯化银皮肤表面电极分别置于眼睛左右、上下两侧,
the
voltage induced in the search coil by
Use a small coil embedded into a contact lens that is tightly fit
就能引起眼球变化方向上的微弱电信号,
经放大后得到眼
an
external
electro-magnetic
field.
over
the sclera . The
user’s gaze is estimated
from measuring the
Pupil-Corneal
reflection
The IR source can generate a glint on corneal and
divide pupil from iris, the difference between can
represent the eye gaze movement
ANN
球运动的位置信息
voltage
induced in the search coil by an external electro-magnetic
field.
Ⅱ. Theory and the Classification of EGT Technology
From View of Theory
Purkinje Image
EOG
Limbus-Scalar
IR Tracking
ANN
Contact Lens
Pupil-Corneal
reflection
The Purkinje images are reflections created at different
layers of the eye structure. Eye gaze can be calculated
from these relative positions of these reflections
By placing electrodes around the eye, it is possible
to measure small differences in the skin potential
corresponding to eye movement.
The Limbus is the boundary between the white sclera
and the dark iris of the eye. This boundary can easily
be optically detected and tracked
Training images are taken when the user is looking at
a specific Calibration markers. Use ANN to decide
the eye gaze
The
IR source
can
a glint
Use
a small
coil embedded
intogenerate
a contact lens that
is tightly fit
over the sclera . The user’s gaze is estimated from measuring the
on
corneal and divide pupil from iris,
voltage induced in the search coil by an external electro-magnetic
field.
the difference between can represent
将两对氯化银皮肤表面电极分别置于眼睛左右、上下两侧,
the
eye
gaze
The
IR
source
can movement
generate a glint on 经放大后得到眼
corneal and
就能引起眼球变化方向上的微弱电信号,
divide
pupil from iris, the difference between can
球运动的位置信息
represent the eye gaze movement
Ⅱ. Theory and the Classification of EGT Technology
EGC Technology
Accuracy
Features
Purkinje Image
0.0170
High accuracy, but the light is
hard to control and can only be
used in Lab
EOG
1.50-2°
low robustness, low accuracy,
high intrusiveness
Limbus Tracking
V 10-70
H 0.50-70
Horizontal accuracy is better
than Vertical, but both are low
ANN
1.3-1.8°
No need of calibration,low
accuracy,
Contact Lens
0.080
high accuracy, high
intrusiveness
Pupil Corneal Reflection 1°
The best one till now
Ⅱ. Theory and the Classification of EGT Technology
Feature of Eye Movement
view
从系统 of
humanity
应用角度
Diagnostic(ÕïÁÆ
ʽ)£º ÊÓÏß ×· ×ٵĽá¹û ×÷Ϊ Ò»¸ö ¶¨ Á¿ÌØÐÔ£¬¿¼²ì È˵ÄÊÓ¾õºÍ ÐÄÀíѧÑо¿
£¨ ¾ø¶Ô׼ȷ ÐÔ£¬¶ÔÑÛÇò¶¯×÷µÄÇø·Ö£©
intrusive
£º
Contact
lens¡¢
Photo-transmiter¡¢
IR light emitting diodes
Interactive (½»»¥Ê½)£º ֻҪ׷
×Ù³ö ÊÓ
Ïß £¬×÷Ϊ ºó ¶Ë½»»¥µÄÊäÈë
£¨ Ïà
¶Ô×¼
È· £¬Ò»°ã
²» Çø·ÖÑÛ
ÇòµÄÔ˶¯ÌØÐÔ£¬Ö»ÒªÖªµÀÊÓÏß ·½Ïò £©
non-intrusive £º
use
imagebased
Method
Taxonomy and Models of Eye Movements
Fixation
duration:100~200ms
Saccades
angle: 1o~4o
duration£º 30~120ms
volecity:400o/s~600o/s
Smooth Pursuit
volecity£º 1o/s~30o/s
Ⅱ. Theory and the Classification of EGT Technology
Development


Early stage : Direct Observation ,Mechanical Method
Initial Methods, not used any more for high intrusiveness and poor accuracy

Development：EOG、 Electromagnetic Oculography
Although have improved a lot and widely used , these method are disappearing because of the
intrusiveness

Advanced：Optical Method：
Because of its’ high accuracy and low intrusiveness, Optical Method have made rapid progress
in recent years.
1.
The Purkinje images are reflections created at different layers of the eye structure. and the
eye gaze can be calculated from these relative positions of these reflections
2.
Photo-Oculography : it measures the eye movement during it's translation/rotation. with
the IR light source, shape of the pupil ,Limbus or corneal reflection is detected
3.
Limbus-Scalar IR Method: They place IR light emitting diodes and IR light sensitive phototransistors, respectively, above and below the eye. Several such IR pairs can be mounted
on goggles or helmets, a photo-transistor transforms the reflected IR light into a voltage.
the resulting voltage difference is proportional to the angular deviation of the eye.
4.
Pupil-Corneal Reflection: The IR source can generate a glint on corneal and divide pupil
from iris, the camera can extract the pupil which represent the eye gaze, on the other hand
the corneal reflection represent the head motion, as a result, the difference between them
represents the real eye gaze movement.。
Ⅱ. Theory and the Classification of EGT Technology
Feature of Remote Eye Gaze Tracker
Body-Mounted£º goggle¡¢ helmet¡¢ back pack, etc
note : although labeled with inrusive ,but more and more body-Mounted systems
are becoming non-intrusive
View of
appearance


Robustness
Remote system£º Remote EYE Gaze Tracker (REGT)
 Light Source
 Eyelash, Eyelids and Camera Position
 System Error
 Eye station (dry or wet)
 head motion
Lower accuracy than
Head Mounted Eye Tracker
Ⅲ A Practical Eye Gaze
Tracking System
Ⅲ A Practical Eye Gaze Tracking System
Framework of EGT System

Hardware：a Pair of Cameras



eye camera
scene camera
eye camera is for acquisition of pupil corneal reflection image,
scene camera is for: 1 mapping from the eye image coordinate to scene image coordinate ;2
Showing the combining result of POR and scene in same image：
Output of Scene
Morniter
Video Stream
Visual Stimulus
Eye Camera
IR Source
Scene Camera
Tracking of
Pupil
-Corne al
Re fle ction
Auto
Mapping
PC
IR Source
Output of Eye
Optical System
Video Stream
Ⅲ A Practical Eye Gaze Tracking System
appearance of EGT System
Ⅲ Practical Eye Gaze Tracking System
Theory of a Head Mounted Eye Tracker
Eye Camera
Capture
Pupil-Corne
al image
Estimate
Pupil-Corneal
reflection
22222222222
222222222D
Eye
Mapping between
xianglian
Coordinate to Scene
Coordinate
Scene Camera
Coordinate
Mapping
Eye
Tracking
Result
Calibration
Tracking
Ⅲ Practical Eye Gaze Tracking System
Bright Pupil and Dark Pupil
Bright Pupil
Dark Pupil
Difference
Image

When IR source is placed near the optical axis of the camera
bright pupil as can be seen ;while IR is placed off the optical
axis, a dark pupil can be seen. By Thresholding, a robust
pupil contour can be extracted .
Ⅲ Practical Eye Gaze Tracking System
Bright Pupil and Dark Pupil
Bright Pupil
Dark Pupil
Corneal Reflection
The IR source can generate a glint on corneal and divide
pupil from iris, the camera can extract the pupil which
represent the eye gaze, on the other hand the corneal
reflection represent the head motion, as a result, the
difference between them represents the real eye gaze
movement.
Ⅲ Practical Eye Gaze Tracking System
Ⅲ Practical Eye Gaze Tracking System
Ⅲ Practical Eye Gaze Tracking System
Part IV :Applications
HOW FAR YOU CAN FLY
2016/3/28
EIE426-AICV
41
Applications

Human Computer Interaction (HCI)—— efficiency, humanity

Intelligent Control —— EGT and Weapon Control

Human Movement Study ——Typing, physical training

Psychology —— Antinational Neuroscience,

Visual attention & Driving ——Aviation, navigation, driving,
traffic accidents inspection

2016/3/28
Scene and Image Perception—— Web, AD, Designing, Scene
EIE426-AICV
42
Most Recent Application
2016/3/28
EIE426-AICV
52