Transcript talk - Howard Zhou

Spatially Constrained Segmentation
of Dermoscopy Images
Howard Zhou1, Mei Chen2, Le Zou2, Richard Gass2,
Laura Ferris3, Laura Drogowski3, James M. Rehg1
1School
of Interactive Computing, Georgia Tech
2Intel Research Pittsburgh
3Department of Dermatology, University of Pittsburgh
1
Skin cancer and melanoma

Skin cancer : most common of all cancers
2
[ Image courtesy of “An Atlas of Surface Microscopy of Pigmented Skin Lesions: Dermoscopy” ]
Skin cancer and melanoma


Skin cancer : most common of all cancers
Melanoma : leading cause of mortality (75%)
3
[ Image courtesy of “An Atlas of Surface Microscopy of Pigmented Skin Lesions: Dermoscopy” ]
Skin cancer and melanoma



Skin cancer : most common of all cancers
Melanoma : leading cause of mortality (75%)
Early detection significantly reduces mortality
4
[ Image courtesy of “An Atlas of Surface Microscopy of Pigmented Skin Lesions: Dermoscopy” ]
Dermoscopy
Clinical View view
5
[ Image courtesy of “An Atlas of Surface Microscopy of Pigmented Skin Lesions: Dermoscopy” ]
Dermoscopy



Improve diagnostic accuracy by 30% in the hands
of trained physicians
May require as much as 5 year experience to have
the necessary training
Motivation for Computer-aided diagnosis (CAD) in
this area
Clinical view
Dermoscopy view
6
First step of analysis:
Segmentation


Separating lesions from surrounding skin
Resulting border



Gives lesion size and border irregularity
Crucial to the extraction of dermoscopic features for
diagnosis
Previous Work :




PDE approach – Erkol et al. 2005, …
Histogram thresholding – Hintz-Madsen et al. 2001, …
Clustering – Schmid 1999, Melli et al. 2006…
Statistical region merging – Celebi et al. 2007, …
7
Domain specific constraints

Spatial constraints


Four corners are skin (Melli et al.2006, Celebi et al. 2007)
Implicitly enforcing Local neighborhood constraints on image
Cartesian coordinates (Meanshift)
8
Domain specific constraints

Spatial constraints


Four corners are skin (Melli et al.2006, Celebi et al. 2007)
Implicitly enforcing Local neighborhood constraints on image
Cartesian coordinates (Meanshift)
Meanshift (c = 32, s = 8)
9
We explore …

Spatial constraints arise from the growth
pattern of pigmented skin lesions
Meanshift (c = 32, s = 8)
10
We explore …

Spatial constraints arise from the growth
pattern of pigmented skin lesions –
radiating pattern
Meanshift (c = 32, s = 8)
11
Embedding constraints


Radiating pattern from lesion growth
Embedding constraints as polar coords
improves segmentation performance
Meanshift (c = 32, s = 8)
Polar (k = 6)
12
Embedding constraints


Radiating pattern from lesion growth
Embedding constraints as polar coords
improves segmentation performance
Meanshift
Polar
Polar (k = 6)
13
Comparison to the Doctors


Radiating pattern from lesion growth
Embedding constraints as polar coords
improves segmentation performance White: Dr. Ferris
Red : Dr. Zhang
Blue : computer
Meanshift
Polar
14
Dermoscopy images
Common radiating appearance
15
Growth pattern of pigmented
skin lesions



lesions grow in both radial and vertical direction
Skin absorbs and scatters light.
Appearance of pigmented cells varies with depth


Dark brown  tan  blue-gray
Common radiating appearance pattern on skin surface
[ Image courtesy of “Dermoscopy : An Atlas of
Surface Microscopy of Pigmented Skin Lesions]
16
Radiating growth pattern on
skin surface

Difference in appearance: more significant
along the radial direction than any other
direction.
17
Radiating growth pattern on
skin surface

Difference in appearance: more significant
along the radial direction than any other
direction.
18
Embedding spatial constraints
Feature vectors

Each pixel  feature vector in R4


3D: R,G,B or L, a, b in the color space
1D: polar radius measured from the center of
the image (normalized by w)
original
r
{R, G, B}
19
Embedding spatial constraints
Grouping features



Each pixel  feature vector in R4
Clustering pixels in the feature space
Replace pixels with mean for compact
representation
original
filtered
r
{R, G, B}
20
Radiating pattern
Dermoscopy vs. natural images
…
Derm dataset (216)
…
BSD dataset (300)
21
Embedding spatial constraints
Grouping features
Cartesian

Mean per-pixel residue:
average per-pixel color
difference of each pair
{Rc, Gc, Bc}
original
polar
{Ro, Go, Bo}
{Rp, Gp, Bp}
22
Dermoscopy vs. natural images
Polar vs. Cartesion

Mean per-pixel residue (k-means++, k = 30)
Residue (Cartesian)
Residue (polar)
Residue (Cartesian)
Residue (polar)
23
Derm dataset (216)
BSD dataset (300)
Dermoscopy vs. natural images
Polar vs. Cartesion

Mean per-pixel residue (k-means++, k = 30)
24
Polar vs. Cartesian

The regions appear more blocky in the
Cartesian case
Polar (k = 30)
Cartesian (k = 30)
25
Six super-regions

30 clusters  6 super clusters (K-means++)
Polar (k = 6)
Cartesian (k = 6)
26
Final segmentation
Polar
Cartesian
27
Polar vs. Meanshift

The regions appear more blocky in the
Meanshift case
Polar (k = 6)
Meanshift (c = 32, s = 8)
28
Final segmentation
Polar
Meanshift
29
Algorithm overview

Given a dermoscopy image
30
Algorithm overview

Given a dermoscopy image
original
31
Algorithm overview
1. First round clustering: K-means++ (k = 30)
original
30 clusters
32
Algorithm overview
2. Second round: clusters(30) super-regions(6)
original
30 clusters
6 Super-regions
33
Algorithm overview
3. Apply texture gradient filter (Martin, et al. 2004)
original
30 clusters
Texture edge map
6 Super-regions
34
Algorithm overview
4. Find optimal boundary (color+texture)
original
30 clusters
Texture edge map
6 Super-regions
35
Final segmentation
1. First round clustering

First round clustering: K-means++ (k = 30)




Reduce noise
Groups pixels into homogenous regions – a
more compact representation of the image
Artuhur and Vassilvitskii, 2007
R4 : {L*a*b* (3D), w * polar radius (1D)}
original
36
1. First round clustering

First round clustering: K-means++ (k = 30)




Reduce noise
Groups pixels into homogenous regions – a
more compact representation of the image
Artuhur and Vassilvitskii, 2007
R4 : {L*a*b* (3D), w * polar radius (1D)}
original
30 clusters
37
2. Second round clustering

K = 6 : clusters(30) super-regions(6)



Account for intra-skin and intra-lesion variations
Avoid a large k
Super-regions correspond to meaningful
regions such as skin, skin-lesion transition,
and inner lesion, etc.
original
30 clusters
38
2. Second round clustering

K = 6 : clusters(30) super-regions(6)



Account for intra-skin and intra-lesion variations
Avoid a large k
Super-regions correspond to meaningful
regions such as skin, skin-lesion transition,
and inner lesion, etc.
original
30 clusters
6 super-regions
39
3. Color-texture integration

Incorporating texture information can
improve segmentation performance.

Severely sun damaged skin; texture variations
at boundaries in addition to color variations
original
40
3. Color-texture integration

Incorporating texture information can
improve segmentation performance.


Severely sun damaged skin; texture variations
at boundaries in addition to color variations
Apply texture gradient filter (Martin, et al. 2004)
original
41
3. Color-texture integration

Incorporating texture information can
improve segmentation performance.



Severely sun damaged skin; texture variations
at boundaries in addition to color variations
Apply texture gradient filter (Martin, et al. 2004)
Texture edge map: pseudo-likelihood
original
Texture edge map
42
4. Optimal boundary

Optimal skin-lesion boundary

Color: Earth Mover’s Distance (EMD) between every
pair of super-regions
6 super-regions
43
4. Optimal boundary

Optimal skin-lesion boundary


Color: Earth Mover’s Distance (EMD) between every pair
of super-regions
Texture: Texture edge map
6 super-regions
Texture edge map
44
4. Optimal boundary

Optimal skin-lesion boundary



Color: Earth Mover’s Distance (EMD) between every pair
of super-regions
Texture: Texture edge map
Minimizing the integrated color-texture measure
6 super-regions
Texture edge map
45
Validation and results




Our collaborating dermatologist Dr. Ferris manually
outline the lesions in 67 dermoscopy images
The border error is given by
Computer : binary image obtained by filling the automatic
detected border
ground-truth : obtained by filling in the boundaries
outlined by Dr. Ferris
46
Typical segmentation result
Error = 12.96%
White: Dr. Ferris
Red : Dr. Zhang
Blue : computer
47
Comparison
Compared to ground-truth outlined by Dr. Ferris
Percentage error
35
Dr. Zhang
RGB
30
25
20.64
20
21.41
19.49
CIELAB
Color + texture
20.13
16.92
15.91 14.93
15
11.32
10
5
0
none
Cartesian
polar
Dr. Zhang
Spatial constraints
To account for inter-operator variation, we also asked Dr. Alex Zhang to
manually outline boundaries on the same dataset
48
Additional results
White: Dr. Ferris
Red : Dr. Zhang
Blue : computer
Error = 5.80%
49
Additional results
White: Dr. Ferris
Red : Dr. Zhang
Blue : computer
Error = 13.61%
50
Additional results
White: Dr. Ferris
Red : Dr. Zhang
Blue : computer
Error = 16.60%
51
Additional results
White: Dr. Ferris
Red : Dr. Zhang
Blue : computer
Error = 34.09%
52
Limitation


Assumption that lesions appear relatively
near the center may not hold
Fairly low number of super regions (6) may
limit the algorithm to perform well on lesions
with more colors
53
Conclusion



Growth pattern of pigmented skin lesions can be used to
improve lesion segmentation accuracy in dermoscopy
images.
An unsupervised segmentation algorithm incorporating
these spatial constraints
We demonstrate its efficacy by comparing the
segmentation results to ground-truth segmentations
determined by an expert.
54
Future work

Extend to meanshift?
55
Comparison to other methods
Compared to ground-truth outlined by Dr. Ferris
Percentage error
30
26.74
25
20.43
20.77
20.13
20
14.93
15
11.32
10
5
0
Meanshift
JSEG (Celebi
2006)
SRM (Celebi
2007)
SCS Cartesian
SCS polar
Dr. Zhang
Segmentation methods
56
Color and texture cue integration


Apply texture gradient filter (Martin, et al. 2004)
Pseudo-likelihood map - edge caused by texture
variation is present at a certain location
57