Transcript National Alliance for Medical Image Computing

NSF MedIX REU Program
Medical Imaging Projects
@ DePaul CDM
Daniela S. Raicu, PhD
Associate Professor
Email: [email protected]
Lab URL: http://facweb.cs.depaul.edu/research/vc/
Outline
Medical Imaging (Computed Tomography)
– Content-based and semantic-based image retrieval
• Projects 1 and 2
– Mappings from low-level image features to semantic
concepts
• Projects 3 and 4
– Liver segmentation
• Project 5
NSF MedIX REU Program, CDM, DePaul University
Content-based medical image
retrieval (CBMS) systems
Definition of Content-based Image Retrieval:
Content-based image retrieval is a technique for retrieving
images on the basis of automatically derived image
features such as texture and shape.
-
Applications of Content-based Image Retrieval:
• Teaching
• Research
• Diagnosis
• PACS and Electronic Patient Records
NSF MedIX REU Program, CDM, DePaul University
Diagram of a CBIR
Image Database
Image Features
Feature Extraction
[D1, D2,…Dn]
Similarity Retrieval
Query Image
Feedback
Algorithm
User Evaluation
NSF MedIX REU Program, CDM, DePaul University
Query Results
CBIR as a Diagnosis Aid
An image retrieval system can help when the diagnosis
depends strongly on direct visual properties of images in the
context of evidence-based medicine or case-based
reasoning.
NSF MedIX REU Program, CDM, DePaul University
CBIR as a Teaching Tool
An image retrieval system will allow students/teachers to browse
available data themselves in an easy and straightforward fashion by
clicking on “show me similar images”.
Advantages:
- stimulate self-learning and a comparison of similar cases
- find optimal cases for teaching
Teaching files:
• Casimage: http://www.casimage.com
• myPACS: http://www.mypacs.net
NSF MedIX REU Program, CDM, DePaul University
CBIR as a Research Tool
Image retrieval systems can be used:
• to complement text-based retrieval methods
• for visual knowledge management whereby the images
and associated textual data can be analyzed together
• multimedia data mining can be applied to
learn the unknown links between visual
features and diagnosis or other patient
information
• for quality control to find images that might have been
misclassified
NSF MedIX REU Program, CDM, DePaul University
CBIR as a tool for lookup and
reference in CT chest/abdomen
• Case Study: lung nodules retrieval
– Lung Imaging Database Resource for Imaging Research
http://imaging.cancer.gov/programsandresources/Inf
ormationSystems/LIDC/page7
– 29 cases, 5,756 DICOM images/slices, 1,143 nodule images
– 4 radiologists annotated the images using 9 nodule
characteristics: calcification, internal structure, lobulation,
malignancy, margin, sphericity, spiculation, subtlety, and texture
• Goals:
– Retrieve nodules based on image features:
• Texture, Shape, and Size
– Find the correlations between the image features and the
radiologists’ annotations
NSF MedIX REU Program, CDM, DePaul University
LIDC Semantic Concepts
Calcification
1.
2.
3.
4.
5.
6.
Popcorn
Laminated
Solid
Non-central
Central
Absent
Sphericity
1.
2.
3.
4.
5.
Linear
.
Ovoid
.
Round
Internal structure
1.
2.
3.
4.
Soft Tissue
Fluid
Fat
Air
Spiculation
1.
2.
3.
4.
5.
Marked
.
.
.
None
Lobulation
1.
2.
3.
4.
5.
Marked
.
.
.
None
Subtlety
1.
2.
3.
4.
5.
Extremely Subtle
Moderately Subtle
Fairly Subtle
Moderately Obvious
Obvious
Malignancy
1.
2.
3.
4.
5.
Highly Unlikely
Moderately Unlikely
Indeterminate
Moderately Suspicious
Highly Suspicious
Texture
1.
2.
3.
4.
5.
Non-Solid
.
Part Solid/(Mixed)
.
Solid
Margin
1.
2.
3.
4.
5.
Poorly Defined
.
.
.
Sharp
NSF MedIX REU Program, CDM, DePaul University
Extracted Image Features
Shape Features
Size Features
Intensity Features
Circularity
Area
MinIntensity
Roughness
ConvexArea
MaxIntensity
Elongation
Perimeter
MeanIntensity
Compactness
ConvexPerimeter
SDIntensity
Eccentricity
EquivDiameter
MinIntensityBG
Solidity
MajorAxisLength
MaxIntensityBG
Extent
MinorAxisLength
MeanIntensityBG
RadialDistanceSD
SDIntensityBG
IntensityDifference
NSF MedIX REU Program, CDM, DePaul University
Texture Features
11 Haralick features calculated from cooccurrence matrices (Contrast, Correlation,
Entropy, Energy, Homogeneity, 3rd Order
Moment, Inverse Differential Moment,
Variance, Sum Average, Cluster Tendency,
Maximum Probability)
24 Gabor features - mean and standard
deviation of Gabor filters consistency of
four orientations and three scales.
Lung nodule representation
NSF MedIX REU Program, CDM, DePaul University
Choose a nodule
NSF MedIX REU Program, CDM, DePaul University
Choose an image
feature& a similarity
measure
NSF MedIX REU Program, CDM, DePaul University
NSF MedIX REU Program, CDM,
DePaul University
Retrieved
Images
CBIR systems: challenges & REU
projects
•Type of features
• image features:
- texture features: statistical, structural,
model and filter-based
- shape features
• textual features (such as physician annotations)
Project 1: Feature reduction for medical image processing
- Investigate how many features with respect to the number of
unique nodules
- Investigate what the most important low-level image features are
with respect to the retrieval process
- Investigate the uniformity of the features with respect to the same
type of nodules.
NSF MedIX REU Program, CDM, DePaul University
CBIR systems: challenges & REU
projects (cont.)
•Similarity measures
-point-based and distribution based metrics
• Retrieval performance:
• precision and recall
• clinical evaluation
Project 2: Evaluation of CBIR and SBIR systems
•
•
•
Perform a literature review on the current techniques used to evaluate CBIR
systems both for the general and medical domain
Investigate ways to include radiologists’ feedback in the retrieval process
Investigate ways to evaluate the retrieval process by varying various
numbers of parameters such as number of images retrieved, cutoff value for
acceptable precision and recall, and minimum number of
radiologists/observers needed to evaluate the system.
NSF MedIX REU Program, CDM, DePaul University
Correlations between Image
Features and Concepts
Characteristics
Image Features
Lobulation
0.65
Spiculation
-0.42, -0.42, 0.34, 0.30
Eccentricity, Elongation, Extent, Circularity
Margin
Sphericity
0.62
Malignancy
0.47
Subtlety
0.48, 0.48, 0.48,
0.47, 0.47, 0.47,
0.46
0.52, 0.52, 0.52,
0.53, 0.51, 0.51,
0.49
Texture
InternalStructure
Calcification
NSF MedIX REU Program, CDM, DePaul University
Area, ConvexArea, EquivDiameter,
MinorAxisLength, ConvexPerimeter,
Perimeter, MajorAxisLength
Automatic Mappings Extraction
Step-wise multiple regression analysis was applied to
generate prediction models for each characteristic ci
based on all image features fk:
M i : ci   0 

k 1, p
k
fk  i
where p is the # of used image features,  k are the regression coefficients,
and  i are the prediction errors per model.
Goodness of fit for the regression model:
adj _ R  1  1  R
2
2
n  1
n  p  1
NSF MedIX REU Program, CDM, DePaul University
Regression Models
Characteristics
Calcification
Entire dataset
(1106 images, 73
nodules)
At least 2 radiologists
agreed
At least 3 radiologists agreed
0.397
0.578 (884, 41)
0.645 (644, 21)
0.417
- (855, 40)
- (659, 22)
Lobulation
0.282
0.559 (448, 24)
0.877 (137, 6)
Malignancy
0.310
0.641 (489, 23)
0.990 (107, 5)
0.403
0.376 (519, 28)
- (245, 7)
Sphericity
0.239
0.481 (575, 27)
0.682 (207, 9)
Spiculation
0.320
0.563 (621, 29)
0.840 (228, 9)
Subtlety
0.301
0.282 (659, 25)
0.491 (360, 10)
Texture
0.181
0.473 (736, 33)
0.843 (437, 15)
Internal Structure
Margin
NSF MedIX REU Program, CDM, DePaul University
Texture Regression Model
NSF MedIX REU Program, CDM, DePaul University
Malignancy Regression Model
Characteristics
Calcification
InternalStructure
Lobulation
Adj_R2 = 0.990
M alignancy
M argin
Sphericity
F-value = 963.560
p-value = 0.000
(Co nstan t)
Gabormean_4 5¼_0.5
M inIn tensity BG
En ergy
Gabormean_0¼
_0.4
In tesityDifference
In verseVariance
Gabormean_4 5¼_0.4
Gabormean_9 0¼_0.4
Co rrelation
ClusterT endency
Co nv exPerimeter
Re gression
Coe fficients
p-valu e
5.3 77 27 5
-0.0 20 69
0.0 03 81 9
-2 8.5 31 4
-0.0 03 15
0.0 00 27 2
6.3 17 13 3
0.0 09 74 3
-0.0 06 67
-0.3 91 83
5.1 6E-0 6
-0.0 02 91
1.6 4E-5 4
7.8 0E-0 7
3.3 0E-8 2
3.3 1E-1 2
5.8 0E-1 4
0.0 03 60 9
3.4 1E-0 5
0.0 00 25 9
5.7 9E-0 5
5.6 7E-0 5
0.0 00 13 1
0.0 23 03 2
Spiculation
Subtlety
Texture
Estimated M alignancy = 5.3 77 27 5- 0.0 20 69Gabormean_4 5¼_0.5 + 0.0 03 81 9M inIn tensity BG
- 2 8.5 31 4En ergy - 0.0 03 15Gabormean_0¼_0.4
+ 0.0 00 27 2In tesityDifference+ 6.3 17 13 3In verseVariance
+ 0.0 09 74 3Gabormean_4 5¼_0.4 - 0.0 06 67Gabormean_9 0¼_0.4
- 0.3 91 83Co rrelation+ 5.1 6E-0 6 ClusterT endency
- 0.0 02 91Co nv exPerimeter
NSF MedIX REU Program, CDM, DePaul University
Lobulation Regression Model
NSF MedIX REU Program, CDM, DePaul University
Spiculation Regression Model
NSF MedIX REU Program, CDM, DePaul University
Image Features – Semantics
Mappings: challenges & REU projects
Project 3: Multi-view learning classifier for lung nodule
classification
•
Investigate which image features are the best for individual semantic
characteristics, build classifiers for each one of the individual classifiers,
and combine the individual classifies for optimal learning/classification
of lung nodules
Project 4: Bridging the semantic gap in lung nodule
interpretation
•
•
Investigate ways to clinically evaluate the mappings from low-level
image features to semantic characteristics
Investigate the effect of the imaging acquisition parameters (such as
pitch, FOV, and reconstruction kernel) on the proposed mappings
NSF MedIX REU Program, CDM, DePaul University
Liver Segmentation in CT images
Pixel-level Classification:
- tissue segmentation
- context-sensitive tools for radiology reporting
-
Pixel Level Texture
Extraction
d1 , d 2 , d k 
Pixel Level
Classification
tissue _ label 
NSF MedIX REU Program, CDM, DePaul University
Organ Segmentation
Liver Segmentation in CT images
Example of Liver Segmentation:
(J.D. Furst, R. Susomboon, and D.S. Raicu, "Single Organ
Segmentation Filters for Multiple Organ Segmentation", IEEE 2006 International Conference of the Engineering in
Medicine and Biology Society (EMBS'06))
Original Image
Initial Seed at 90%
Split & Merge at 85%
Split & Merge at 80%
Region growing at 70% Region growing at 60% Segmentation Result
NSF MedIX REU Program, CDM, DePaul University
Liver Segmentation using Automatic
Snake
a)
a)
b)
c)
d)
Figure: a) Gradient vector flow segmentation; b) Traditional vector field segmentation; c) and,d) Respective segmentations overlaid on ground truth (white).
Project 5: Automatic selection of initial points for snakebased segmentation
NSF MedIX REU Program, CDM, DePaul University
uestions ?
NSF MedIX REU Program, CDM, DePaul University