Transcript National Alliance for Medical Image Computing

Medical Imaging Projects
Daniela S. Raicu, PhD
Assistant Professor
Email: [email protected]
Lab URL: http://facweb.cs.depaul.edu/research/vc/
IMP & MediX Labs @ DePaul
Faculty:
GM. Besana, L. Dettori, J. Furst, G.
Gordon, S. Jost, D. Raicu, N. Tomuro
CTI Students:
W. Horsthemke, C. Philips, R. Susomboon,
J. Zhang E. Varutbangkul, S.G. Valencia
IMP Collaborators & Funding Agencies
• National Science Foundation (NSF) - Research Experience for Undergraduates (REU)
• Northwestern University - Department of Radiology, Imaging Informatics Section
• University of Chicago – Medical Physics Department
• Argonne National Laboratory - Biochip Technology Center
• MacArthur Foundation
MedIX REU Program, Summer 2007
2
Outline
 Medical Imaging and Computed Tomography
 Soft Tissue Segmentation in Computed Tomography
 Project 1: Region-based classification
 Project 2: Texture-based snake approach
 Content-based Image Retrieval and Annotation
 Project 3: Lung Nodule Retrieval based on image content and
radiologists’ feedback
 Project 4: Associations discovery between image content and
radiologists’ assessment
MedIX REU Program, Summer 2007
3
What is Medical Imaging (MI)?
The study of medical imaging is concerned with the
interaction of all forms of radiation with tissue and the
development of appropriate technology to extract clinically
useful information from observation of this technology.
X-Ray
CT
MedIX REU Program, Summer 2007
fMRI
4
Computed Tomography (CT)
• G. Hounsfield (computer expert) and A.M. Cormack
(physicist) (Nobel Prize in Medicine in 1979)
• CT overcomes limitations of plain radiography
• CT doesn’t superimpose structures (like X-ray)
• CT is an imaging based on a mathematical formalism that
states that if an object is viewed from a number of different
angles than a cross-sectional image of it can be computed
(reconstructed)
_______________________________________________
MedIX REU Program, Summer 2007
5
CT Data
Stages of construction of a voxel dataset from CT data
(a) CT data capture works by taking many one dimensional
projections through a slice (scanning)
(b) CT reconstruction pipeline
MedIX REU Program, Summer 2007
6
CT – Data Acquisition
Slice-by-slice acquisition
• X-ray tube is rotating around
patient to acquire a slice
• patient is moved to acquire
the next slice
Volume acquisition
• X-ray tube is moving continuously
along a spiral (helical) path and
the data is acquired continuously
_______________________________________________
MedIX REU Program, Summer 2007
7
CT – Data Acquisition
(a) slice-by-slice scanning
(b) Spiral (volume) scanning
MedIX REU Program, Summer 2007
8
CT – SPIRAL SCANNING
• a patient is moved 10mm/s (24cm / single scan)
• slice thickness: 1mm-1cm
• faster than slice-by-slice CT
• no shifting of anatomical structures
• slice can be reconstructed with an arbitrary
orientation with (a single breath) volume
CT multi-slice systems:
• parallel system of detectors
• 4/8/16 slices at a time
• generates a large data of thin slices
• better spatial resolution ( better reconstruction)
MedIX REU Program, Summer 2007
9
CT - DATA PROCESSING
CT numbers (Hounsfield units) HU:
• computed via reconstruction
algorithm (~tissue density/
X-ray absorption)
• most attenuation (bone)
• least attenuation (air)
• blood/calcium increases tissue
density
MedIX REU Program, Summer 2007
Understanding Visual Information: Technical, Cognitive and Social Factors
10
CT - DATA PROCESSING
Relationship between CT numbers and brightness level
MedIX REU Program, Summer 2007
Understanding Visual Information: Technical, Cognitive and Social Factors
11
CT - IMAGE DISPLAY
Human eye can perceive only a limited
range gray-scale values
Thoracic image:
a) width 400HU/level 40HU (no lung
detail is seen)
b) width 1000HU/level –700HU (lung
detail is well seen; bone and soft tissue
detail is lost)
MedIX REU Program, Summer 2007
12
CT Medical Imaging (MI)@ CTI
Filtering
Registration
Projects 1&2: Texture-based
Correction
soft-tissue segmentation
Segmentation
Projects 3&4: Content-based
Visualization
Analysis medical image retrieval and
annotation
Classification
Retrieval
MedIX REU Program, Summer 2007
13
Outline
 Medical Imaging and Computed Tomography
 Soft Tissue Segmentation in Computed Tomography
 Project 1: Region-based classification approach
 Project 2: Texture-based snake approach
 Content-based Image Retrieval and Annotation
 Project 3: Lung Nodule Retrieval based on image content and
radiologists’ feedback
 Project 4: Associations discovery between image content and
radiologists’ assessment
MedIX REU Program, Summer 2007
14
Soft-tissue Segmentation in
Computed Tomography
Goal: context-sensitive tools for radiology reporting
Approach: pixel-based texture classification
Pixel Level Texture
Extraction
Pixel Level
Classification
d1 , d 2 , d k 
Organ Segmentation
tissue _ label 
MedIX REU Program, Summer 2007
15
Soft-tissue Segmentation in
Computed Tomography
Pixel-based texture extraction:
Input Patient Data Characteristics:
 hundreds of images per patient
 image spatial resolution: 512 x512
 image gray-level resolution: 212
Challenges:
 Storage:
 Input: 0.5+ terabyte of raw data
dispersed over about 100K+ images
 Output: 90+ terabytes of low-level
features in a 180 dimensional feature space
 Compute:
 24 hours of compute time = 180 features
for a single image on a modern 3GHz
workstation
Pixel Level Texture
Extraction
d1 , d 2 , d k 
Output Data Characteristics:
 low-level image features (numerical
descriptors)
 k=180 Haralick texture features per
pixel (9 descriptors x4 directions x5 displacements)
MedIX REU Program, Summer 2007
16
Project 1: Challenges and
opportunities
 Calculate image features at region-level instead of pixel-level
 Include Gabor features in the feature extraction phase in addition to the cooccurrence texture features
 Explore different approaches for region classification in addition to the decision
tree approach
Current Implementation: Matlab
 feature1
 feature
2



 featurek
Stack of CT slices
Image Partitioning
Feature Extraction
MedIX REU Program, Summer 2007
_ tissue _ label
Region Classification
17
Liver Segmentation Example
J.D. Furst, R. Susomboon, and D.S. Raicu, "Single Organ Segmentation Filters for Multiple Organ Segmentation", IEEE
2006 International Conference of the Engineering in Medicine and Biology Society (EMBS'06)
Original Image
Initial Seed at 90%
Split & Merge at 85%
Split & Merge at 80%
Region growing at 70% Region growing at 60% Segmentation Result
MedIX REU Program, Summer 2007
18
Soft-tissue Segmentation in
Computed Tomography
Snake Application Demo
Next figures are demonstrated how to automatically classify the
CT images of heart and liver.
MedIX REU Program, Summer 2007
19
Demo For HEART
There are 4 main menu to operate this
application.
MedIX REU Program, Summer 2007
OPEN: SEGMENT:
Toa automatically
To open
new Image.
segment the
region of interest
TEXTURE:
organ
To calculate
the texture
CLASSIFICATION:
models: coTo automatically
occurrence/
classify the
run-length
segmented organ
20
HEART: Segmentation
The application
allows users
to change
Snake/
Active contour
algorithm
parameters
MedIX REU Program, Summer 2007
21
HEART: Segmentation (cont.)
Button is
clicked
User selects
points
around the
region of
MedIX REU Program, Summer 2007
interest
22
HEART: Segmentation (result)
Show
segmented
organ
If the user likes the result of the segmentation,
then the user will go to the classification step
MedIX REU Program, Summer 2007
23
HEART: Classification
Selection of
texture models:
Co-occurrence,
Run-length,
Or Combine
both models
Texture features corresponding to the selected
texture model are calculated and shown here
MedIX REU Program, Summer 2007
24
HEART: Classification Result
Results are
shown as
follows.
Predicted organ:
Heart
Probability:0.86
And also rule
which is used
to predict that
this segmented
organ is HEART
MedIX REU Program, Summer 2007
25
Demo For LIVER
Start application by open and load the image.
MedIX REU Program, Summer 2007
26
LIVER: Segmentation
The application
allows users
to change
Snake/
Active contour
algorithm
parameters
MedIX REU Program, Summer 2007
27
LIVER: Segmentation (cont.)
Segmentation
Button is
clicked
User selects points
around the region of
interest
MedIX REU Program, Summer 2007
28
LIVER: Segmentation Result
Show
segmented
organ
If user is satisfied with the result, then it will go to the
classification step
MedIX REU Program, Summer 2007
29
LIVER: Classification
Select texture
models:
Co-occurrence,
Run-length,
Or Combine both
models
Texture features is calculated for the selected model
MedIX REU Program, Summer 2007
30
LIVER: Classification Result
Results are
shown as
follows.
Predicted organ:
Liver
Probability:1.00
And also rule
which is used
to predict that
this segmented
organ is LIVER
MedIX REU Program, Summer 2007
31
Project 2: Challenges and
opportunities
 Calculate texture image features at the pixel level instead of using the graylevels
 Apply snake on the texture features
 Investigate different ways to objectively compare two segmentation algorithms,
in particular the snake and the classification-based approach
Current Implementation: Matlab
MedIX REU Program, Summer 2007
32
Outline
 Medical Imaging and Computed Tomography
 Soft Tissue Segmentation in Computed Tomography
 Project 1: Region-based classification approach
 Project 2: Texture-based snake approach
 Content-based Image Retrieval and Annotation
 Project 3: Lung Nodule Retrieval based on image content and
radiologists’ feedback
 Project 4: Associations discovery between image content and
radiologists’ assessment
MedIX REU Program, Summer 2007
33
Outline
 Medical Imaging and Computed Tomography
 Soft Tissue Segmentation in Computed Tomography
 Project 1: Region-based classification approach
 Project 2: Texture-based snake approach
 Content-based Image Retrieval and Annotation
 Project 3: Lung Nodule Retrieval based on image content and
radiologists’ feedback
 Project 4: Associations discovery between image content and
radiologists’ assessment
MedIX REU Program, Summer 2007
34
Content-based medical image
retrieval (CBMS) systems
Definition of Content-based Image Retrieval:
Content-based image retrieval is a technique for retrieving
images on the basis of automatically derived image
features such as texture and shape.
Applications
of Content-based Image Retrieval:
 Teaching
 Case-base reasoning
 Evidence-based medicine
MedIX REU Program, Summer 2007
35
Diagram of a CBIR
Image Database
Image Features
Feature Extraction
[D1, D2,…Dn]
Similarity Retrieval
Query Image
Feedback
Algorithm
User Evaluation
MedIX REU Program, Summer 2007
Query Results
36
CBIR as a tool for lookup and
reference
• Case Study: lung nodules retrieval
– Lung Imaging Database Resource for Imaging Research
http://imaging.cancer.gov/programsandresources/Inf
ormationSystems/LIDC/page7
– 29 cases, 5,756 DICOM images/slices, 1,143 nodule images
– 4 radiologists annotated the images using 9 nodule
characteristics: calcification, internal structure, lobulation,
malignancy, margin, sphericity, spiculation, subtlety, and texture
• Goals:
– Retrieve nodules based on image features:
• Texture, Shape, and Size
– Find the correlations between the image features and the
radiologists’ annotations
MedIX REU Program, Summer 2007
37
Examples of nodule images
MedIX REU Program, Summer 2007
38
CBIR as a tool for lung nodule
lookup and reference
Low-level feature extraction:
MedIX REU Program, Summer 2007
39
Nodule Characteristics
– Calcification
• (1. Popcorn, 2. Laminated, 3. Solid,
4. Non-Central, 5. Central, 6. Absent)
– Internal Structure
• (1. soft tissue, 2. fluid, 3. fat, 4. air)
– Subtlety
• (1. extremely subtle,..................., 5. obvious)
– Sphericity
• (1. Linear, 2. ......, 3. Ovoid, 4. ....., 5. Round)
– Texture
• (1. Non-Solid, 2. ....., 3. Part Solid, 4. ......., 5. Solid)
MedIX REU Program, Summer 2007
40
Nodule Characteristics
– Margin
• (1. Poorly, ......................., 5. Sharp)
– Lobulation
• (1. Marked, ....................., 5. No Lobulation)
– Spiculation
• (1. Marked, ....................., 5. No Spiculation)
– Malignancy
• (1. Highly Unlikely for Cancer, ...............,
5. Highly Suspicious for Cancer)
MedIX REU Program, Summer 2007
41
Choose a nodule
MedIX REU Program, Summer 2007
42
Choose an image
feature& a similarity
measure
M. Lam, T. Disney, M. Pham, D. Raicu, J. Furst, “Content-Based Image Retrieval for
MedIX
REU Program,
2007Nodule Images”, SPIE Medical Imaging 43
Pulmonary
ComputedSummer
Tomography
Conference,
San Diego, CA, February 2007
MedIX
REU Program,
Summer 2007
Retrieved
Images
44
Project 3: Challenges and
opportunities
 Calculate co-occurrence texture features at the local level instead of global level
 Incorporate shape and size features in the retrieval process in addition to
texture features
 Integrate radiologists’ assessments/feedback into the retrieval process
 Investigate different approaches for retrieval in addition to similarity measures
 Report the retrieval results with a certain confidence level (probability) instead
of just a binary output (similar/not similar)
Current implementation: C#
Available Open Source at: http://brisc.sourceforge.net/
MedIX REU Program, Summer 2007
45
Outline
 Medical Imaging and Computed Tomography
 Soft Tissue Segmentation in Computed Tomography
 Project 1: Region-based classification approach
 Project 2: Texture-based snake approach
 Content-based Image Retrieval and Annotation
 Project 3: Lung Nodule Retrieval based on image content and
radiologists’ feedback
 Project 4: Associations discovery between image content and radiologists’
assessment
MedIX REU Program, Summer 2007
46
Associations between image
content and semantics
Characteristics
Regression
Coefficients
Calcification
InternalStructure
Lobulation
Adj-R2 = 0.990
Malignancy
Margin
Sphericity
F-value = 963.560
p-value = 0.000
(Constant)
gabormean_1_2
MinIntensityBG
Energy
gabormean_0_1
IntesityDifference
inverseVariance
gabormean_1_1
gabormean_2_1
Correlation
clusterTendency
ConvexPerimeter
5.377275
-0.02069
0.003819
-28.5314
-0.00315
0.000272
6.317133
0.009743
-0.00667
-0.39183
5.16E-06
-0.00291
p-value
1.64E-54
7.80E-07
3.30E-82
3.31E-12
5.80E-14
0.003609
3.41E-05
0.000259
5.79E-05
5.67E-05
0.000131
0.023032
Spiculation
Subtlety
Texture
Estimated Malignancy = 5.377275 - 0.02069 gabormean_1_2 + 0.003819 MinIntensityBG
- 28.5314 energy - 0.00315 gabormean_0_1
+ 0.000272 IntesityDifference + 6.317133 inverseVariance
+ 0.009743 gabormean_1_1 - 0.00667 gabormean_2_1
- 0.39183 correlation + 5.16E-06 clusterTendency
- 0.00291 ConvexPerimeter
MedIX REU Program, Summer 2007
47
Project 4: Challenges and
opportunities
 Investigate other approaches for finding the associations between image
features and radiologists’ assessment in addition to logistic regression and
decision trees
 from image content to semantics
 from semantics to semantics
 from image features and semantics to semantics
 Create GUIs to display examples of images for each semantic concept
 Investigate how the current associations discovery approaches apply to
mammography assessment (Northwestern project)
Current implementation: Matlab, Weka, SPSS
MedIX REU Program, Summer 2007
48
Questions?
Thank you!
MedIX REU Program, Summer 2007
49