Transcript For input image

Registration of Pathological Images
Xiao Yang1, Xu Han1, Eunbyung Park1,
Stephen Aylward2, Roland Kwitt3, Marc Niethammer1
Department of Computer Science
1University of North Carolina at Chapel Hill
3University of Salzburg
3Kitware
Inc.
Registration of Pathological Images Image Registration
Deformable Image Registration
Goal: Spatial alignment of two images (beyond affine).
?
Source Image
Target Image
minimize Irregularity of transformation + image mismatch
2
Registration of Pathological Images Image Registration
Issue: Intensity/Structural changes
Traumatic brain injury
Brain tumor
What do we do if we need to deal with pathologies?
“Similar looking regions” do not correspond.
Structures only exist in one of the two images.
3
Registration of Pathological Images Cost-function masking
Simplest Solution: Cost-function masking
One solution: Cost function masking [Brett2001]
= ignoring matching cost in region of source image
X
Image: gerdleonhard.typepad.com
… let’s just not look at it!
4
Registration of Pathological Images Other Approaches
Other Solution Approaches
Segmentation-based methods
• Cost function masking
• Joint Segmentation and Registration
• Geometric Metamorphosis
All these methods require segmentations!
5
Registration of Pathological Images Direct Mapping
What if?
What if there were a method to transform an image with
pathology into a healthy-looking image?
magic
We can learn such a method from population data ...
6
Registration of Pathological Images Quasi-Normal by Regression
Other Solution Approaches
Low-rank/sparse approach (LRS)
• Separate the image into low-rank part and sparse part
• Requires interleaved registration and LRS
decomposition
Our approach is similar in spirit to LRS, but
• directly predicts a quasi-normal image w/o registrations
• preserves fine image detail
7
Registration of Pathological Images Quasi-Normal by Regression
Prediction via a variational autoencoder
Goal: Directly learn a regression model
from pathological to quasi-normal
We use a variational autoencoder, but other regression
models are of course possible too ...
... with this model we can also estimate output uncertainties.
8
Registration of Pathological Images Quasi-Normal by Regression
Denoising variational autoencoder (DVAE)
Pathology is considered noise:
• removing the noise result in the quasi-normal image
Variational formulation
• hidden layer values are random variables (Gaussian)
• sampling the network allows for uncertainty estimates
Autoencoder Denoising AE Variational AE
(AE))
DVAE
9
Registration of Pathological Images Network Training
Training the network
We train the network using pathological data
+ includes deformations of pathologies (mass effect)
- unclear what the quasi-normal image should be
We use two approaches
• Loss function masking:
ignores tumor area during training
• Simulated lesion:
we add a “QuasiLesion” layer to simulate lesions
for which we know the normal appearance
10
Registration of Pathological Images Network Training
Loss Function Masking
For input image
• replace pathology by constant intensity plus noise
For output image
• disregard the pathological area in backpropagation
Lesion segmentations are only required at training time!
11
Registration of Pathological Images Network Training
Quasi-Lesion Layer (i.e., simulated lesions)
Add simulated (quasi) lesions to normal areas
• and then train the network to remove those
12
Registration of Pathological Images Network Training
FakeLesion Layer (i.e., simulated lesions)
Add simulated (quasi) lesions to normal areas
• and then train the network to remove those
Choice of simulated lesion texture
• real tumor; mean or random intensity; random noise
Choice did experimentally not make a significant difference
13
Registration of Pathological Images Uncertainty
Uncertainty-Guided Registration
Can assess uncertainty if the reconstruction via sampling
• Mean = reconstruction result
• Variance = reconstruction uncertainty
14
Registration of Pathological Images Experiments
Experimental Settings
2D synthetic image + BRATs training dataset
Network training: torch + rmsprop
Uncertainty-weighted registration: modified NiftyReg
Synthetic experiment:
• register OASIS images to BRATs (cost fcn. masking)
• add BRATs tumor to deformed OASIS images
• 500 training images, 50 for testing, 196x232
BRATs experiment:
• cross-validation: 4 sets of 244 training images
+ 30 test images
• Adaptive histogram equalization
15
Registration of Pathological Images Experiments
Example Results (based on simulated data for illustration)
Original
Original + Tumor
Improved/sharper
reconstruction
Uncertainty map
to weight
registration
LRS reconstruction direct reconstruction
uncertainty
16
Registration of Pathological Images Experiments
Image Registration Result: Synthetic
Ground truth deformation
Our result without uncertainty
LRS
Our result with uncertainty
17
Registration of Pathological Images Experiments
Evaluation on BRATS 2015 Dataset
• 274 images in training dataset
• Cross-validation: 4 sets of 244 training + 30 testing images
• Landmarks for evaluation (avg. 10 landmarks per image)
Direct reconstruction ("our model") works generally best.
Uncertainty helps.
Cost function masking does not perform well.
18
Registration of Pathological Images Experiments
BRATs testcase
Original image
LRS result
Our result
Uncertainty map
19
Registration of Pathological Images Experiments
BRATs testcase
Cost function masking
LRS method
Original image
Our method,
no uncertainty
Our method
with uncertainty
20
Registration of Pathological Images Conclusion / Discussion
Conclusion / Discussion
Potential for 3D data:
• 2.5D network; 14 slices at once; slow but feasible
• 3D patch-based network (including location)
More realistic simulated lesion appearance:
• likely possible with more training data
Faithful reconstruction of normal areas is important
• excessive smoothing by LRS loses fine details
important for registration
Faithful reconstruction of pathological areas may not be
that important if combined with an uncertainty measure
• may be possible to outperform cost function masking
21
Registration of Pathological Images The End …
Thanks
Thanks for your attention!
Questions?
22