Last updated: Date and time
Intro-operation 2D-3D registration assists surgeon with spatial localization. However, current method is performed by manually identifying corresponding landmarks in 2D X-ray images, which is time consuming. The goal of this project is to utilize deep neural network with annotated simulated X-ray data to perform landmarks detection.
This project consists of the following components:
a. Development of landmarks detection pipeline
b. Deep neural network architecture design for landmarks detection\
c. Accuracy evaluation on simulated / real test data set
Intuition
Intuitively, landmark detection problem could be represented as position regression problem. Therefore, we could formulate the problem as, given training data set of images with corresponding landmarks positions, train model such that given a test image of same size, regress exact number of landmarks positions. Based on this problem formulation, we could derive various network architectures.
Landmark Detection Pipeline
General framework for deep learning problem includes several relatively individual components. The first component is to read dataset and transform the data into appropriate form. The second component is to define the network architecture and specify settings and parameters. Then we need components for training and testing the data. We might also implement component for visualization the results, including position error and registration difference.
Deep Neural Network Design
As a start, here we will present two deep neural network architecture designs for the landmark detection.
The first architecture is to couple convolutional layers and fully connected layers for position regression. The convolutional layers could efficiently extract feature map from the input image, and the following fully connected layer could perform the position regression with feature map extracted by convolutional layers. Here, the loss function here could be the sum of square error of the positions.
The second architecture treats landmark positions in a heatmap. Therefore, the architecture generally takes image as input and infer a heatmap as output. One of the advantage of this specific architecture is that we could also address the visibility of landmarks (if a landmark does not exist in the field of view, the corresponding heatmap is empty), while the disadvantage is that it might lead to inaccurate position due to the downsampling and discrete localization.
Accuracy Evaluation
Once we have trained the network by training data set and validated and tuned by validation data set. We could evaluate the accuracy by test data set, which may consist of simulated and / or real X-ray data set. To evaluate the generalization, we will use a separate data set generating from a separated patient’s model. As for the evaluation criteria, we will use two stage evaluation, firstly by mean square error of landmarks position, and secondly by the difference of led 2D-3D registration error.
Deep Learning Background
o LeCun, Yann, et al. “Gradient-based learning applied to document recognition.” Proceedings of the IEEE 86.11 (1998): 2278-2324.
o LeCun, Yann, et al. “Backpropagation applied to handwritten zip code recognition.” Neural computation 1.4 (1989): 541-551.
o Krizhevsky, Alex, Ilya Sutskever, and Geoffrey E. Hinton. “Imagenet classification with deep convolutional neural networks.” Advances in neural information processing systems. 2012.
o LeCun, Yann, Yoshua Bengio, and Geoffrey Hinton. “Deep learning.” nature 521.7553 (2015): 436.
o Goodfellow, Ian, et al. Deep learning. Vol. 1. Cambridge: MIT press, 2016.
Feature / Landmark Detection
o Wang, Ching-Wei, et al. “Evaluation and comparison of anatomical landmark detection methods for cephalometric x-ray images: a grand challenge.” IEEE transactions on medical imaging 34.9 (2015): 1890-1900.
o Payer, Christian, et al. “Regressing heatmaps for multiple landmark localization using CNNs.” International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, Cham, 2016.
o Urschler, Martin, Thomas Ebner, and Darko Štern. “Integrating geometric configuration and appearance information into a unified framework for anatomical landmark localization.” Medical image analysis 43 (2018): 23-36.
o Sofka, Michal, et al. “Fully convolutional regression network for accurate detection of measurement points.” Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support. Springer, Cham, 2017. 258-266.
o Ronneberger, Olaf, Philipp Fischer, and Thomas Brox. “U-net: Convolutional networks for biomedical image segmentation.” International Conference on Medical image computing and computer-assisted intervention. Springer, Cham, 2015.
o Bulat, Adrian, and Georgios Tzimiropoulos. “Human pose estimation via convolutional part heatmap regression.” European Conference on Computer Vision. Springer, Cham, 2016.
X-Ray Simulation
o Badal, Andreu, and Aldo Badano. “Accelerating Monte Carlo simulations of photon transport in a voxelized geometry using a massively parallel graphics processing unit.” Medical physics 36.11 (2009): 4878-4880.
o Badal, Andreu, and Aldo Badano. “Monte Carlo simulation of X-ray imaging using a graphics processing unit.” Nuclear Science Symposium Conference Record (NSS/MIC), 2009 IEEE. IEEE, 2009.
o Bert, Julien, et al. “Geant4-based Monte Carlo simulations on GPU for medical applications.” Physics in Medicine & Biology58.16 (2013): 5593.
o Jia, Xun, et al. “A GPU tool for efficient, accurate, and realistic simulation of cone beam CT projections.” Medical physics39.12 (2012): 7368-7378.