Last updated: Date and time

• Students: Liujiang Yan
• Mentor(s): Robbert Grupp, Professor Russell Taylor

Intro-operation 2D-3D registration assists surgeon with spatial localization. However, current method is performed by manually identifying corresponding landmarks in 2D X-ray images, which is time consuming. The goal of this project is to utilize deep neural network with annotated simulated X-ray data to perform landmarks detection.

• Minimum: (April 12)
  1. Environment setup and data preparation.
  2. Data loading, pre-processing, and post-processing and evaluation framework
  3. Network architecture for landmark detection
  4. Deploy the designed network with larger data set.
  5. Accuracy Report by evaluated on simulated / real data.

• Expected: (April 30)
  1. Tools segmentation from field of view.
  2. Use better simulation software for data generation.

• Maximum: (May 11)
  1. Edge / contour detection.

This project consists of the following components:
a. Development of landmarks detection pipeline
b. Deep neural network architecture design for landmarks detection\
c. Accuracy evaluation on simulated / real test data set

Intuition
Intuitively, landmark detection problem could be represented as position regression problem. Therefore, we could formulate the problem as, given training data set of images with corresponding landmarks positions, train model such that given a test image of same size, regress exact number of landmarks positions. Based on this problem formulation, we could derive various network architectures.

Landmark Detection Pipeline
General framework for deep learning problem includes several relatively individual components. The first component is to read dataset and transform the data into appropriate form. The second component is to define the network architecture and specify settings and parameters. Then we need components for training and testing the data. We might also implement component for visualization the results, including position error and registration difference.

Deep Neural Network Design
As a start, here we will present two deep neural network architecture designs for the landmark detection.
The first architecture is to couple convolutional layers and fully connected layers for position regression. The convolutional layers could efficiently extract feature map from the input image, and the following fully connected layer could perform the position regression with feature map extracted by convolutional layers. Here, the loss function here could be the sum of square error of the positions.

The second architecture treats landmark positions in a heatmap. Therefore, the architecture generally takes image as input and infer a heatmap as output. One of the advantage of this specific architecture is that we could also address the visibility of landmarks (if a landmark does not exist in the field of view, the corresponding heatmap is empty), while the disadvantage is that it might lead to inaccurate position due to the downsampling and discrete localization.

Accuracy Evaluation
Once we have trained the network by training data set and validated and tuned by validation data set. We could evaluate the accuracy by test data set, which may consist of simulated and / or real X-ray data set. To evaluate the generalization, we will use a separate data set generating from a separated patient’s model. As for the evaluation criteria, we will use two stage evaluation, firstly by mean square error of landmarks position, and secondly by the difference of led 2D-3D registration error.

1. Milestone name: Project Proposal / Presentation
   • Planned Date: Feb 22
   • Expected Date: Feb 22
   • Status: Done
2. Milestone name: Environment Set Up
   • Planned Date: Mar 07
   • Expected Date: Mar 07
   • Status: Done
3. Milestone name: Network Architecture for Landmark Detection
   • Planned Date: Mar 29
   • Expected Date: Mar 29
   • Status: Done
4. Milestone name: Data Augmentation for Tool in the View
   • Planned Date: Apr 05
   • Expected Date: Apr 05
   • Status: Done
5. Milestone name: Seminar Presentation / Report
   • Planned Date: Apr 27
   • Expected Date: Apr 27
   • Status: Ongoing
6. Milestone name: Network Architecture for Contour Detection
   • Planned Date: Apr 30
   • Expected Date: Apr 30
   • Status: Ongoing
7. Milestone name: Final Report / Poster
   • Planned Date: May 11
   • Expected Date: May 11
   • Status: Not Start

• Project Plan
  • Project plan presentation
  • Project plan proposal
• Project Background Reading
  • See Bibliography below for links.
• Project Checkpoint
  • Project checkpoint presentation
• Paper Seminar Presentations
  • Background Presentation
  • Paper Summary
• Project Final Presentation
  • Poster
• Project Final Report
  • Final Report

Deep Learning Background
o LeCun, Yann, et al. “Gradient-based learning applied to document recognition.” Proceedings of the IEEE 86.11 (1998): 2278-2324.
o LeCun, Yann, et al. “Backpropagation applied to handwritten zip code recognition.” Neural computation 1.4 (1989): 541-551.
o Krizhevsky, Alex, Ilya Sutskever, and Geoffrey E. Hinton. “Imagenet classification with deep convolutional neural networks.” Advances in neural information processing systems. 2012.
o LeCun, Yann, Yoshua Bengio, and Geoffrey Hinton. “Deep learning.” nature 521.7553 (2015): 436. o Goodfellow, Ian, et al. Deep learning. Vol. 1. Cambridge: MIT press, 2016.

Feature / Landmark Detection
o Wang, Ching-Wei, et al. “Evaluation and comparison of anatomical landmark detection methods for cephalometric x-ray images: a grand challenge.” IEEE transactions on medical imaging 34.9 (2015): 1890-1900.
o Payer, Christian, et al. “Regressing heatmaps for multiple landmark localization using CNNs.” International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, Cham, 2016.
o Urschler, Martin, Thomas Ebner, and Darko Štern. “Integrating geometric configuration and appearance information into a unified framework for anatomical landmark localization.” Medical image analysis 43 (2018): 23-36.
o Sofka, Michal, et al. “Fully convolutional regression network for accurate detection of measurement points.” Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support. Springer, Cham, 2017. 258-266.
o Ronneberger, Olaf, Philipp Fischer, and Thomas Brox. “U-net: Convolutional networks for biomedical image segmentation.” International Conference on Medical image computing and computer-assisted intervention. Springer, Cham, 2015.
o Bulat, Adrian, and Georgios Tzimiropoulos. “Human pose estimation via convolutional part heatmap regression.” European Conference on Computer Vision. Springer, Cham, 2016.

X-Ray Simulation
o Badal, Andreu, and Aldo Badano. “Accelerating Monte Carlo simulations of photon transport in a voxelized geometry using a massively parallel graphics processing unit.” Medical physics 36.11 (2009): 4878-4880.
o Badal, Andreu, and Aldo Badano. “Monte Carlo simulation of X-ray imaging using a graphics processing unit.” Nuclear Science Symposium Conference Record (NSS/MIC), 2009 IEEE. IEEE, 2009.
o Bert, Julien, et al. “Geant4-based Monte Carlo simulations on GPU for medical applications.” Physics in Medicine & Biology58.16 (2013): 5593.
o Jia, Xun, et al. “A GPU tool for efficient, accurate, and realistic simulation of cone beam CT projections.” Medical physics39.12 (2012): 7368-7378.