Last updated: 02/13/2020

Note: This project was changed in mid-semester, in response to the COVID-19 outbreak. The new project is “A County-level Dataset for Informing the United States' Response to COVID-19”, and its web page can be found here.

Develop a method that improves pelvis landmark detection for intraoperative 2D to 3D registration on real, scarce X-rays that takes advantage of simulated, readily available X-rays.

• Students: Benjamin D. Killeen
• Mentor(s): Cong Gao and Mathias Unberath

Minimally invasive hip surgery is a desirable method for many patients. Although its benefits remain controversial with regard to pain management and recovery time, many patients strongly prefer a smaller incision to more traditional hip surgery. Unfortunately, these cosmetic advantages translate to additional complexity for the surgeon. Minimally invasive hip surgery requires navigation in and manipulation of anatomical structures which are underneath unbroken skin and thus not reliably visible to the surgeon. At the same time, correctly aligning the cup and stem is crucial to the operation's success. In the past, this has been achieved by using the minimal incision as a “mobile window” for identifying anatomical landmarks, but this can result in unreliable outcomes.

Alternatively, fluoroscopic imaging provides intraoperative 2D visualization of the hip anatomy, but it presents its own set of challenges. First and foremost, the mental interpretation of 2D X-ray images places an undesirable burden on the surgeon, at a time when her chief concern should be correctly aligning the hip. Computer-assisted tracking systems overcome the requirement for mental 2D/3D registration of the image with the anatomy. Based on the fluoroscopic image of the hip, they can automatically track desired objects and display their poses in the context of a preoperative plan.

The systems we investigate here involve the registration of intraoperative 2D fluoroscopic images with a 3D preoperative model. Improper initialization of traditional registration algorithms, such as Iterative Closest Point (ICP) and its many variants, can lead to large registration errors. This is because of the numerous local minima which may exist in the cost optimization's function. A better “first guess” makes it much more likely that ICP converges on the actual minimum, a reliable registration. This first guess typically takes the form of human input, identifying anatomical landmarks in the hip anatomy. Yet human input is undesirable for two reasons. First and foremost, the time required for human landmark annotation is not insignificant. Even a 4-5 second delay interrupts the surgical procedure, resulting a disjointed alignment process. Sub-second registration, on the other hand, would allow more continuous adjustment of the cup and stem.

Prior work has shown that deep learning (DL) based techniques can identify anatomical landmarks in a fast and reliable manner, in order to initialize an ICP algorithm. Unlike ICP algorithms, deep neural networks (DNNs) learn generalizable features from labeled training data, and use them to interpret previously unseen images. For example, use simulated images to generate arbitrarily large training data with perfectly known ground truth anatomical landmarks. They show that a multi-stage DNN trained on these simulated images can generalize well to real-world images but are susceptible to scenarios not seen during training. A surgical tool which occludes the image can severely compromise the DNN's ability to detect anatomical landmarks. Since the goal of automatic landmark detection is continuous, intraoperative feedback for the surgeon, it would be impractical for the surgeon to withdraw her tools during every registration. Thus, we aim to improve the generalization of DNNs from simulation to real-world scenarios which are not encountered during training.

There are many possible approaches for improving sim-to-real generalization. We propose using a novel patch-normalized convolution (PNC) layer, which constrains feature descriptors to a local region at every scale, described in the Technical Approach of the project plan. Based on preliminary results, PNC shows an improved ability to generalize to unseen types of noise, especially additive noise patterns and contrast adjustments. We anticipate that DNNs which employ PNC will be particularly effective for occlusions by surgical tools due to the high contrast between these tools and typical intensities for an unoccluded X-ray.

Deliverables:
---------- ---------------- --------------------------------------------------------------------------------------------------
           Algorithm        DNN for landmark detection.
           Implementation   PyTorch Implementation, Made Public on [GitHub](https://github.com)
 Minimum   Validation       Anatomical landmark detection results on real data, matching prior work.
           Documentation    Inline code documentation.
           Presentation     Final written report, in-class presentation.
---------- ---------------- --------------------------------------------------------------------------------------------------
           Algorithm        DNN for landmark detection **using PNC**.
           Implementation   PyTorch implementation, made public on [GitHub](https://github.com), **ready for academic use**.
 Expected  Validation       Anatomical landmark detection results on real data, **exceeding prior work**.
           Documentation    **Organized and complete code** documentation.
           Presentation     Final written report, in-class presentation.
---------- ---------------- --------------------------------------------------------------------------------------------------
           Algorithm        DNN for landmark detection **using PNC**.
           Implementation   PyTorch implementation, made public on [GitHub](https://github.com), ready for academic use.
 Maximum   Validation       Anatomical landmark detection results on real data **with demonstrable generalization**.
           Documentation    Organized, complete code documentation, final report, **academic publication**.
           Presentation     Final written report, in-class presentation, **academic publication**.
---------- ---------------- --------------------------------------------------------------------------------------------------

Much of the recent success in computer vision is due to the advent of the deep convolutional neural network, which has at its core the convolutional layer. We propose to apply DNN architectures based on prior work to anatomical landmark detection, incorporating a novel type of convolutional layer, the Patch-normalized Convolution (PNC). We hypothesis that the spatially local nature of the PNC layer as well as its robustness to noise will enable greater generalization to real X-ray data. We refer to and for a discussion of the U-Net and stage-based DNN architectures, respectively, which we will employ for landmark detection.

State-of-the-art DNNs, including the afforementioned U-Net and stage-based network, usually pair a convolutional layer with a normalization layer. In our project proposal, we review these concepts briefly in order to lay the groundwork for PNC, which combines a convolution with a kernel-dependent normalization. Please see project_proposal.pdf for a detailed mathematical formulation.

Our primary dependencies are simulated and real fluoroscopic images of the hip with anatomical landmarks. Simulated X-ray data has been used in an ongoing manner by Cong Gao. Fortunately, these are already resolved. The real X-ray data requires some formatting, for which Robb Grupp is an ongoing contact. Additionally, we are heavily reliant on advanced computational resources for experimentation and ablation studies of any proposed method. The MARCC compute cluster is a reliable high-compute system with multiple redundancies for high-capacity data storage. Alternatively, we have guaranteed access to two personal workstations with high-speed SSD primary drives and high-capacity HDD backup data drives. Any code, documentation, or statistical results are version-controlled and backed up using GitHub.

Recently, based on, we realized it might be of academic interest to evaluate our method's generalization ability to images which are occluded by surgical tools in a previously unseen manner. Although this is not core to our aim of improving sim-to-real generalization, it is nevertheless of interest. Therefore the effort to obtain real images with surgical tool occlusions is ongoing.

Dependency                                         Solution                                                     Alternative                                   Status
-------------------------------------------------- ------------------------------------------------------------ --------------------------------------------- -----------------------------------------------------
Anatomical Landmark Detection Software             `Generalizing_Pelvis_Landmark_Detection` Repository Access                                                 Solved
DeepDRR Dataset of Simulated Fluoroscopic Images   Transfer from Cong Gao                                                                                  NA On Personal Workstation
Computational Resources (GPU)                      MARCC Cluster Access                                                 Personal Workstations (3x total GPUs) Allocation Granted
Real X-ray Images for Testing                      [Robb Grupp](mailto:grupp@jhu.edu)                                                                      NA On BIGSS Shared Drive
Real X-ray Images with Occlusions (new)            Authors of(mailto:unberath@jhu.edu) IN PROGRESS
Efficient PNC PyTorch Implementation               [Xingtong Liu](mailto:XingtongLiu@jhu.edu)                                                              NA Solved
-------------------------------------------------- ------------------------------------------------------------ --------------------------------------------- -----------------------------------------------------

Milestone                                    Date    Status
------------------------------------------- ------- --------
Obtain simulated X-ray data from Cong Gao    02/15    Done
Obtain Real X-ray data from Robb Grupp       02/11    Done
Finalize simulation training pipeline        03/01  
Finalize Real X-ray validation pipeline      03/07  
Finalize DNN architecture/algorithm          03/21  
Finish ablation study                        04/14  
Finish statistical analysis                  04/21  
Presentation                                 05/05  
Final report                                 05/15  
Academic publication                          TBD   
------------------------------------------- ------- --------

}

• Project Plan
  • Project plan presentation
  • Project plan proposal
• Project Background Reading
  • See Bibliography below for links.
• Critical Review
  • Paper: "Learning to Detect Anatomical Landmarks of the Pelvis in X-rays from Arbitrary Views
  • Paper Presentation
  • Critical Review
• Project Checkpoint
  • Project checkpoint presentation
• Paper Seminar Presentations
  • here provide links to all seminar presentations
• Project Final Presentation
  • PDF of Poster
• Project Final Report
  • Final Report
  • links to any appendices or other material

[1] H. Roth et al., “A new 2.5 D representation for lymph node detection in CT,” The Cancer Imaging Archive, 2015.

[2] A. Khan, A. Sohail, U. Zahoora, and A. S. Qureshi, “A Survey of the Recent Architectures of Deep Convolutional Neural Networks,” arXiv:1901.06032 [cs], Feb. 2020.

[3] R. Grupp et al., “Automatic Annotation of Hip Anatomy in Fluoroscopy for Robust and Efficient 2D/3D Registration,” arXiv:1911.07042 [cs, eess], Nov. 2019.

[4] S. Ioffe and C. Szegedy, “Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift,” arXiv:1502.03167 [cs], Mar. 2015.

[5] K. He, X. Zhang, S. Ren, and J. Sun, “Deep Residual Learning for Image Recognition,” presented at the Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 770–778.

[6] M. Unberath et al., “Enabling machine learning in X-ray-based procedures via realistic simulation of image formation,” Int J CARS, vol. 14, no. 9, pp. 1517–1528, Sep. 2019, doi: 10.1007/s11548-019-02011-2.

[7] Y. Wu and K. He, “Group Normalization,” presented at the Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 3–19.

[8] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNet Classification with Deep Convolutional Neural Networks,” in Advances in Neural Information Processing Systems 25, F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger, Eds. Curran Associates, Inc., 2012, pp. 1097–1105.

[9] D. Ulyanov, A. Vedaldi, and V. Lempitsky, “Instance Normalization: The Missing Ingredient for Fast Stylization,” arXiv:1607.08022 [cs], Nov. 2017.

[10] J. L. Ba, J. R. Kiros, and G. E. Hinton, “Layer Normalization,” arXiv:1607.06450 [cs, stat], Jul. 2016.

[11] B. Bier et al., “Learning to detect anatomical landmarks of the pelvis in X-rays from arbitrary views,” Int J CARS, vol. 14, no. 9, pp. 1463–1473, Sep. 2019, doi: 10.1007/s11548-019-01975-5.

[12] A. Malik and L. D. Dorr, “The Science of Minimally Invasive Total Hip Arthroplasty,” Clinical Orthopaedics and Related Research®, vol. 463, pp. 74–84, Oct. 2007, doi: 10.1097/BLO.0b013e3181468766.

[13] O. Ronneberger, P. Fischer, and T. Brox, “U-Net: Convolutional Networks for Biomedical Image Segmentation,” in Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015, Cham, 2015, pp. 234–241, doi: 10.1007/978-3-319-24574-4_28.

[14] M. Woerner et al., “Visual intraoperative estimation of cup and stem position is not reliable in minimally invasive hip arthroplasty,” Acta Orthopaedica, vol. 87, no. 3, pp. 225–230, May 2016, doi: 10.3109/17453674.2015.1137182.

[15] B. Bier et al., “X-ray-transform Invariant Anatomical Landmark Detection for Pelvic Trauma Surgery,” arXiv:1803.08608 [cs], Mar. 2018.

Here give list of other project files (e.g., source code) associated with the project. If these are online give a link to an appropriate external repository or to uploaded media files under this name space (2020-03).