======3D Imaging, Analysis, and Guidance for Robotic-Assisted Ankle Fracture / Dislocation Surgery======
**Last updated: 4/20/2020 at 11:56PM**


======Summary======
This project concerns developing and validating an image guidance framework for application to a robotic-assisted fibular reduction in ankle fracture surgery. The aim is to produce and demonstrate proper functioning of software for automatic determination of directions for fibular repositioning with the ultimate goal of application to a robotic reduction procedure that can reduce the time and complexity of the procedure as well as provide the benefits of reduced error in ideal final fibular position, improved syndesmosis restoration and reduced incidence of post-traumatic osteoarthritis. The focus of this product will be developing and testing the image guidance software, from the input of preoperative images through the steps of automated segmentation and registration until the output of a final transformation that can be used as instructions to a robot on how to reposition the fibula, but will not involve developing or implementing the hardware of the robot itself. 


  * **Students:** Anthony Wu, Asef Islam, Jayaram Mandavilli
  * **Mentor(s):** Jeff Siewerdsen, Wojtek Zbijewski

{{:courses:456:2020:projects:456-2020-10:screen_shot_2020-02-21_at_11.17.58_am.png?200|}}

======Background, Specific Aims, and Significance======
Ankle fractures occur with a frequency of around 174 cases per 100,000 adults per year,
with over 5 million yearly cases in the U.S. alone (Goost et al), affecting mainly young active
people and the elderly. Ankle fractures most commonly involve a fracture in the lower fibula
which can also result in disruption of the syndesmosis, or the alignment of other bones and
ligaments within the ankle joint, if the fibula is displaced. This is due to the displacement of the
lower fibula causing damage and forceful shifting of these ligaments and other connective tissue.
This can lead to many long-term complications including post-traumatic osteoarthritis (PTOA)
if the proper syndesmosis is not restored.

The proximal fracture can be treated with relatively little difficulty such as by fixation
with metal screws. However, the second part of the surgery is fibular reduction, or movement of
the lower fibula back into proper position to reconstruct the joint and restore syndesmosis. The
current standard of care for this procedure is visual estimation by the surgeon to determine
where to place the fibula and screw-based fixation of syndesmosis. In one study over 20% of
patients who underwent reduction via this method were shown through CT scans shortly
afterwards to have syndesmotic malreduction, which was defined as a greater than 2 mm
widening of syndesmosis compared to the patient’s other, healthy ankle (Nagvi et al). The most
common reason for re-operation in the weeks immediately following the initial ankle fracture
surgery is syndesmotic malreduction (Ovaska et al). Furthermore, the incidence of PTOA in
ankle fracture patients is as high as 70% (Mehta et al). Evidently, the standard of care for
reduction is inadequate for proper syndesmotic restoration and prevention of PTOA.
======Deliverables======
 
  * **Minimum:** (Expected by 3/11)
    - Working coupled Active Shape Model (cASM) with an average error of 5 mm for high-quality cone beam images
      - Our team will first segment healthy ankle C-arm images without metal artifacts. After consulting with mentors, our segmentation error for each bone should not exceed a mean of 5mm.
      - The purpose of this deliverable is to ensure that our ASM model is working correctly, which we plan to use as a layer of our neural network for the deep learning segmentation model.
    - Perform error analysis and show failure modes on low-quality CT images
      - Our mentors do not expect the cASM model to perform adequately for C-arm images with metal artifacts. As a result, our team will create an analysis that documents why the cASM model cannot be implemented on its own.

  * **Expected:** (Expected by 4/11)
    - Working deep learning - cASM segmentation model
      - Our team will first segment healthy ankle C-arm images. After consulting with mentors, our segmentation error for each bone should improve to a mean of 2mm.
      - Our team will next apply our deep learning model to unhealthy ankle C-arm images (this is dependent on if unhealthy ankle images can be obtained remote). After consulting with mentors, our segmentation error for each bone should not exceed a mean of 2mm.

  * **Maximum:** (Expected by 5/6)
    - Incorporate ankles with metal artifacts into deep learning - cASM model with average segmentation error of 2 mm
      - We will apply a metal reduction algorithm to the Dicom ankle images with metal artifacts and adjust the model in order to incorporate this data.
      - Due to the circumstances, the data available is limited, that is why a metal reduction algorithm is needed.
    - Written Paper

======Technical Approach======
Steps in Approach:

1. Automated Segmentation

The first step in an image-guided approach is fast, automated segmentation of the
pre-operative images in order to identify and save the relevant anatomical features, in this case
the fibula and other bones of the ankle. A promising existing approach to automated
segmentation is active shape models (ASMs) (Brehler et al). An ASM model is trained by taking
in as input a training set of several segmentations of a single bone from different patients and
then performing a PCA-based analysis to identify and characterize the principal modes of
variation in the surface morphology of the bone within the population of training samples. The
model comprises a set of basis functions for each principal component of variation, which can be
tuned and create a deformable template that can be mapped onto any new example by
minimizing residuals. Thus, this can be applied to automated segmentation by using this
template object in order to map onto a new image and segment the bone given some
initialization.

ASMs can be improved even further by updating them to Coupled Active Shape Models
(cASMs). Traditional ASMs are trained independently for individual bones and separately
initialized, however as a result they are susceptible to errors in the narrow articular joint spaces
between bones if there is poor contrast in the image and can result in producing segmentations
with overlap in adjacent bones. cASMS are trained with consideration of multiple bones
simultaneously and incorporate the spatial relations and articular joint widths between adjacent
bones as features as well, and thus they are able to improve on the accuracy of segmentation
within these joint spaces and prevent overlaps.

While in this work the cASMs were evaluated in high-resolution cone-beam CT (CBCT)
images, they have not yet been tested in pre-operative C-arm images for 2D-3D reconstruction.
Thus, an important step in adopting the cASM approach for this project will be to evaluate its
accuracy in segmenting C-arm images and improving it to do so, including improving the
initializations and adding corrections for noise and artifacts.

A deep learning approach will also be taken, as it is likely that the ASM model will not
provide sufficient segmentation accuracy. It will be used to also conduct image segmentation,
like the ASM. The purpose of this is to build this model to hopefully get a higher segmentation
accuracy than the ASM model. The final approach chosen will be the one that yields the highest
accuracy. It is possible that the final model chosen is a combination of the ASM and the neural
network. The plan for the deep neural network is to utilize a convolutional neural network
(CNN). These are commonly used for image processing in a lot of domains. The specifics of the
neural network are currently unknown. However, there are some major characteristics that will
most likely be implemented. Maximum pooling will most likely be used over average pooling as
it is known to be more effective in image classification. This is because this technique focuses on
the most important characteristics in the image. This is helpful in classification because it does
not let other, less significant, pixels affect the classification and only looks at what is really
important. Pooling is also beneficial because it is able to focus on what is really important. This
reduces the computational burden of training the neural net. Other techniques such as dropout.
Another strategy that could be used is increasing the number of epochs. This will need to be
carefully managed because if intraoperative c-arm data is being utilized, the segmentation will
need to be determined quickly. The model will need to be fine-tuned, which will be an ongoing
process. The goal is to achieve high accuracy, the exact accuracy goals for the deep learning
model can be seen in the deliverables section.

Whichever final approach is used for automated segmentation, it would be used to
automatically segment the ankle bones of both of the patient’s ankles (the healthy one and the
injured one). The model, or combination of models, which yields the highest segmentation
accuracy (as compared to manually segmented images through minimizing square residuals) will be
used.

2. Metal Artifacts

After segmenting healthy ankles, ankles with metal artifacts will be looked at. Since there is a lack of data with metal artifacts and it is not currently possible to obtain more of it, metal reduction algorithms will be applied to this data. Then this data will be incorporated into the final segmentation model.
======Dependencies======

  * **Feedback from Mentors**
      * Plan: Meet with mentors via zoom every Friday between 3:30pm and 5pm
      * Backup: Communicate via email and schedule meeting times outside of standard meeting times
      * Point of Contact: Dr. Siewerdsen and Dr. Zbijewski
      * Date Solved: 02/07/2020
  * **Imaging Libraries**
      * Plan: Utilize already established MATLAB packages, such as Image Processing Toolbox, Computer Vision Toolbox, and Robotics System Toolbox
      * Backup: N/A
      * Point of Contact: N/A
      * Date Solved: 02/07/2020
  * **Deep Learning Libraries**
      * Plan: Utilize already established Python packages (Tensorflow, PyTorch)
      * Backup: N/A
      * Point of Contact: N/A
      * Date Solved: 02/07/2020
  * **Version Control**
      * Plan: Utilize GitHub to backup code
      * Backup: Use a backup hard drive
      * Point of Contact: N/A
      * Date Solved: 02/07/2020
  * **Segmented C-Arm Images**
      * Plan: Segment already generated C-Arm data with iStar’s segmentation software
      * Backup: Use C-arm provided by the lab to get data that can be segmented with segmentation software. This will require radiation training with Dr. Siewerdsen
      * Point of Contact: Dr. Qian Cao
      * Date Needed: 03/11/2020
  * **Computers (GPU Array for Deep Learning)**
      * Plan: Work entirely from personal computers and GPUs
      * Backup: Connect remotely to lab computers and GPU clusters from personal computers
      * Point of Contact: Dr. Siewerdsen
      * Date Needed: 03/11/2020
  * **Validation Data**
      * Plan: Use Cadaver ankles at Carnegie as test
      * Backup: Use phantom model ankles
      * Point of Contact: Dr. Siewerdsen
      * Date Needed: 04/02/2020 (only have healthy ankles)
  * **C-Arm Mentoring**
      * Plan: Find member in iStar lab to help our team perform imaging on C-Arm
      * Backup: Obtain training of C-arm ourselves by performing online training
      * Point of Contact: Niral Sheth
      * Date Needed: 04/02/2020

======Milestones and Status ======
  - Milestone name:  ASM Model within 5mm Accuracy for CBCT Healthy Ankle Images
    * Planned Date: 03/11
    * Expected Date: 03/11
    * Status: Complete
  - Milestone name:  DL-cASM Model within 2mm Accuracy for C-Arm Healthy Ankle Images
    * Planned Date: 4/11
    * Expected Date: 4/11
    * Status: In Progress
  - Milestone name:  DL-cASM model within 2mm Accuracy for Healthy Ankle Images with Metal Artifacts
    * Planned Date: 5/1
    * Expected Date: 5/1
    * Status: Incomplete

======Reports and presentations======

  * Project Plan
    * {{ :courses:456:2020:projects:456-2020-10:project_plan_presentation.pdf | Project Plan Presentation}}
    * {{ :courses:456:2020:projects:456-2020-10:project_plan.pdf | Project Plan}}
  * Project Checkpoint
    * {{ :courses:456:2020:projects:456-2020-10:checkpoint_presentation.pdf | Project Checkpoint Presentation }}
  * Paper Seminar Presentations - Asef Islam
    * {{ :courses:456:2020:projects:456-2020-10:nihms-1041163.pdf |}}
    * {{ :courses:456:2020:projects:456-2020-10:seminar_asef_islam.pdf |}}
    * {{ :courses:456:2020:projects:456-2020-10:critical_review_asef_islam.pdf |}}
  * Paper Seminar Presentations - Jayaram Mandavilli
    * [[https://ieeexplore-ieee-org.proxy1.library.jhu.edu/stamp/stamp.jsp?tp=&arnumber=8681706|NAS-Unet: Neural Architecture Search for Medical Image Segmentation]]
    * {{ :courses:456:2020:projects:456-2020-10:cis_seminar_presentation_updated.pdf | Seminar Presentation}}
    * {{ :courses:456:2020:projects:456-2020-10:critical_review.pdf | Critical Review}}
  * Paper Seminar Presentations - Anthony Wu
    * {{ :courses:456:2020:projects:456-2020-10:seminar_paper_-_metal_reduction_with_polyenergetic_kcr.pdf |}}
    * {{ :courses:456:2020:projects:456-2020-10:anthony_wu_critical_review.pdf |}}
    * {{ :courses:456:2020:projects:456-2020-10:anthony_wu_-_seminar_presentation.pdf |}}
  * Project Mini Checkpoint
    * {{ :courses:456:2020:projects:456-2020-10:mini_checkpoint_presentation_4_30.pdf | Mini Checkpoint Presentation}}
  * Project Final Presentation
    * {{ :courses:456:2020:projects:456-2020-10:final_presentation.pdf | Final Presentation}}
  * Project Final Report
    * {{ :courses:456:2020:projects:456-2020-10:final_report.pdf | Final Report}}

======Project Bibliography=======

[1] M. Brehler, A. Islam, L. Vogelsang, D. Yang, W. Sehnert, D. Shakoor, S. Demehri, J. H. Siewerdsen, W. Zbijewski, “Coupled Active Shape Models for Automated Segmentation and Landmark Localization in High-Resolution CT of the Foot and Ankle”

[2] Goost H, Wimmer MD, Barg A, Kabir K, Valderrabano V, Burger C. Fractures of the ankle joint: investigation and treatment options. Dtsch Arztebl Int. 2014;111(21):377–388. doi:10.3238/arztebl.2014.0377

[3] Ovaska MT, Mäkinen TJ, Madanat R, Kiljunen V, Lindahl J. A comprehensive analysis of patients with malreduced ankle fractures undergoing re-operation. Int Orthop. 2014;38(1):83–88. doi:10.1007/s00264-013-2168-y

[4] Mehta, Saurabh Sagar, et al. “Understanding Risks and Complications in the Management of Ankle Fractures.” Indian Journal of Orthopaedics, Medknow Publications & Media Pvt Ltd, Sept. 2014, www.ncbi.nlm.nih.gov/pmc/articles/PMC4175856/.

[5] Naqvi, G. A., Cunningham, P., Lynch, B., Galvin, R., & Awan, N. (2012). Fixation of Ankle Syndesmotic Injuries: Comparison of TightRope Fixation and Syndesmotic Screw Fixation for Accuracy of Syndesmotic Reduction. The American Journal of Sports Medicine, 40(12), 2828–2835. https://doi.org/10.1177/0363546512461480

[6] Y. Otake, R. J. Murphy, R. B. Grupp, Y. Sato, R. H. Taylor, and M. Armand, “Comparison of Optimization Strategy and Similarity Metric in Atlas-to-subject Registration Using Statistical Deformation Model”, in Proc. SPIE 9415, Medical Imaging 2015: Image-Guided Procedures, Robotic Interventions, and Modeling, San Francisco, 8-10 Feb., 2015. p. 94150Q. 10.1117/12.2081310

[7] Weng, Y., Zhou, T., Li, Y., & Qiu, X. (2019). NAS-Unet: Neural Architecture Search for Medical Image Segmentation. IEEE Access, 7, 44247–44257. doi: 10.1109/access.2019.2908991

[8] Kuo-Sheng Cheng, Jzau-Sheng Lin and Chi-Wu Mao, "The application of competitive Hopfield neural network to medical image segmentation," in IEEE Transactions on Medical Imaging, vol. 15, no. 4, pp. 560-567, Aug. 1996.

[9] Xu, S., Uneri, A., Jay Khanna, A., et al. Polyenergetic known-component CT reconstruction with unknown material compositions and unknown x-ray spectra. Physics in Medicine & Biology 62 (8), 3352–3374 (2017).

======Other Resources and Project Files======
CODE:
https://github.com/jmandavilli/CIS2

DATA:
https://livejohnshopkins-my.sharepoint.com/:f:/r/personal/awu42_jh_edu/Documents/C-Arm?csf=1&web=1&e=o3fQIi