Table of Contents

Cross Modality Medical Image Synthesis and Registration through Machine Learning

Last updated: Mar 23th, 2021

Summary

Magnetic Resonance Imaging (MRI) is commonly used for diagnosis of the osteonecrosis of the hip. Core decompression is a commonly used surgical method for the removal of the ostenecrotic tissue (dead bone) from the femoral head. In order to remove necrotic tissue in the femoral head, surgeons rely on preoperative MR scans for tool trajectory planning and take intraoperative X-ray shots to monitor the procedure. The aim of this project is to develop algorithms that allow us to register the planned paths from the preoperative MR images to intraoperative X-rays to assist surgeons during core decompression surgery.

Background, Specific Aims, and Significance

Background

Translating annotated paths in pre-operative MR to X-rays requires a robust registration method between MR images and X-ray images. Being able to visualize the drilling paths in intraoperative X-ray images would significantly help the surgeons during the core decompression surgery. Nevertheless, direct registration between MR images and X-ray images has been challenging due to the lack of structural information between the two modalities.

Specific Aims

The aim of this project is to create a workflow that allows this registration process to be achieved. In this work, a two-step registration method is proposed: (1) Synthesization of CT images given MR images as inputs (2) Registration between synthesized CT images and target X-ray images. The algorithms developed for the two steps will be tested and evaluated using the AVN Dataset, which consists of medical images of different image modalities from 30 patients.

Significance

The currently available MR to X-ray registration algorithms are mostly intensity- or feature-based methods, which are likely to work well on specific datasets and may not produce satisfactory results when applied to images focusing on another body part. This project provides a learning-based solution for the registration task. If the proposed method is successfully developed, it would be a more flexible approach that could work well on most datasets compared to existing methods as most model parameters would be learned during the training process.

Deliverables

Technical Approach

In order to perform MR to X-ray registration, the following two steps are performed. (Updated on Feb 24th)

Synthesize CT images from MR images

A cycle-consistent adversarial networks (CycleGAN) \cite{zhu_park_isola_efros_2017} is an unsupervised image-to-image translation network. Unlike other image translation network architectures which require paired images as the training dataset, CycleGANs learn from unpaired images and perform image translation in the absence of paired examples.

Fig.2 below illustrates the cycleGAN network architecture that would be developed in the project. Using the unpaired CT images and MR images from our datasets, four models would be trained, including the generator that synthesizes CT images from given MR images, the generator that synthesizes MR images from CT, the discriminator of CT images, and the discriminator of CT images.

The goal for this step is to produce a reliable MR to CT generator that could successfully generate realistic CT images from MR images as shown in Fig.3 below.

Progress

- 3/23 Update: The current result of the CT generator of the trained cycleGAN model is shown in the image below. Images in the first row are the input MR images and the images in the second row are the synthesized CT scans. (The CT scans are standardized during the pre-processing process so the values don't look exactly the same as original CT scans)

Registration between CT and x-ray (2D/3D Registration)

The second step of our workflow is considered as a 2D/3D registration process. In a 2D/3D registration problem, the objective is to find a rigid pose of 3D data so that it aligns with the target 2D image. Once the 3D CT images are generated from the CycleGAN network, we will then perform 2D/3D registration between the synthesized CT and our target X-ray image.

One of the most common approaches to performing 2D/3D registration is through the production of digitally reconstructed radiographs (DRRs) from CT images through ray-casting \cite{sherouse_novins_chaney_1990}. Since the DRR generation process simulates the X-ray capturing process, the registration task could be formulated as an optimization problem that attempts to find the ideal rigid CT poses ($\theta$) that produce DRRs with the highest closeness to the target X-ray image as shown in the equation below:

In this work, the optimization-based approach will be performed to tackle the 2D/3D registration between the synthesized CT and the X-rays. Methods such as multi-start strategies \cite{otake_wang_webster} could be implemented for better initial pose estimation to prevent the optimization process being trapped in local optima.

Dependencies

Milestones and Status

Status updated on Apr 27th, 2021.

  1. Milestone name: MR to CT synthesis - Dataset collection
    • Planned Date: 2/15
    • Expected Date: 3/01
    • Status:
      • 2/10: Collected AVN dataset. Still waiting for additional CT scans for more training data.
      • 2/22: Completed (Received approval to access NMDID CT dataset)
  2. Milestone name: MR to CT synthesis - Dataset preprocessing
    • Planned Date: 2/18
    • Expected Date: 2/18
    • Status:
      • 2/18: Completed for AVN dataset, but may require some adjustment base on the performance of the network (Ex: additional data augmentation). If the new dataset comes in, some preprocessing still needs to be done.
      • 3/01: Completed
  3. Milestone name: MR to CT synthesis - CycleGAN network architecture design
    • Planned Date: 3/08
    • Expected Date: 3/08
    • Status:
      • 2/24: 15% complete. Currently working on the training of a simple CycleGAN network without any modifications.
      • 3/08: Completed
  4. Milestone name: MR to CT synthesis - Network performance improvement
    • Planned Date: 3/15
    • Expected Date: 3/15
    • Status:
      • 3/08: 40% complete. Currently working on the modification of loss function to improve network performance. The impact of data augmentation will be tested soon.
      • 3/14: 60% complete. Data augmentation implemented. But multiple issues remain to be resolved, including some modification of preprocessing process (to improve result) and adjusting loss functions for better performance.
      • 3/23: Completed MIND loss function implemented. Training and testing dataset issue resolved.
  5. Milestone name: MR to CT synthesis - Validation and debugging
    • Planned Date: 3/21
    • Expected Date: 3/21 3/30
    • Status:
      • 3/14: 20% complete. Working on codes that calculate the similarity between paired images.　
      • 3/20: 50% complete. Similarity metrics implemented. However, a full model transformation model has to be implemented, which is a new task that was not previously expected. The current model trained also does not work significantly well on the test sets. Lots of training and debugging are still required in this situation, therefore I have decided to push back the validation process (and some corresponding training) to 3/30.
      • 4/27: Complete (With evaluation metricies implemented)
  6. Milestone name: CT to X-ray registration - Model dependency setup
    • Planned Date: 3/21
    • Expected Date: 3/21 3/30
    • Status:
      • 3/23: 100% complete. I have collected all the required codes and have the proper environment setup. Bue the codes have not been tested yet.
  7. Milestone name: CT to X-ray registration - Model training and evaluation
    • Planned Date: 4/18
    • Expected Date: 5/6
    • Status:
      • 4/16: 30% Running into issues while training with large spacing synthesized CTs. May adjust final deliverable due to this issue.
      • 4/27: *Removed*
  8. Milestone name: Code optimization - Combination of model loss
    • Planned Date: 4/29
    • Expected Date: 4/29
    • Status: *Removed*
  9. Milestone name: Code optimization - Code cleanup and documentation
    • Planned Date: 4/29
    • Expected Date: 5/06
    • Status:
      • 4/27: 30% Complete
      • 5/5: 60% Complete - (I will still be adding helping functions and try to create a Github Wiki to walkthrough potential users how to run the code after the presentation on 5/7)
  10. Milestone name: Final report writeup and presentation
    • Planned Date: 5/06
    • Expected Date: 5/06
    • Status: Completed

Reports and presentations

Project Bibliography

Other Resources and Project Files

Here give list of other project files (e.g., source code) associated with the project. If these are online give a link to an appropriate external repository or to uploaded media files under this name space (2021-10).