Last updated: 18:21, April 3, 2022

Eustachian tube dysfunction is a disorder resulting from impairment in middle ear ventilation and pressure regulation. Patients affected with experience a myriad of symptoms such as ear pain, pressure, clicking, and difficulty hearing. Eustachian tube dilation allows for the surgical management of this condition but to date, there is a lack of a registration-based image-guided surgical system that utilizes CT segmentations. As a first step in the development of this surgical system, we aim to utilize deep learning to develop a platform for automated segmentation of the eustachian tube.

• Students: Ameen Amanian, Yuliang Xiao, Chanha Kim
• Mentor(s): Francis Creighton, Mathias Unberath, Russell Taylor, Manish Sahu

Figure 1 - Anatomy of the Eustachian tube with Near-by Critical Structures

Eustachian tube dilation is a procedure approved for the surgical management of eustachian tube dysfunction (ETD). ETD results from impairment of middle ear ventilation and pressure regulation. As a result, patients experience a range of symptoms ranging from ear pain, pressure, cracking, or difficulty hearing which has a significant impact on patients’ quality of life [1]. Given the proximity of the eustachian tube to certain critical structures such as the internal carotid artery, it is important to carefully review the preoperative scans and assess for anatomical variations present amongst patients. Currently, existing registration-segmentation propagation pipelines has varying accuracy and can be computationally expensive. Furthermore, there is a lack of image-registration surgical navigation system utilizing automated segmentations of preoperative CT’s for this procedure. Therefore, we aim to assess the utility of and develop a deep learning pipeline to perform automated segmentation of the eustachian tube, define near-by critical structures, and establish the first pipeline that can be integrated into a surgical navigation system. A paragraph or so here.

The specific aims for developing of the deep learning pipeline include:

1. Utilize a supervised learning platform for automated segmentation of the Eustachian tube
2. Validate the pipeline predictions in comparison with the ground truth
3. Explore unsupervised methods for image registration and segmentation

Explain significance

• Minimum: (Expected by March 30th)
  1. CT registration script
  2. Dataset containing ground truth segmentations
  3. Preprocessed Dataset of CT Heads
  4. Trained nnU-Net pipeline
  5. Documentation
  6. Final Report

• Expected: (Expected by April 30th)
  1. Validation script calculating metrics on the predicted segmentations (WHD, DSC, SD)
  2. Heat map script demonstrating point to point difference between ground truth and predicted segmentations

• Maximum: (Expected by July 31st)
  1. Trained VoxelMorph or DeepReg imaging registration model
  2. Validation script comparing nnUNet model with VoxelMorph/DeepReg image registration models
  3. Conference Presentation and Manuscript Draft

The proposed pipeline will utilize a ground truth dataset which will be co-registered as part of the preprocessing. The data will then serve as input to the deep learning pipeline for performing semantic segmentation. The proposed deep learning algorithms include nnUNet, VoxelMorph, and DeepReg. The predicted segmentations will then be validated in comparison with the ground truth (Figure 1).

i. Data Preprocessing

In order to make the data compatible with the nnUNet, VoxelMorph and DeepReg frameworks, the raw images are required to be co-registered. We will be using ANTsPy, a Python library which includes blazing-fast IO, registration, segmentation, statistical learning, and visualization functionalities amongst others [1]. For this, we will randomly choose the first image as a template, register the remaining images to the template, and apply the ‘forward’ deformation field to each image to ensure co-aligned within the dataset. Furthermore, this will confirm that images have the same rotation, angle, and spacing.

ii. nnU-Net Algorithm

nnU-net algorithm is the first segmentation method designed to deal with dataset diversity found in the medical image segmentation domain [2]. For our project, we will be focusing on using nnU-net as the basis for our deep learning model for semantic segmentation of CT images (Figure 2). The workflow of nnU-net is as follows: nnU-net first uses its novel heuristic rule to determine the data-dependent hyperparameters, or data fingerprints, to automatically ingest the training data set. The blueprint parameters (such as loss function, and network architecture), inferred parameters (such as image resampling and batch size) along with the data fingerprint generate the pipeline fingerprints. The pipeline fingerprints then form network training for 2D, 3D, and 3D-Cascade U-Net using the hyperparameters determined so far. Using post-processing and ensembling strategies (i.e., assigning weights to each model and combining them together), the best configuration will be used by nnU-net to produce the final prediction.

A motivation behind using nnU-net is due to its ability to handle a wide variety of target structures [2]. Unlike other deep learning models, nnU-net is not a specialized solution for a certain type of data set, but rather an algorithm that is generalizable and has proven to surpass most existing approaches for data segmentation tasks. Furthermore, nnU-Net has a self-configuring ability in that it allows us to quickly train and use the model, which is computationally feasible. Finally, the results can serve as a benchmark that can be improved upon if the training is not successful.

Proposed workflow is shown in figure 3. First, we will obtain and preprocess our training (as discussed in previous section) and generate test data sets manually and feed them into the nnU-net. Then, we will see if the trained model generalizes well to test data (i.e., high Mean Surface Distance), and if yes (which is not likely at the first trial) we can move on to building and training other unsupervised models such as VoxelMorph and DeepReg and compare the results. If the training with nnU-net is not successful, we will consider our initial result as a baseline and attempt the following three modifications to the nnU-net algorithm. First, we will try CT-specific pre-processing which includes denoising, CT data interpolation with different splines, CT data registration, and finally windowing to increase the contrast across a region of interest which all can be improved to target CT data sets. Second, we will try manual adaptation of the loss function. nnU-net uses a dice loss (region-based loss function) OR a cross-entropy loss (distribution-based loss function) but we can try to cascade the distribution based loss and region based loss, or even try giving weights to the background area of the label, which can soften the hard label used in loss functions. This can result in a regularization effect, increasing the robustness of the model, lowering the chances of overfitting as suggested in recent research papers [4] that are trying to improve the dice loss for segmentation tasks. Finally, we can extend or modify the heuristics used in nnU-net as suggested in the original nnU-net paper if the training fails, because the current heuristics may not be generalized enough to handle our domain-specific head CT scans.

iii. VoxelMorph Algorithm VoxelMorph is a fast unsupervised-learning-based framework for deformable, pairwise medical image registration [3]. Compared to traditional registration methods, it treats registration as a function to map paired input images to a deformation field that make them aligned. Registration is formulated as an objective function and used in the convolution neural network to build the model that can optimize this function. In this algorithm, the first setting includes training the model to maximize standard image matching objective functions that are based on the image intensities. In the second setting, the auxiliary segmentations are leveraged in the training data, which increase accuracy when predicting on test datasets.

iv. nnUNet Model Validation

One measure used for model validation includes the dice similarity coefficient (DSC), a scoring system which measures volumetric overlap between two images. However, as the eustachian tube is a very thin structure, the conventional use of DSC is not appropriate for our project due to the Eustachian tube’s long narrow structure. Thus, we will use the following metrics that capture the structure similarity.

Heat Map

Compute the closest distance between each vertex of prediction mesh to ground truth mesh, draw the heat map about this closest distance with the prediction mesh.

Mean Surface Distance

The mean surface distance, dmean, is the distance between the the surface (S) and the reference surface (Sref) where d(S,Sref) is the mean of distances between every surface voxel in S and the closest surface voxel in Sref, while d(Sref,S) is computed in a similar way.

Weighted Hausdorff Distance (WHD)

The maximum Hausdorff distance (HD) is the maximum distance of a set to the nearest point in the other set. More formally, the maximum Hausdorff distance from set X to set Y is a max-min function, defined as: WHD is similar to the maximum HD; however, it is based on the probability map for the region of interest. Larger weight will take more concerns and vice versa. The purpose for using WHD is to make the clinician focus on the important parts or the parts they expect to observe.

describe dependencies and effect on milestones and deliverables if not met

1. CT head dataset:
   • Planned Date: February 11, 2022
   • Expected Date: February 11, 2022
   • Status: 100% - Obtained access to 44 de-identified head CT's.
2. Set-up GPU training environment:
   • Planned Date: February 20th, 2022
   • Expected Date: February 20th, 2022
   • Status: 100% - Remote access obtained.
3. Develop Ground truth labels:
   • Planned Date: February 28, 2022
   • Expected Date: March 30, 2022
   • Status: 100% - Finish manual segmentation of the eustachian tube on 3D slicer.
4. Image Pre-processing (registration):
   • Planned Date: February 24, 2022
   • Expected Date: February 24, 2022
   • Status: 100% - Developed scripts on ANTsPy to perform image registration + label propagation to ensure all are aligned prior to being input to nnU-Net.
5. Implement script to 1) split data into training and testing sets 2) Generate file structures required for nnU-Net:
   • Planned Date: February 25, 2022
   • Expected Date: February 25, 2022
   • Status: 100%
6. Develop nnU-Net pipeline:
   • Planned Date: March 13, 2022
   • Expected Date: March 13, 2022
   • Status: 100%
7. nnUNet Internal Validation:
   • Planned Date: March 30, 2022
   • Expected Date: March 30, 2022
   • Status: 75% - Developed scripts for DSC, AHD, and heat map visualization. Working on developing a weight Hausdorff Distance function.
8. Develop VoxelMorph/DeepReg Registration + Segmentation Pipeline:
   • Planned Date: April 15, 2022
   • Expected Date: April 15, 2022
   • Status: 10% - Exploring VoxelMorph Pipeline
9. Comparison of nnUNet with VoxelMorph and accuracy reporting.:
   • Planned Date: April 30, 2022
   • Expected Date: April 30, 2022
   • Status: 0%
10. Final Report:
    • Planned Date: May 6, 2022
    • Expected Date: May 6, 2022
    • Status: 0%
11. Manuscript Publication:
    • Planned Date: July 1, 2022
    • Expected Date: July 1, 2022
    • Status: 0%

• Project Plan
  • Project plan presentation
  • Project plan proposal
• Project Background Reading
  • See Bibliography below for links.
• Project Checkpoint
  • Project Checkpoint Presentation
• Paper Seminar Presentations
  • Background Reading Presentation
  • Background Reading Critical Review
  • Paper #1
  • Paper #2
• Project Final Presentation
  • PDF of Poster
• Project Final Report
  • Final Report
  • links to any appendices or other material

Reading Material

1. Magro I, Pastel D, Hilton J, Miller M, Saunders J, Noonan K. Developmental Anatomy of the Eustachian Tube: Implications for Balloon Dilation. Otolaryngol Head Neck Surg. 2021;165(6):862-867. doi:10.1177/0194599821994817.
2. Froehlich MH, Le PT, Nguyen SA, McRackan TR, Rizk HG, Meyer TA. Eustachian Tube Balloon Dilation: A Systematic Review and Meta-analysis of Treatment Outcomes. Otolaryngol Head Neck Surg. 2020;163(5):870-882. doi:10.1177/0194599820924322.
3. Keschner D, Garg R, Loch R, Luk LJ. Repeat Eustachian Tube Balloon Dilation Outcomes in Adults With Chronic Eustachian Tube Dysfunction. Otolaryngol Head Neck Surg. August 2021. doi:10.1177/01945998211037975.
4. Isensee, F., Jaeger, P.F., Kohl, S.A.A. et al. nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nat Methods 18, 203–211 (2021). https://doi.org/10.1038/s41592-020-01008-z.
5. Balakrishnan G, Zhao A, Sabuncu MR, Guttag J, Dalca AV. VoxelMorph: A Learning Framework for Deformable Medical Image Registration. IEEE Trans Med Imaging. 2019 Feb 4. doi: 10.1109/TMI.2019.2897538. Epub ahead of print.
6. Fu Y, Brown NM, Saeed SU, et al., (2020). DeepReg: a deep learning toolkit for medical image registration. Journal of Open Source Software, 5(55), 2705. doi.org/10.21105/joss.02705.
7. Fu Y, Lei Y, Wang T, Curran WJ, Liu T, Yang X. Deep learning in medical image registration: a review. Phys Med Biol. 2020 Oct 22;65(20):20TR01. doi: 10.1088/1361-6560/ab843e.

References

1. Froehlich MH, Le PT, Nguyen SA, McRackan TR, Rizk HG, Meyer TA. Eustachian Tube Balloon Dilation: A Systematic Review and Meta-analysis of Treatment Outcomes. Otolaryngol Head Neck Surg. 2020;163(5):870-882. doi:10.1177/0194599820924322.
2. Magro I, Pastel D, Hilton J, Miller M, Saunders J, Noonan K. Developmental Anatomy of the Eustachian Tube: Implications for Balloon Dilation. Otolaryngol Head Neck Surg. 2021;165(6):862-867. doi:10.1177/0194599821994817.
3. Sinha A, Leonard S, Reiter A et al. Automated segmentation and statistical shape modeling of the paranasal sinuses to estimate natural variations. Proc SPIE Int Soc Opt Eng. 2016; 9784:97840D. doi: 10.1117/12.2217337.
4. Ding AS, Lu A, Li Z, et al. Automated Registration-Based Temporal Bone Computed Tomography Segmentation for Applications in Neurotologic Surgery. Otolaryngol Head Neck Surg. 2021; Online Ahead of Print. doi: 10.1177/01945998211044982.
5. Isensee F., Jaeger PF, Kohl SAA. et al. nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nat Methods. 2021;18, 203–211. doi.org/10.1038/s41592-020-01008-z.
6. Balakrishnan G, Zhao A, Sabuncu MR, Guttag J, Dalca AV. VoxelMorph: A Learning Framework for Deformable Medical Image Registration. IEEE Trans Med Imaging. 2019; Epub ahead of print. doi: 10.1109/TMI.2019.2897538.
7. Fu Y, Brown NM, Saeed SU, et al. DeepReg: a deep learning toolkit for medical image registration. Journal of Open Source Software. 2020; 5(55), 2705. doi.org/10.21105/joss.02705.
8. Avants, B. B., Tustison, N., & Song, G. (2009). Advanced normalization tools (ANTS). Insight j, 2(365), 1-35.
9. Lu, W., Chaoli, W., Zhanquan, S., & Sheng, C. (2020). An Improved Dice Loss for Pneumothorax Segmentation by Mining the Information of Negative Areas. IEEE Access PP(99):1-1.
10. Zou KH, Warfield SK, Bharatha A, et al. Statistical validation of image segmentation quality based on a spatial overlap index. Acad Radiol. 2004;11(2):178-189. doi:10.1016/s1076-6332(03) 00671-8.
11. Dubuisson M-P, Jain AK. A modified Hausdorff distance for object matching. In: Proceedings of 12th International Confer- ence on Pattern Recognition. IEEE; 1994:566-568. doi:10.1109/ ICPR.1994.576361.

Here give list of other project files (e.g., source code) associated with the project. If these are online give a link to an appropriate external repository or to uploaded media files under this name space (2022-01).

Source + Version Control (GitHub): https://github.com/mikami520/CIS2-EustachianTube
Eustachian tube Dataset: https://teams.microsoft.com/l/team/19%3aM09Sep-UUQloVvhUHp_OQk7sKNRhfUd6s2LqZgAPBIg1%40thread.tacv2/conversations?groupId=fef5d618-9991-4439-9a14-fedc6818965b&tenantId=9fa4f438-b1e6-473b-803f-86f8aedf0dec
nnUNet (Source Code): https://github.com/MIC-DKFZ/nnUNet
VoxelMorph (Source Code): https://github.com/voxelmorph/voxelmorph
DeepReg (Source Code): https://github.com/DeepRegNet/DeepReg
MONAI (Source Code): https://github.com/Project-MONAI/MONAI