Table of Contents

Automatic Segmentation and 3D Reconstruction of the Inner Ear Vasculature from Histology Slides

Last updated: 5/10

Summary

Researchers have often relied on analysis of histopathology slides of the inner ear to understand the impact that different structures have on the various pathologies in otolaryngology.Currently, vascular disorders of the inner ear remain poorly understood. Some researchers have hypothesized that abnormal vasculature may play a role in pathologies such as vestibular neuritis.[3] The aim of this project is to create software for automatic segmentation and 3D reconstruction of vasculature within the inner ear to facilitate more efficient and spatial analysis of structures within inner ear

Background, Specific Aims, and Significance

Background

Researchers have often relied on analysis of histopathology slides of the inner ear from deceased humans/mammals to understand the impact that different structures have on the various pathologies in otolaryngology. Understanding the pathophysiology of the inner ear remains crucial to guiding treatment strategies in otolaryngology. For instance, the stria vascularis, a structure in the inner ear analyzed on histology slides, has been thought to be critical to understanding sensorineural hearing loss in mammals [1]. Additionally, through studying histology slides of patients with inner ear malformation, researchers were able to create a better model for cochlear implants in this population [2].

One area of interest within the realm of inner ear histopathology is the study of vascular in the inner ear diseases. Currently, vascular disorders of the inner ear remain poorly understood. Some researchers have hypothesized that abnormal vasculature may play a role in pathologies such as vestibular neuritis [3]. These researchers note that being able to analyze the vasculature with 3D models of histopathology could prove to be valuable as it would enable them to better understand the positional relationship between vasculature and adjacent structures.

However, despite the importance of analyzing histology slides, most researchers still rely on manual methods for segmentation of structures within the inner ear. Manual segmentation techniques can also lead to poor interrater reliability [4]. Furthermore, few researchers have the tools to be able to generate 3D models of structures within the inner ear. While automatic techniques to extract relevant features from whole slide histology images (WSI) is rapidly expanding, little work has been done to automate segmentation histology slides of the inner ear and construction 3D models from these segmentations.

Goals

The key deliverable at the end of this project is to create software that can

  1. Use deep learning methods to segment histopathology slides of the ear
  2. Use these segmentations to create a 3D reconstruction of the ear

These goals can further be organized into

Significance

Completion of this project will enable researchers to analyze WSI of the inner ear rapidly and accurately compared to existing manual techniques. Additionally, by adding the ability to view these segmentations in 3D, researchers will have new insights into the role that structures such as the vasculature can play in various pathologies. Creation of this software could also facilitate the ability to study other structures within inner ear WSI. The proposed project would be first example of reconstructing vasculature from inner ear histology slides using deep learning. Overall, the proposed project has the potential to dramatically improve the way researchers analyze WSI within the inner ear.

Deliverables

Technical Approach

Broadly, the technical approach can be divided into two phases. 1) Creation of the deep learning framework for semantic segmentation of inner ear vasculature. 2) Alignment of segmentations and generation of 3D model

Overall Approach

Deep Learning Pipeline

Data Base: Temporal bone slices of 4 macaque monkey ears have already been sectioned for this project. Each ear contains 110 slides spaced 10 micrometers apart. The ears themselves are assumed to have no pathologies associated with them. These slides were converted into digital images using an Olympus microscope setup in the Lauer lab at JHMI. As shown above, Two structures were specifically labeled for training: the vasculature in the mid-modiolus of the inner ear and the scala media, scala vestibuli, and scala tympani (these were combined and referred to as scalas). Of the 110 slides in the database, 30 were randomly sampled for labeling.

Model Training: The training process for both structures was done within a Roboflow© environment. For the scalas, each slide and corresponding label was first compressed to 512×512 pixel image and then augmented by rotation from -15 to 15 degrees as well as magnified from 0-20% percent. Overall, 30 scalas were represented in 267 images. Data augmentation remains a valuable tool to increase deep learning generalizability when working with relatively small datasets. Instead of compressing the original image by ~20x, training for the vascular segmentation model involved tiling a ~4x compressed slide (2048×2048 pixel) into 512×512 patches before data augmentation. The vasculature in the inner ear is a relatively small structure and is often lost if represented in an overly compressed image. Overall, 1312 512×512 patches were used for vascular segmentation model.

The nnUnet architecture has already been described in detail in other works.[7] A generic overview of the architecture is shown above. nnUnet has additional pre-processing steps including normalizing the intensity range of the image. The nnUnet was trained on 5 folds of data with a training, validation, testing split of 60-20-20. Each fold was trained for a maximum of 20-30 epochs. After training, the optimal weights of nnUnet based on the 5 folds were chosen and used for inference.

Image Registration and 3D Reconstruction

Once the deep learning pipeline produced binary masks, the image registration module aligned these masks. This done in Python using the SimpleElastix toolbox. The overall process of this module is outlined above. Broadly, a compressed (512 x 512px) WSI and its subsequent WSI were aligned by finding a transform T. This transform T was then applied to both the vascular mask as well as the scalas mask to align them to the previous slide. The transform T is represented by two separate processes. First, an affine transform T1 was computed between the two images (I1 and I2) to generate an initial aligned image I1a. Then, a rigid transform T2 is computed between I1a and I2. The final transform T can be represented as T1 x T2. The method for calculating each type of transform is a traditional iterative closest point algorithm implemented by SimpleElastix.

Once the aligned images were produced, they were uploaded in 3D Slicer to produce a volume rendering. This was done through the SlicerMorph extension which enables users to upload a series of images as a volume stack.

Dependencies

This project requires digital WSI that will be provided by the Lauer Lab. Outside of this dependency most requirement of this project will be based on open source platforms. Other dependencies are outlined below

Milestones and Status

  1. Milestone name: 40 labeled Whole Slide Images(WSI); 81 scanned slides total
    • Planned Date: 3/6/23
    • Expected Date: 4/4/13
    • Status: Completed
  2. Milestone name: Functional software that segments vasculature from WSI
    • Planned Date: 3/16
    • Expected Date: 4/28
    • Status: In progress
  3. Milestone name: Internal validation report that analyzes non-labeled slides
    • Planned Date: 3/23
    • Expected Date: 4/28
    • Status: Completed
  4. Milestone name: Software that can align WSI slides
    • Planned Date: 4/4
    • Expected Date: 4/4
    • Status: Completed
  5. Milestone name: 3D mesh file of Monkey Inner Ear
    • Planned Date: 4/31
    • Expected Date: 5/5
    • Status: Completed
  6. Milestone name: Submitted manuscript
    • Planned Date: 5/11
    • Expected Date: 5/29
    • Status: In progress
  7. Milestone name: 3D mesh file of Human Ear
    • Planned Date: 5/15
    • Expected Date: TBD
    • Status: In progress

Reports and presentations

Project Bibliography

1.Yu W, Zong S, Du P, Zhou P, Li H, Wang E, Xiao H. Role of the Stria Vascularis in the Pathogenesis of Sensorineural Hearing Loss: A Narrative Review. Front Neurosci. 2021 Nov 19;15:774585. doi: 10.3389/fnins.2021.774585. PMID: 34867173; PMCID: PMC8640081.

2.Monsanto RDC, Sennaroglu L, Uchiyama M, Sancak IG, Paparella MM, Cureoglu S. Histopathology of Inner Ear Malformations: Potential Pitfalls for Cochlear Implantation. Otol Neurotol. 2019 Sep;40(8):e839-e846. doi: 10.1097/MAO.0000000000002356. PMID: 31361687; PMCID: PMC7377297.

3.Büki B, Mair A, Pogson JM, Andresen NS, Ward BK. Three-Dimensional High-Resolution Temporal Bone Histopathology Identifies Areas of Vascular Vulnerability in the Inner Ear. Audiol Neurootol. 2022;27(3):249-259. doi: 10.1159/000521397. Epub 2021 Dec 29. PMID: 34965531; PMCID: PMC9133178.

4.Khened, M., Kori, A., Rajkumar, H. et al. A generalized deep learning framework for whole-slide image segmentation and analysis. Sci Rep 11, 11579 (2021). https://doi.org/10.1038/s41598-021-90444-8

5.Guo, Z., Liu, H., Ni, H. et al. A Fast and Refined Cancer Regions Segmentation Framework in Whole-slide Breast Pathological Images. Sci Rep 9, 882 (2019). https://doi.org/10.1038/s41598-018-37492-9

6.Jansen, I., Lucas, M., Savci-Heijink, C.D. et al. Three-dimensional histopathological reconstruction of bladder tumours. Diagn Pathol 14, 25 (2019). https://doi.org/10.1186/s13000-019-0803-7

7.Isensee, F., Jaeger, P.F., Kohl, S.A.A. et al. nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nat Methods 18, 203–211 (2021). https://doi.org/10.1038/s41592-020-01008-z

Other Resources and Project Files

Here give list of other project files (e.g., source code) associated with the project. If these are online give a link to an appropriate external repository or to uploaded media files under this name space (2023-06).

https://github.com/2014ajain/Vasculature_Seg_Recon.git