Last updated: 04/13/2023 07:42AM

This project intends to evaluate the accuracy of an existing image-based 3D reconstruction pipeline of the sinus anatomy by implementing a framework for global and local registration to the ground-truth CT scan. The initial evaluation of the pipeline will then serve as a baseline for subsequent changes to account for uncertainties and integrate robot kinematics.

• Students: Jan Mangulabnan
• Mentor(s): Roger Soberanis, Mathias Unberath

Sample Dense Reconstruction and CT Scan

Nasal obstructions are a common clinical problem that can significantly impact patients' quality of life. Septoplasty and turbinate reduction are two of the most common surgical interventions used to address this problem. Clinical studies have provided evidence towards positive impact of these interventions in patients’ quality of life. However, these assessments rely on subjective measures such as patient-reported symptoms and there is limited quantitative evidence to support the effectiveness of these surgeries. This creates a need for objective evaluation methods to inform clinicians about which patients are most likely to benefit from surgical intervention and provide tools for longitudinal assessment of the surgical outcomes.

Quantitative evaluation of the patient sinus anatomy requires geometric information of the nasal cavity. Computed tomography (CT) scans provide this information, however, these scans are expensive and expose patients to harmful radiation. This motivates an image-centered approach that leverages routine endoscopy procedures that physicians use to examine patients. An accurate 3D model of the sinus anatomy is required to retrieve clinically relevant parameters, such as aperture and volume, which can be used for quantitative assessment of patient anatomy.

Liu et. al developed a pipeline to generate a dense 3D reconstruction of the sinus anatomy from endoscopic video as shown in Figure 1.

Figure 1. Dense reconstruction pipeline adapted from X. Liu et al., “Reconstructing sinus anatomy from endoscopic video–towards a radiation-free approach for quantitative longitudinal assessment,” MICCAI 2020.

The pipeline utilizes a Structure from Motion (SfM) algorithm which matches common visible points in input images to produce a 3D structure of the object in the image. The visible points are matched using dense feature descriptors which are extracted using a deep learning model. These dense descriptors are then integrated into SfM to produce a dense point cloud of the anatomy and the camera trajectory for each frame in the input sequence. The sequence is also used to estimate the depth of the images from the camera using a deep learning model. The depth estimation, point cloud, and camera trajectories are then used in a depth fusion method to produce a 3D reconstruction of the sinus cavity. Currently, there is no assessment framework towards the correctness of the image-based 3D reconstruction.

The main goal of this project is to implement a quantitative framework to evaluate the accuracy of the dense reconstruction based on the ground truth CT. The development of this assessment and framework would enable further research towards the usage of the sinus reconstruction pipeline in clinical settings. The specific aims of this project are listed as follows:

1. Implement a rigid registration framework to evaluate the image-based 3D reconstruction of the sinus anatomy with respect to the corresponding CT image.
   1. Evaluate global registration considering entire reconstruction.
   2. Evaluate local registration of specific anatomical regions of interest in the reconstruction.
   3. Implement methods to report evaluation metrics and visualizations to allow for both quantitative and qualitative assessment of the registration.
2. Analyze the influence of uncertainty in the reconstruction pipeline, and evaluate the resulting reconstruction with respect to the CT.
   1. Analyze the distribution of uncertainties in depth estimation for features present in multiple input image sequences.
   2. Integrate probabilistic model to adjust the influence of estimations based on analysis of uncertainty distribution.
3. Integrate robot kinematics in the registration process and evaluate results with respect to the CT.

• Minimum: (Expected by 4/16)
  1. Pre-processed Data (3D Reconstructions and Segmented CT scans)
  2. Code for registration and transformation of camera poses & meshes
  3. Report of error evaluation of rendered depth maps

• Expected: (Expected by 4/30)
  1. Code for adjusting for depth estimation uncertainties
  2. 3D Reconstructions based off of implementation
  3. Report of error evaluation for global and local registrations

• Maximum: (Expected by 5/11)
  1. Processed data for remaining patients
  2. Transactions in Medical Imaging Paper Extension (for MICCAI 2020)

• Extended Maximum: (Summer work)
  1. Code for integration of robot kinematics data into reconstruction pipeline
  2. 3D Reconstructions based off of implementation
  3. Report of error evaluation for global and local registrations

This project requires input data of endoscopic video sequences and CT scans of the same sinus anatomy. The data utilized for the maximum goal requires sequences obtained using the Galen robot to retrieve corresponding robot kinematics at the time of video capture. The endoscopic sequence for the dense reconstruction pipeline and the resulting 3D structure will be used for the registration with the corresponding CT scan to report an accuracy evaluation.

The input data must be pre-processed for use in this project. The endoscopic video and robot kinematics are collected using a Robot Operating System based platform and need to be extracted to use as input in the dense reconstruction pipeline. The image frames of the video sequence will be used in the pipeline to generate the 3D reconstruction. The CT scans will also need to be segmented using 3D slicer. The 3D reconstruction and segmented CT will then be used towards the rigid registration.

The general approach is to first implement the registration framework and evaluation methods, analyze uncertainties in the dense reconstruction, and finally integrate robot kinematics in the registration. The proposed workflow of the project is shown in Figure 2.

Figure 2. Proposed workflow of planned modifications and implementations shown in blue.

I plan to integrate rigid registration methods including the iterative closest point (ICP) algorithm and iterative most likely point algorithm to register the dense reconstruction to the corresponding CT image. The iterative most likely point algorithm and variations will be integrated using the cisstICP library available on Github. These methods will be used to perform both local and global registrations. At the global scale, the entire reconstruction will be used for the registration. In order to perform local registration, I plan to implement methods to isolate specific anatomical regions of interest in the reconstruction by isolating a subset of frames from the input video sequence to reconstruct only a portion of the sinus anatomy. The resulting reconstruction will then be used to apply local registration to the CT.

Additionally, I plan to develop methods to report evaluation metrics and visualizations for both quantitative and qualitative assessment of the registration. This will include a summary of the error magnitude between projected points of the reconstruction to the ground truth CT points and an overlay of the registered reconstruction and CT. I plan to include visual differences of the points in the overlay to allow for clearer comparison of the variation in error magnitude. This framework will allow me to produce a baseline evaluation of the accuracy of the 3D sinus reconstruction to use as a point of comparison for subsequent changes.

The dense reconstruction pipeline utilizes depth estimations in addition the SfM point cloud and camera trajectories to generate the 3D structure. This information is integrated into a fusion method which resolves variation between the estimates of common points in multiple frames of the input sequence. The fusion method currently considers every estimate equally; however, the points in the sinus anatomy that are further away from the camera when the image is captured is shown to have more uncertainty as seen in Figure 3.

Figure 3. Heat map of mean and standard deviation of depth estimates with corresponding input image. The deeper mean depth estimation, meaning further away from the camera at capture, exhibits higher uncertainty. Adapted from X. Liu et al., “Dense depth estimation in monocular endoscopy with self-supervised learning methods,” IEEE transactions on medical imaging, 2019.

We hypothesize that this uncertainty may be introducing errors which are propagated into the reconstruction. I plan to analyze these uncertainties by examining the depth estimations for features that contribute to the same point in the point cloud to evaluate the distribution of the uncertainty. Based on this analysis, I will then integrate a probabilistic model in the pipeline to account for these uncertainties and limit the influence of inaccurate depth estimations to generate new reconstructions. The effectiveness of this adjustment will then be evaluated using the registration framework.

Robot kinematic data has been collected which corresponds to the endoscopic video sequences and CT scans. This data provides information related to the location of the endoscope camera. The SfM in the current reconstruction pipeline produces camera trajectories which is used as input to the depth fusion and surface extraction to produce the 3D structure. I plan to further analyze this step in the pipeline to develop a stronger understanding of the current implementation using the camera trajectory input. I will then work towards integrating the additional robot kinematic data to fine-tune the SfM trajectories. This integration will produce another reconstruction which will also be evaluated using the rigid registration methods.

1. Milestone name: Environment setup
   • Planned Date: 2/19
   • Expected Date: 2/19
   • Status: Complete
2. Milestone name: Generate 3D reconstructions
   • Planned Date: 2/24
   • Expected Date: 2/24
   • Status: Complete
3. Milestone name: Segment CT scans
   • Planned Date: 2/26
   • Expected Date: 2/26
   • Status: Complete
4. Milestone name: Integrated registration methods
   • Planned Date: 3/5
   • Expected Date: 3/30
   • Status: Complete
5. Milestone name: Code to isolate local regions and register
   • Planned Date: 3/19
   • Expected Date: 3/19
   • Status: Cancelled
6. Milestone name: Baseline error evaluation
   • Planned Date: 4/1
   • Expected Date: 4/17
   • Status: Complete
7. Milestone name: Depth uncertainty weighting for fusion step
   • Planned Date: 4/7
   • Expected Date: 4/30
   • Status: Complete
8. Milestone name: Error evaluation with uncertainty weighting
   • Planned Date: 4/14
   • Expected Date: 4/30
   • Status: Complete
9. Milestone name: Process additional patient data
   • Planned Date: 5/5
   • Expected Date: 5/5
   • Status: In Progress
10. Milestone name: Transactions in Medical Imaging Paper Extension (for MICCAI 2020)
    • Planned Date: 5/11
    • Expected Date: 5/11
    • Status: In Progress
11. Milestone name: Integrate Robot Kinematic Data and Evaluate
    • Planned Date: 4/28
    • Expected Date: 4/28
    • Status: Cancelled (Future Work)

• Project Plan
  • Project plan presentation
  • Project plan proposal
• Project Background Reading
  • Background reading presentation
  • Background reading report
  • See Bibliography below for links.
• Project Checkpoint
  • Project checkpoint presentation
• Project Final Presentation
  • PDF of Poster
• Project Final Report
  • Final Report
  • links to any appendices or other material

[1] M. Lavinsky‐Wolff et al., “Effect of turbinate surgery in rhinoseptoplasty on quality‐of‐life and acoustic rhinometry outcomes: a randomized clinical trial,” The Laryngoscope, vol. 123, no. 1, pp. 82-89, 2013.

[2] M. L. Hytönen, M. Lilja, A. A. Mäkitie, H. Sintonen, and R. P. Roine, “Does septoplasty enhance the quality of life in patients?,” European archives of oto-rhino-laryngology, vol. 269, pp. 2497-2503, 2012.

[3] D. Roblin and R. Eccles, “What, if any, is the value of septal surgery?,” Clinical Otolaryngology & Allied Sciences, vol. 27, no. 2, pp. 77-80, 2002.

[4] X. Liu et al., “Reconstructing sinus anatomy from endoscopic video–towards a radiation-free approach for quantitative longitudinal assessment,” in Medical Image Computing and Computer Assisted Intervention–MICCAI 2020: 23rd International Conference, Lima, Peru, October 4–8, 2020, Proceedings, Part III 23, 2020: Springer, pp. 3-13.

[5] J. L. Schonberger and J.-M. Frahm, “Structure-from-motion revisited,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 4104-4113.

[6] X. Liu et al., “Dense depth estimation in monocular endoscopy with self-supervised learning methods,” IEEE transactions on medical imaging, vol. 39, no. 5, pp. 1438-1447, 2019.

[7] R. Kikinis, S. D. Pieper, and K. G. Vosburgh, “3D Slicer: a platform for subject-specific image analysis, visualization, and clinical support,” Intraoperative imaging and image-guided therapy, pp. 277-289, 2014.

[8] S. D. Billings, E. M. Boctor, and R. H. Taylor, “Iterative most-likely point registration (IMLP): A robust algorithm for computing optimal shape alignment,” PloS one, vol. 10, no. 3, p. e0117688, 2015.

[9] A. Sinha, “cisstICP Library,” ed. https://github.com/AyushiSinha/cisstICP: GitHub, 2019.

[10] B. Curless and M. Levoy, “A volumetric method for building complex models from range images,” in Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, 1996, pp. 303-312.

Here give list of other project files (e.g., source code) associated with the project. If these are online give a link to an appropriate external repository or to uploaded media files under this name space (2023-12).