Contact Us
CiiS Lab
Johns Hopkins University
112 Hackerman Hall
3400 N. Charles Street
Baltimore, MD 21218
Directions

Lab Director
Russell Taylor
127 Hackerman Hall
rht@jhu.edu

Trace:

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
courses:456:2023:projects:456-2023-12:project-12 [2023/05/11 03:47] – [Milestones and Status] jmangul1courses:456:2023:projects:456-2023-12:project-12 [2023/05/11 12:40] (current) – [Finalized Approach] jmangul1
Line 1: Line 1:
 ======Evaluation of CT Registration for Image-Based Sinus Reconstruction====== ======Evaluation of CT Registration for Image-Based Sinus Reconstruction======
-**Last updated: 04/13/2023 07:42AM**+**Last updated: 05/10/2023 06:42PM**
  
  
 ======Summary====== ======Summary======
-This project intends to evaluate the accuracy of an existing image-based 3D reconstruction pipeline of the sinus anatomy by implementing a framework for global and local registration to the ground-truth CT scan. The initial evaluation of the pipeline will then serve as a baseline for subsequent changes to account for uncertainties and integrate robot kinematics.+This project intends to evaluate the accuracy of an existing image-based 3D reconstruction pipeline of the sinus anatomy by implementing a framework for global and local registration to the ground-truth CT scan. The initial evaluation of the pipeline will then serve as a baseline for subsequent changes to investigate the influence of depth uncertainties and integrate robot kinematics.
  
   * **Students:** Jan Mangulabnan   * **Students:** Jan Mangulabnan
Line 25: Line 25:
 The main goal of this project is to implement a quantitative framework to evaluate the accuracy of the dense reconstruction based on the ground truth CT. The development of this assessment and framework would enable further research towards the usage of the sinus reconstruction pipeline in clinical settings. The specific aims of this project are listed as follows: The main goal of this project is to implement a quantitative framework to evaluate the accuracy of the dense reconstruction based on the ground truth CT. The development of this assessment and framework would enable further research towards the usage of the sinus reconstruction pipeline in clinical settings. The specific aims of this project are listed as follows:
   - Implement a rigid registration framework to evaluate the image-based 3D reconstruction of the sinus anatomy with respect to the corresponding CT image.   - Implement a rigid registration framework to evaluate the image-based 3D reconstruction of the sinus anatomy with respect to the corresponding CT image.
-    - Evaluate global registration considering entire reconstruction. +    - Integrate multiple registration methods to align meshes. 
-    - Evaluate local registration of specific anatomical regions of interest in the reconstruction. +    - Report error evaluation metrics between dense reconstruction and CT with visualizations.
-    - Implement methods to report evaluation metrics and visualizations to allow for both quantitative and qualitative assessment of the registration.+
   - Analyze the influence of uncertainty in the reconstruction pipeline, and evaluate the resulting reconstruction with respect to the CT.   - Analyze the influence of uncertainty in the reconstruction pipeline, and evaluate the resulting reconstruction with respect to the CT.
-    - Analyze the distribution of uncertainties in depth estimation for features present in multiple input image sequences. +    - Adjust depth fusion step in pipeline using inverse weighting of uncer- tainties, removal of outliers, and removal of larger depth estimates. 
-    - Integrate probabilistic model to adjust the influence of estimations based on analysis of uncertainty distribution. +    - Evaluation of resulting dense reconstructions. 
-  - Integrate robot kinematics in the registration process and evaluate results with respect to the CT. +  - Transactions in Medical Imaging paper extension for MICCAI 2020 Dense Reconstruction paper. 
 +    - Process additional cadaveric data in dense reconstruction pipeline and evaluate. 
 +    - Ablation experiments of dense reconstruction pipeline.
  
 ======Deliverables====== ======Deliverables======
Line 56: Line 56:
  
  
-======Technical Approach======+====== Technical Approach ======
 This project requires input data of endoscopic video sequences and CT scans of the same sinus anatomy. The data utilized for the maximum goal requires sequences obtained using the Galen robot to retrieve corresponding robot kinematics at the time of video capture. The endoscopic sequence for the dense reconstruction pipeline and the resulting 3D structure will be used for the registration with the corresponding CT scan to report an accuracy evaluation. This project requires input data of endoscopic video sequences and CT scans of the same sinus anatomy. The data utilized for the maximum goal requires sequences obtained using the Galen robot to retrieve corresponding robot kinematics at the time of video capture. The endoscopic sequence for the dense reconstruction pipeline and the resulting 3D structure will be used for the registration with the corresponding CT scan to report an accuracy evaluation.
  
Line 65: Line 65:
 {{ :courses:456:2023:projects:456-2023-12:workflow.png?nolink&600 |}} {{ :courses:456:2023:projects:456-2023-12:workflow.png?nolink&600 |}}
 //Figure 2. Proposed workflow of planned modifications and implementations shown in blue.// //Figure 2. Proposed workflow of planned modifications and implementations shown in blue.//
 +
 +Based on initial registration results and changes in the deliverables, the final registration workflow is shown in Figure 3.
 +
 +{{ :courses:456:2023:projects:456-2023-12:slide1.png?nolink&600 |}}
 +// Figure 3. Final registration workflow. //
 +
 +====Data Preprocessing====
 +Registration is dependent on input data of the dense reconstruction (DRECO) pipeline output (processed image sequence, estimated camera trajectories from SfM, and fused mesh of sinus anatomy), the ground-truth anatomical structure of the sinus from the corresponding CT scan and optical tracking data of the endoscope camera. The collected video sequence was preprocessed for use in the DRECO pipeline by curating the images to isolate the subsequence of frames that capture the sinus cavity. The input image frames were also downsampled and undistorted based on checkerboard camera calibration. The preprocessed images were then used as input to the DRECO pipeline. The CT scans were processed using 3D Slicer to segment both the attached marker spheres and sinus anatomy. Instructions for segmentation can be found on the GitHub repository [[https://git.lcsr.jhu.edu/sinusendoscopy/video_ct_registration/-/ blob/main/HowToSegment.md|here]].
 +
 +
  
 ===== Registration Framework ===== ===== Registration Framework =====
 +
 +==== Proposed Plan ====
  
 I plan to integrate rigid registration methods including the iterative closest point (ICP) algorithm and iterative most likely point algorithm to register the dense reconstruction to the corresponding CT image. The iterative most likely point algorithm and variations will be integrated using the cisstICP library available on [[https://github.com/AyushiSinha/cisstICP|Github.]] These methods will be used to perform both local and global registrations. At the global scale, the entire reconstruction will be used for the registration. In order to perform local registration, I plan to implement methods to isolate specific anatomical regions of interest in the reconstruction by isolating a subset of frames from the input video sequence to reconstruct only a portion of the sinus anatomy. The resulting reconstruction will then be used to apply local registration to the CT. I plan to integrate rigid registration methods including the iterative closest point (ICP) algorithm and iterative most likely point algorithm to register the dense reconstruction to the corresponding CT image. The iterative most likely point algorithm and variations will be integrated using the cisstICP library available on [[https://github.com/AyushiSinha/cisstICP|Github.]] These methods will be used to perform both local and global registrations. At the global scale, the entire reconstruction will be used for the registration. In order to perform local registration, I plan to implement methods to isolate specific anatomical regions of interest in the reconstruction by isolating a subset of frames from the input video sequence to reconstruct only a portion of the sinus anatomy. The resulting reconstruction will then be used to apply local registration to the CT.
  
 Additionally, I plan to develop methods to report evaluation metrics and visualizations for both quantitative and qualitative assessment of the registration. This will include a summary of the error magnitude between projected points of the reconstruction to the ground truth CT points and an overlay of the registered reconstruction and CT. I plan to include visual differences of the points in the overlay to allow for clearer comparison of the variation in error magnitude. This framework will allow me to produce a baseline evaluation of the accuracy of the 3D sinus reconstruction to use as a point of comparison for subsequent changes. Additionally, I plan to develop methods to report evaluation metrics and visualizations for both quantitative and qualitative assessment of the registration. This will include a summary of the error magnitude between projected points of the reconstruction to the ground truth CT points and an overlay of the registered reconstruction and CT. I plan to include visual differences of the points in the overlay to allow for clearer comparison of the variation in error magnitude. This framework will allow me to produce a baseline evaluation of the accuracy of the 3D sinus reconstruction to use as a point of comparison for subsequent changes.
 +
 +==== Finalized Approach ====
 +Direct rigid registration of sampled point clouds from the DRECO and CT meshes fails since the dense reconstruction only constructs a portion of the segmented anatomy present in the CT mesh. Therefore, this framework required advanced options including camera pose, keypoint, and coherent point drift registration to align the meshes.
 +=== Camera Pose Registration ===
 +The segmented anatomy marker spheres are required to obtain the ground-truth positions of the endoscope camera in CT space. Based on the segmentation, the centers of these spheres are extracted using a sphere-fitting algorithm to register the CT to the tracked marker geometry of the anatomy recorded by the NDI Polaris tracker. A checkerboard hand-eye calibration was also done to determine the transformation between the endoscope camera and endoscope marker sphere geometry. These transformations along with the tracked endoscope may then be used to compute the ground-truth position of the camera in CT space.
 +
 +The tracked positions were also manually adjusted by comparing the recorded images and CT renderings when large errors were observed (camera position was outside of the anatomy). This alignment transformation was applied at the beginning of the chain in Equation 1.
 +The DRECO pipeline estimates the camera trajectories of each input image frame in the SfM algorithm which can be matched to the corresponding ground- truth position of the endoscope for rigid registration. The resulting transforma- tion was then used to transform the DRECO mesh to CT space and initialize the iterative closest point algorithm for Video-CT registration.
 +The estimated camera trajectories were observed to have large variations com- pared to the ground-truth, reducing the accuracy of the overall registration. Due to these errors, local reconstructions of the sinus anatomy were also evaluated based on a section of the original input sequence as shown in Figure 4.
 +{{ :courses:456:2023:projects:456-2023-12:poses_all00.png?600 |}}
 +//Figure 4. Camera pose registration.//
 +
 +=== Coherent Point Drift Registration ===
 +In addition to rigid registration methods, the Coherent Point Drift (CPD) reg- istration algorithm was also investigated. CPD is a probabilistic method integrated for rigid and affine point cloud registration between sampled points from the dense reconstruction and CT meshes. This method optimizes regis- tration based on the most likely shape of the DRECO mesh within the CT structure, considering that the computed mesh only represents the section of the sinus anatomy visible in the input video.
 +
 +=== Comparison of Registration Types ===
 +
 +Table 1 displays the mean errors of the various registration algorithms. Rigid camera pose registration using ICP had the lowest translational error, ranging from 1 to 3mm differences, whereas both Coherent Point Drift algorithms had translational errors on the magnitude of centimeters for the transformed camera poses. All three registration results have significant rotational error, ranging from 12 to 21 degrees. These errors are the same for each type as only the camera position was transformed.
 +
 +{{ :courses:456:2023:projects:456-2023-12:table1.png?600 |}}
 +//Table 1. Comparison of various registration types using the entire image se- quence (indexes 0 - 1059) and multiple sections for local registration reported as the mean across poses, pixels, and sampled points for camera pose, scale invariant depth, and mesh distance errors, respectively.//
 +
 +The CPD rigid and affine registration algorithms have lower errors in the mean distance between meshes but based on visual inspection of the layered meshes and depth renderings, this does not seem to mean that the anatomy is more aligned. The smaller magnitudes may be a result of the intricate sinus anatomy as the closest points between the meshes do not necessarily correspond to the same points within the sinus cavity.
 +
 +It is expected that the camera pose + ICP registrations have lower pose errors and the CPD registrations have lower mesh errors because these algorithms com- pute the transformation by minimizing those parameters. The scale invariant depth error serves as a metric independent of the registration. Since the camera pose + ICP registration type had the smallest error in the depth renderings, this algorithm was used to further investigate adjustments to the depth fusion step in the dense reconstruction pipeline.
  
 ===== Influence of Uncertainties in Dense Reconstruction Pipeline ===== ===== Influence of Uncertainties in Dense Reconstruction Pipeline =====
  
-The dense reconstruction pipeline utilizes depth estimations in addition the SfM point cloud and camera trajectories to generate the 3D structure. This information is integrated into a fusion method which resolves variation between the estimates of common points in multiple frames of the input sequence. The fusion method currently considers every estimate equally; however, the points in the sinus anatomy that are further away from the camera when the image is captured is shown to have more uncertainty as seen in Figure 3.+The dense reconstruction pipeline utilizes depth estimations in addition the SfM point cloud and camera trajectories to generate the 3D structure. This information is integrated into a fusion method which resolves variation between the estimates of common points in multiple frames of the input sequence. The fusion method currently considers every estimate equally; however, the points in the sinus anatomy that are further away from the camera when the image is captured is shown to have more uncertainty as seen in Figure 5.
  
 {{ :courses:456:2023:projects:456-2023-12:depth_estimation.png?nolink&400 |}} {{ :courses:456:2023:projects:456-2023-12:depth_estimation.png?nolink&400 |}}
-//Figure 3. Heat map of mean and standard deviation of depth estimates with corresponding input image. The deeper mean depth estimation, meaning further away from the camera at capture, exhibits higher uncertainty. Adapted from X. Liu et al., "Dense depth estimation in monocular endoscopy with self-supervised learning methods," IEEE transactions on medical imaging, 2019.//+//Figure 5. Heat map of mean and standard deviation of depth estimates with corresponding input image. The deeper mean depth estimation, meaning further away from the camera at capture, exhibits higher uncertainty. Adapted from X. Liu et al., "Dense depth estimation in monocular endoscopy with self-supervised learning methods," IEEE transactions on medical imaging, 2019.//
  
-We hypothesize that this uncertainty may be introducing errors which are propagated into the reconstruction. I plan to analyze these uncertainties by examining the depth estimations for features that contribute to the same point in the point cloud to evaluate the distribution of the uncertainty. Based on this analysis, I will then integrate a probabilistic model in the pipeline to account for these uncertainties and limit the influence of inaccurate depth estimations to generate new reconstructions. The effectiveness of this adjustment will then be evaluated using the registration framework.+We hypothesize that this uncertainty may be introducing errors which are propagated into the reconstruction. To further investigate, I generated reconstructions using inverse weighting based on uncertainty to reduce the influence on the final depth estimates. I also removed outliers  
 +outside of the 68th percentile (one standard deviation) of uncertainty and mean depth estimations. These alternate reconstructions are shown in Figure 6 and were also evaluated using the implemented registration framework. 
 +{{ :courses:456:2023:projects:456-2023-12:fusion.png?nolink&600 |}} 
 +//Figure 6. The resulting dense reconstructions with different adjustment schemes (from left to right): original, weighting by uncertainty, large depths removed, and outliers removed. 
 +//
  
 ===== Integration of Robot Kinematics ===== ===== Integration of Robot Kinematics =====
Line 150: Line 191:
     * {{:courses:456:2023:projects:456-2023-12:checkpoint_presentation.pdf| Project checkpoint presentation}}     * {{:courses:456:2023:projects:456-2023-12:checkpoint_presentation.pdf| Project checkpoint presentation}}
   * Project Final Presentation   * Project Final Presentation
-    * {{:courses:456:2023:projects:456-2023-12:final_poster_pdf.pdf|PDF of Poster}}+    * {{:courses:456:2023:projects:456-2023-12:final_poster.pdf|PDF of Poster}}
   * Project Final Report   * Project Final Report
     * {{:courses:456:2023:projects:456-2023-12:final_report.pdf|Final Report}}     * {{:courses:456:2023:projects:456-2023-12:final_report.pdf|Final Report}}
-    * links to any appendices or other material+  * Additional Links 
 +    * [[https://git.lcsr.jhu.edu/sinusendoscopy/video_ct_registration|GitHub Repository]] 
 +    * [[https://git.lcsr.jhu.edu/sinusendoscopy/video_ct_registration/-/blob/main/README.md|Codebase Usage]] 
 +    * [[https://git.lcsr.jhu.edu/sinusendoscopy/video_ct_registration/-/blob/main/DataSpec.md#data-specification|Data Specification]] 
 +    * [[https://git.lcsr.jhu.edu/sinusendoscopy/video_ct_registration/-/blob/main/HowToSegment.md|Segmentation Instructions]] 
 + 
  
 ======Project Bibliography======= ======Project Bibliography=======
Line 175: Line 222:
  
 **[10]** B. Curless and M. Levoy, "A volumetric method for building complex models from range images," in Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, 1996, pp. 303-312.  **[10]** B. Curless and M. Levoy, "A volumetric method for building complex models from range images," in Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, 1996, pp. 303-312. 
 +
 ======Other Resources and Project Files====== ======Other Resources and Project Files======
 Here give list of other project files (e.g., source code) associated with the project.  If these are online give a link to an appropriate external repository or to uploaded media files under this name space (2023-12). Here give list of other project files (e.g., source code) associated with the project.  If these are online give a link to an appropriate external repository or to uploaded media files under this name space (2023-12).
  
courses/456/2023/projects/456-2023-12/project-12.1683776830.txt.gz · Last modified: by jmangul1




ERC CISST    LCSR    WSE    JHU