Last updated: 05/05/2016 at 10:15PM

The goal of this project is to improve the calibration of overlaying Cone Based Computer Tomography (CBCT) and Red-Green-Blue-Depth (RGBD) camera meshes by creating an automated calibration algorithm. This will allow a faster process to generate a real-time 3D mixed-reality visualization and allow surgeons to more easily locate guide wire and screw placement with less time and usage of x-rays (i.e. less radiation exposure).

• Students: Dan Adler, Tiffany Chung
• Mentor(s): Dr. Nassir Navab, Bernhard Fuerst, Sing Chun Lee, Javad Fotouhi

Orthopedic surgeries often demand correct placement of a medical instrument and/or implant. Therefore, imaging techniques are utilized to confirm that tools are being inserted into correct bodily areas, and also any other necessary implants. Current solutions utilize continuous acquisition of X-ray images which result in high radiation exposure, numerous X-Rays captured, long surgical durations, and significant taskload (Fischer et. al).

• Minimum: (COMPLETED: 03/04/2016)
  1. Mentor meetings
  2. Understand manual calibration
  3. Develop automated calibration
  4. Automated Matching of CBCT-RGBD
• Expected: (COMPLETED: 04/15/2016)
  1. Automated CBCT Data Pre-processing
  2. Automated RGBD Data Pre-processing
  3. RGBD and CBCT Point Cloud Processing
• Maximum: (COMPLETED: 05/02/2016)
  1. Research improved algorithm
  2. Present Implementation
  3. Implement

This calibration process has 3 main phases. Many of these tedious steps are conducted manually, such as the segmentation (Lee et. al). The first phase is done in ImFusion, using KinFu to generate the point cloud from the Kinect camera and segment the meshes from the CBCT. The second phase is done in MeshLab, extracting the useful point clouds from each data set. The third phase is done in PCL, transforming the point clouds using Fast Point Feature Histograms and ICP.

To validate the solution, comparisons between the resulting registrations of Lee’s manual calibration and the automated calibration will be done at the end of each phase of development. The solution is expected to be significantly faster than manual calibration, and the user only needs to click a single button.

Room:

• Mock OR environment with computer that can access CBCT and RGBD Data (Met)
• 24-hour access to computers with data (Met)

Hardware and Tools:

• Phantoms that have been used in manual calibration (Met)
  • If a phantom breaks, it is cheap and there has been confirmation it is easy to replace
• RGBD and CBCT scanners (Unmet, will decide by 4/8/2016)
  • NOTE: This is only needed if the maximum deliverable is reached, because more scan data might need to be taken to work with a different registration algorithm. There is a 30 minute radiation safety training that would need to be taken, and then this equipment could be accessed.

Software:

• Open source libraries: KinFu, MeshLab, and PCL libraries (Met)
• ImFusion (Unneeded, decided on 3/7/16)
  • The software developers have a close relationship with Dr. Navab’s lab and thus personal licenses can be acquired. The third part of the implementation may be changed to use a different SDK. Currently deciding NOT to use ImFusion and using ITK as a replacement.
• Compatible Environments (Met on 2/24/2016)
• Local Linux environment with MeshLab, PCL, and all source code for each team member.
• ITK (Insight Segmentation and Registration Toolkit) (Met on 3/8/2016.)

Code Storage:

• LCSR Git access (Met)
• Private-repository of needed data and code on BitBucket (Met on 2/24/2016)

1. Implement Automated Matching between CBCT and RGBD
   • Completed Date: 3/4/16
2. Validate Matching Algorithm (FINISH MINIMUM DELIVERABLE)
   • Completed Date: 3/4/16
3. Milestone name: Implement Automated CBCT Pre-processing
   • Completed Date: 3/18/16
4. Milestone name: Implement Automated RGBD Pre-processing
   • Completed Date: 4/08/16
5. Milestone name: Validate Pre-processing
   • Completed Date: 4/08/16
6. Milestone name: Implement CBCT and RGBD Mesh Pre-Processing
   • Completed Date: 4/15/16
7. Milestone name: Validation of Calibration (FINISH EXPECTED DELIVERABLE)
   • Completed Date: 4/15/16
8. Milestone name: Track Weakest Link in Algorithm
   • Completed Date: 4/22/16
9. Milestone name: Present Improved Algorithm for Registration
   • Completed Date: 4/29/16
10. Implement Improved Algorithm (FINISH MAXIMUM DELIVERABLE)
    • Completed Date: 5/2/16

• Project Plan
  • Project plan presentation
  • Project plan
• Project Background Reading
  • See Bibliography below for links.
• Project Checkpoint
  • Project checkpoint presentation
• Paper Seminar Presentations
  • dadler_fpfh_presentation_final.pdf
  • dadler_fpfh_review_final.pdf
  • tchung12_paperreview.pdf
  • tchung12_seminarpresentation.pdf
• Project Final Presentation
  • group6_teaser_presentation.pdf
• Project Final Poster
  • final_poster.pdf
• Project Final Report
  • project6_finalwriteup.pdf

Besl PJ, McKay ND. A Method for Registration of 3-D Shapes. IEEE Transactions on Pattern Analysis and Machine Intelligence, February 1992.

Fischer M, Fuerst B, Lee SC, Fotouhi J, Habert S, Weidert S, Euler E, Osgood G, Navab N. Pre-Clinical Usability Study of Multiple Augmented Reality Concepts for K-Wire Placement. International Journal of Computer Assisted Radiology and Surgery / International Conference on Information Processing in Computer-Assisted Interventions (IPCAI), Heidelberg, June 2016.

Kojcev R, Fuerst B, Zettining O, Fotouhi J, Lee SC, Taylor R, Sinibaldi E, Navab N. Dual-Robot Ultrasound-Guided Needle Placement: Closing the Planning-Imaging-Action Loop. International Journal of Computer Assisted Radiology and Surgery / International Conference on Information Processing in Computer-Assisted Interventions (IPCAI), Heidelberg, June 2016

Lee SC, Fuerst B, Fotouhi J, Fischer M, Osgood G, Navab N. Calibration of RGBD Camera and Cone-Beam CT for 3D Intra-operative Mixed Reality Visualization. International Journal of Computer Assisted Radiology and Surgery / International Conference on Information Processing in Computer-Assisted Interventions (IPCAI), Heidelberg, June 2016.

Lorensen W, Cline H. Marching Cubes: A High Resolution 3D Surface Construction Algorithm. Computer Graphics 21, pp 63-169 (1987).

Rusu RB, Blodow N, Beetz M. 2009. Fast point feature histograms (FPFH) for 3D registration. In Proceedings of the 2009 IEEE international conference on Robotics and Automation (ICRA'09). IEEE Press, Piscataway, NJ, USA, 1848-1853.

Rusu RB, Cousins S. 3D is here: Point Cloud Library (PCL). IEEE International Conference on Robotics and Automation (ICRA), Shanghai, China, May 2011.

Rusu, RB, Marton ZC, Blodow N, Beetz M. Peristent Point Feature Histograms for 3D Point Clouds. Intelligent Autonomous Systems, Munich, Germany, December 2007.

Schroeder W, Martin KM, Lorensen WE. 1998. The Visualization Toolkit (2nd Ed.): An Object-Oriented Approach to 3D Graphics. Prentice-Hall, Inc., Upper Saddle River, NJ, USA.

Torr, P., Zisserman, A.: Mlesac: A new robust estimator with application to estimating image geometry. Computer Vision and Image Understanding 78(1), 138 – 156 (2000). DOI http://dx.doi.org/10.1006/cviu.1999.0832. URL http://www.sciencedirect.com/ science/article/pii/S1077314299908329

Wahl E, Hillenbrand U, Hirzinger G. Surflet-Pair-Relation Histograms: A Statistical 3D-Shape Representation for Rapid Classification. IEEE 3-D Digital Imaging and Modeling, Banff, Alta, October 2003.

Yoo TS, Ackerman MJ,Lorensen WE, Schroeder W, Chalana V, Aylward S, Metaxas D, Whitaker R. Engineering and Algorithm Design for an Image Processing API: A Technical Report on ITK - The Insight Toolkit. In Proc. of Medicine Meets Virtual Reality, J. Westwood, ed., IOS Press Amsterdam pp 586-592 (2002).

Dan Adler (daadler0309@gmail.com) Tiffany Chung (tiphchung@gmail.com) 600.446 CIS2: CamC Calibration Project

Computer Aided Medical Procedures (CAMP) Lab

Automated Calibratrion of CBCT to RGBD

Created: 4/9/2016 Last Updated: 4/20/2016

DEPENDENCIES

1. Point Cloud Library (PCL): http://pointclouds.org/
2. Insight Segmentation and Registration Toolkit (ITK): http://www.itk.org/
3. The Visualization Toolkit (VTK): http://www.vtk.org/
4. Qt: http://www.qt.io/
5. Boost: http://www.boost.org/
6. Eigen: http://eigen.tuxfamily.org/index.php?title=Main_Page

BUILDING

1. Go to MAIN directory.
2. Type “mkdir build”
3. cd into the build directory
4. Type “cmake ..”
5. Type “make”

ALGORITHM

CBCT POINT CLOUD PROCESSING:

Uses raw CBCT DICOM data and filters/thresholds the data to then create a 3D point cloud. It outputs this 3D point cloud as a single .obj file, with prefix specified by the user. The recommended LB/UB based upon our work is 3.0/10.0 respectively.

File: CBCT_phase1.cpp Usage: ./phase1.cpp CBCT_Folder Output.obj LB UB

Specific Inputs:

1. CBCT_Folder contains all DICOM images from the CBCT scanner
2. Output.obj is the name of the .obj file that will represent the CBCT point cloud
3. LB is the lower bound standard deviations from the mean that will be used to threshold the taken point cloud
4. UB is the upper bound standard deviations from the mean that will be used to threshold the taken point cloud

Output(s):

1. Ordered List ItemOutput.obj that is a point cloud of the CBCT data.

RGBD POINT CLOUD PROCESSING:

Uses raw RGB and depth data (in .png files) to build a 3D point cloud of the RGBD data.

File: main.cpp Usage: ./png2pcd fileNames.txt

Specific Inputs:

1. Ordered List ItemA .txt file where each line is a filename of a 2D RGB/Depth file

Output(s):

1. Ordered List Itemmesh.pcd is the point cloud mesh of RGBD data

CBCT/RGBD INITIAL ALIGNMENT:

Computes a registration of a CBCT point cloud to an RGBD point cloud using a feature alignment algorithm. It then saves a visual alignment of the two clouds in a .pcd file and the registration (rotation and translation) in a .txt file called “TemplateMatchingResult.txt”.

File: template_alignment.cpp Usage: ./PCLTemplateMatching PointCloud1.pcd PointCloud2.pcd Output.pcd

Specific Inputs:

1. PointCloud1.pcd is the CBCT point cloud
2. PointCloud2.pcd is the RGBD point cloud
3. Output.pcd is the name of the aligned CBCT/RGBD output visual

Output(s)

1. Ordered List ItemOutput.pcd a .pcd file that shows the CBCT/RGBD alignment
2. Ordered List ItemTemplateMatchingResult.txt a .txt file that holds the registration (rotation and translation) from the CBCT to RGBD. This file can be used as an intial registration in the matching section of the algorithm

CBCT/RGBD POINT CLOUD MATCHING:

Takes an initial registration of a CBCT to RGBD point cloud, and then computes a more precise alignment of the two clouds. Outputs this registration (rotation and translation) as a timestamped .txt file.

File: phase3_icp.cpp Usage: ./phase3_icp CBCT.pcd RGBD.pcd initial_reg.txt

Specific Inputs:

1. CBCT.pcd is a processed point cloud of the CBCT data in .pcd format
2. RGBD.pcd is a processed point cloud lf the RGBD data in .pcd format
3. initial_reg.txt is the initial registration found in the initial alignment step

Output(s):

1. Ordered List ItemICPAlignmentResult_Y-M-D.Time.txt .txt file with the output registration saved to it (translation

and rotation).

MAX DELIVERABLE DOCUMENTATION (BUNNY)

max_deliverable_documentation.pdf