Last updated: 3/24/2015 at 12:58AM

For this project, we will be developing computer vision to track tool tip locations during sinus surgeries. We will detect contours and register them to our CT image data to provide real-time tracking.

• Students: John Lee, Kyle Xiong, Calvin Zhao
• Mentor(s): Austin Reiter, Balazs P. Vagvolgyi, Seth Billings

Currently, magnetic trackers are common in robotic microsurgery where maintaining sight is difficult. While these trackers are relatively inexpensive, interference with metal instruments in the operating room limits their accuracy and effectiveness. We aim to enhance the tracker registration, and hence accuracy, by combining high resolution patient CT data with real time video from the endoscopic surgical camera. Through a these inexpensive software-based algorithms, we hope to greatly enhance the tracking accuracy. We hope to demonstrate these improvements using a real-time augmented-reality overlay that will provide the surgeon with valuable anatomical data. This project will provide great accuracy improvements to magnetic trackers, enabling new uses for this existing technology at a low cost.

We will be contributing to the registration algorithm currently being developed by Seth Billings. Specifically, we will be working on extracting occluding contours from video. Occluding contours are edges of physical features that obscure the background. These contours delineate distinct “layers” in a scene. They are distinct from texture contours, which merely outline small surface deviations and discolorations. By examining the occluding contours from the endoscope and mapping them to the CT model, an accurate registration can be achieved. Once our occluding contour algorithm interfaces properly with the registration algorithm, we hope to implement an augmented reality overlay.

Specific Aims

1. An algorithm that accurately and efficiently extracts occluding contours from sinus surgery videos
2. Proper integration with existing registration software
3. Real-time augmented reality overlay on endoscope video feed

• Minimum: (Expected 3/27/15)
  1. Algorithm for accurately and efficiently extracting occluding contours from sinus surgery videos
  2. Implementation of algorithm in Matlab

• Expected: (Expected 4/10/15)
  1. Implementation of algorithm in C
  2. Registration software between contour detecting algorithm and CT images

• Maximum: (Expected 5/1/15)
  1. Augmented reality software from real-time contour detection and video-CT registration to provide surgeon with useful information

Our CIS II project on image processing can be divided into three steps:

1. Contour Detection: The goal of this project is to be able to rigorously register and track surgical tool position in CT coordinates by extracting occluding contours from video data.
   • Develop a new algorithm, based on existing optical flow algorithms, specifically to efficiently and accurately extract occluding contours. Algorithm will be developed in MATLAB and then in C/C++ for efficiency.
   • Using Horn-Schunk in combination with Canny edge detection, we expect to find accurate contours of tissue in sinus surgery and filter out extraneous details to achieve well-defined and clear edges in real-time video feed.
   • Another approach is to use train an SVM using a training set of images that will look for a number of image-related features (TBD). Using a binary classifier, we can determine which edges are part of the hand-drawn ground truth edges and which ones are unnecessary edges. Once the SVM is trained, we can use it on a number of different images to extract the occluding contours.
   • Currently, the SVM method above has shown the most promising results. The product is a package of MATLAB code that reads in input images from a directory, trains the SVM to classify true positives and false positives, and uses the classification to predict edge pixels in any sinus surgery image.
     1. The program uses hand-labeled ground truths during training. The labeled images are found in the 'labeled' directory of the package.
     2. The inputs of the package are found in the 'input' directory. The program automatically reads these images and predicts where their occluding contours are.
     3. The ouputs for each image are:
        1. A binary image of occluding contours (variable name: testFeatures)
        2. Normal vectors for each pixel in the above binary image (variable name: normals)
        3. X and Y positions in the binary image of each normal vector (variable names: x, y)
2. Integration: Contour detection will integrate with existing registration algorithm developed by Seth Billings. (Expected Deliverable)
   • Once integration is complete and we develop an efficient algorithm, we will be able to use real-time contour detection to track the position of the tool tip in CT coordinates.
3. Augmented Reality: AR overlay will be developed to provide real time information for the surgeon
   • Use CT data with registration to overlay an augmented reality interface over the video in real-time.

* Sinus Surgery Videos provided by Dr. Reiter

• The sinus surgery videos/images will be the initial testing environment for our algorithm.
• We will get these videos directly from Dr. Reiter at the beginning of our project.
• Resolved: 2/20/15
• Status: Resolved

* Video-CT Registration algorithm from Seth Billings

• The CT registration is the planned end result (not including the stretch goal of augmented reality). Our image processing must be compatible with Seth Billings' algorithm.
• We plan on getting this from Seth Billings directly or through Dr. Reiter. This is a key part of our project; however, if we don't get it, we will be creating a standalone contour detection package.
• Planned date for resolving: 3/30/15
• Status: Resolved (Not applicable anymore to current timeline)

* Sinus surgery video with corresponding CT images

• The sinus surgery video with corresponding CT images will allow us to test our integration with the registration algorithm without creating a phantom.
• Planned date for resolving: 5/6/15
• Status: Not resolved

1. Research: Decide on which technique we want to use and which paper(s) we will develop our algorithm from
   • Planned Date: 2/27/15
   • Expected Date: 2/27/15
   • Status: Complete
2. Design: Construct and modify a draft of our algorithm with pseudocode
   • Planned Date: 3/06/15
   • Expected Date: 3/10/15
   • Status: Complete
3. Implementation: Have a package that can successfully run contour detection on an image or video
   • Planned Date: 3/20/15
   • Expected Date: 3/23/15
   • Status: Complete
4. Testing: Make sure our package is successful with our surgical video data based on hand labeled ground truth
   • Planned Date: 3/27/15
   • Completed Date: 5/4/15
   • Status: Complete
5. Integration with CT Registration: Ensure our generated occlusion contours correctly register to the CT images by calculating edge normals
   • Planned Date: 4/10/15
   • Completed Date: 5/4/15
   • Status: Complete

• Project Plan
  • Project plan presentation
  • Project plan proposal
• Project Background Reading
  • See Bibliography below for links.
• Project Checkpoint
  • Project Checkpoint Presentation
• Paper Seminar Presentations
  • John Lee:
    • Paper presentation
    • Paper critical review
    • Selected paper
  • Kyle:
    • Paper Presentation
    • Paper Critical Review
    • Selected Paper
  • Calvin:
    • Paper Critical Review
    • Paper Presentation
    • Selected Paper
• Project Final Presentation
  • PDF of Poster
• Project Final Report
  • Final Report

• Contour Detection and Hierarchical Image Segmentation P. Arbelaez, M. Maire, C. Fowlkes and J. Malik. IEEE TPAMI, Vol. 33, No. 5, pp. 898-916, May 2011.
• D. Burschka , M. Li , M. Ishii , R. Taylor and G. D. Hager “Scale-invariant registration of monocular endoscopic images to CT-scans for sinus surgery”, Med. Image Anal., vol. 9, no. 5, pp.413 -426 2005
• Berthold K. Horn ; Brian G. Schunck; Determining Optical Flow. Proc. SPIE 0281, Techniques and Applications of Image Understanding, 319 (November 12, 1981); doi:10.1117/12.965761.