The main goal of the project is to provide registration of preoperative images (OCT, fundus) to intraoperative microscopic stereo feed, with image overlay of specific landmarks from the surgeon during preoperative planning, using C++ and modern hardware acceleration (multicore, GPU)

• Students: Vincent Ng
• Mentor(s): Gregory Hager, Marcin Balicki, Rogerio Richa, Russ Taylor

During the preoperative procedure, a wide angled fundus scan is taken of the retina. This interior surface of the eye allows the surgeon to pinpoint certain distinct or visually interesting locations to inspect. An Optical Coherence Tomography (OCT) scan is done at various locations to provide a higher resolution image of the cross section and layer structure of the eye.

Assisting the peeling of Internal Limiting Membrane (ILM) is a well target for this plan. Inside the retina, macular holes develop in the retina with no obvious cause or warning. ILM is a common procedure done during Macular Hole Surgery to repair these holes. Here, the OCT scan can be used to determine which portion of the surface to initiate peeling, and integrate it to the microscope feed.

Visual information can be useful in guiding the surgeon during an operation where he experiences limited visual detail. During the operation, the surgeon already has limited mobility of the tools inside the retina. The surgeon also has to mentally track where he wants to explore from the preoperative images, and not re-trace his previous steps. With landmarks added by the computer, the surgeon can focus more on the actual procedure. The preoperative data from the OCT image can be useful in guiding the surgeon during the procedure, by integrating it to the live video feed from the microscope and displaying in a high resolution format.

Our Specific Aims are:

1. Registration
   1. Preoperative images to intraoperative images
   2. Overlay of landmarks on live microscopic feed
2. Integration
   1. GUI for surgeons
   2. Frameworks – CISST, OpenCV

• Minimum
  • Understand and implement SURF
  • Test/Validate program using manually picked annotations as ground truth
  • Data: Surgical and non-surgical

• Expected
  • Multicore processor speedup
  • Simple GUI that marks annotation for surgeon
  • Initial Image processing for real surgery data

• Maximum
  • Deploy solution into OR.
  • Integrate fundus/OCT data to surgeon’s screen
  • Real-time image processing with GPU acceleration

The major part of the problem involves registration. The registration of the images is feature-based. The preoperative data will be obtained as video (continuous images) or images, and tests can be done to track a specific feature through a specific time interval. Existing tools are available to extract features and match images (SURF, SIFT, ICP). Initial testing will be done on a short video, with the first 10 frames to provide feature detection, and the rest of the video to be used for registration. Validation methods have to be developed to ensure the accuracy and repeatability of the feature detection and registration.

To ensure the quality of feature extraction and matching of data, initial image processing for surgery video is important. Existing tools are openCV segmentation utility function, CISST framework to integrate with surgical tools.

1. Milestone name: Intra-intra matching program
   • Planned Date: Week of March 7th
   • Realized Date: March 13th
   • Status: Completed. Working program, 200ms on a Intel Core i5 (single thread)
2. Milestone name: Integrating program into existing CISST framework with preop-intra matching program
   • Planned Date: Week of March 15st
   • Realized Date: March 31st
   • Status: Completed. Working program.
3. Milestone name: Validation of program, using real intra-operative data
   • Planned Date: Week of March 21st
   • Expected Date: Week of April 18th
   • Status: Completed. Updated Algorithm.
4. Milestone name: Integrating program with other filters
   • Planned Date: Week of April 25th
   • Realized Date: Week of April 25th
   • Status: Completed. Working with Rogerio's Template Tracker
5. Milestone name: Updating Algorithm, cleaning up code for CISST filters integration
   • Planned Date: Week of April 25th
   • Expected Date: Week of May 9th
   • Status: Completed. Working with Rogerio's Template Tracker

• Removing GPU/multi-core speedup

Currently, we have a working intra-intra registration program. This program takes the first video frame and calculate the features. It then tries to match these features with the following video stream(calculating features for these new frames, match the two sets of feature using normalized cross correlation, then finding a suitable homography transformation between the two matched sets of points). The user can click on any point in the initial frame, and the program uses the transformation in each frame to overlay that point in the new frame.

This process takes 200ms per new frame using a single core. Analyzing the algorithm, doing preop-intra registration should take a shorter time(by minimizing the number of initial features). Speed up using multi-core and the CISST pipelining framework will help in the time.

CISST Platform: Currently working on a Template/SURF tracker hybrid. The video stream will use a (fast, realtime) Template Tracker Filter initially. When the Template Tracker does not have a good confidence level, it switches over to the SURF tracker, which runs at 200ms. The Template Tracker gets the transformation from the SURF tracker, and takes over when the resulting transformation gives a high confidence level.

• Project Plan
  • Initial Project Proposal
  • Initial Project Presentation(pdf) (pptx)
• Project Background Reading
  • SURF: Speeded Up Robust Features
  • Seminar Paper Presentation
  • Seminar Paper Critical Review
• Project Checkpoint
  • Project Checkpoint 4/5/2011 (pptx)
  • Link to sample video Slow down to 0.06x speed. Actual speed ~ 200ms per frame
  • Zip File containing images extracted from above video
• Project Final Presentation
  • Poster Teaser
  • Poster
• Project Final Report
  • Final Report

NOTE: In uploading media, upload them to your own name space or to private sub-namespace, depending on whether the general public should be able to see them.

[1]          I. Fleming S. Voros, B. Vágvölgyi, Z.A. Pezzementi, J. Handa, R.H. Taylor,G.D. Hager. “Intraoperative Visualization of Anatomical Targets in Retinal Surgery.” 2008. WACV. <http://www.cs.jhu.edu/~rht/RHT%20Papers/2008/wacv2008_Fleming.pdf>.

[2]          Hong Shen Stewart, C.V.; Roysam, B.; Gang Lin; Tanenbaum, H.L. “Frame-rate Spatial Referencing Based on Invariant Indexing and Alignment with Application to Online Retinal Image Registration.” Mar. 2003. Web. <http://ieeexplore.ieee.org/xpl/freeabs_all.jsp?arnumber=1182101>.

[3]           V. Mester, F. Kuhn. “Internal Limiting Membrane Removal in the Management of Full-Thickness Macular Holes.” Macular Surgery. June 1999. Web. 14 Feb. 2011. <http://www.ncbi.nlm.nih.gov/pubmed/10926987>.

[4]          Johannes P.W. Grimm, Clemens Wagner and Reinhard Männer. “Interactive RealTime Simulation of ILM.” 2004. Web. <http://www.springerlink.com/content/64grw7h72r7l6t75/>.

[5]          Paul M. Novotny, Jeff A. Stoll, Nikolay V. Vasilyev, Pedro J. Del Nido, Pierre E. Dupont, Todd E. Zickler, Robert D. Howe. “GPU Based Real-time Instrument Tracking with Three-dimensional Ultrasound.” July 2007. Web. <http://www.cs.jhu.edu/~rht/RHT%20Papers/2010/SPIE%20Ophtalmology%202010%20-%20Liu.pdf>.

[6]    Herbert Bay , Andreas Ess , Tinne Tuytelaars, and Luc Van Gool. “Speeded-Up Robust Features(SURF).” 2008. Web. <http://www.vision.ee.ethz.ch/~surf/papers.html>. 

Here give list of other project files (e.g., source code) associated with the project. If these are online give a link to an appropriate external repository or to uploaded media files under this name space.