Software for an Intra-Operative "Kinect"
Last updated: 5/14/2017 10pm
Summary
We are working to develop the software for a system to do 3D reconstruction based on a structured light approach using a flexible endoscope. We are currently working with a basic camera in place of an endoscope to allow us to focus on developing the appropriate software, but making sure to make the software portable so it can be used in a system with a flexible endoscope.
Students: Shohini Ghosh, Elli Tian
Mentors: Dr. Austin Reiter, Dr. Russell Taylor
Background, Specific Aims, and Significance
Video-guided minimally invasive surgery can reduce adverse effects for patients through the use of an endoscope that is inserted through a small incision instead of open surgery. However, endoscopy is limited in that the surgeon only sees a geometrically distorted 2D image of the surgical site. It would be desirable for the surgeon to have more accurate information on the surface topology of the surgical site from a 3D imaging system as this can guide diagnosis and surgery. For example, a collapsed airway that can be difficult to see in a 2D image could be clearly visualized using a 3D reconstruction of the space. Stereo reconstruction has been used to do reconstruction in endoscopy, but these systems may not perform as well on human tissue with few features. A structured light approach to 3D reconstruction can address this issue, as the projected light can create feature points on tissue where there were previously no distinguishable features.
By using a small camera and laser fiber that projects a pseudo-random pattern of green dots, we hope to develop software that will allow for precise and accurate 3D reconstruction of a variety of objects. Our goal is to make this software portable and flexible, so that it can eventually be adapted for real-time use during surgery with the laser fiber inserted down the working channel of a flexible endoscope.
Our primary aims for the project include:
1. Real-time 3D reconstruction based on structured light approach. This reconstruction will allow surgeon to understand the shape of the space in which they are operating.
2. Real-time camera tracking simultaneously with reconstruction. This will indicate to the surgeon where their endoscope is relative to what they see through the endoscope.
Deliverables
Minimum:
Rigid fixation method of camera to laser (4/12/17)
Template for laser pattern in appropriate coordinates relative to camera (4/12/17)
Code to compute depth map based on camera's field of view (4/22/17)
Assess accuracy using setup with objects at known positions relative to camera, need at least millimeter accuracy
Expected:
Code to create 3D reconstruction of simple objects and track camera movement relative to static scene (5/6/17)
Assess accuracy using simple objects with precisely known 3D shapes, need at least millimeter accuracy
Maximum:
Code to create 3D reconstruction of complex objects and track camera movement relative to static scene (5/13/17)
Assess accuracy using complex objects with precisely known 3D shapes, need at least millimeter accuracy
Technical Approach
3D Reconstruction
Mechanical Setup
Camera: Chameleon 1.3 MP Color USB 2.0
Lens: Fujinon varifocal lens (model YV2.8×2.8SA-2)
Laser fiber manufactured by 3Dintegrated, emits a 520nm light at 50mW with approximately 4 dots per square centimeter
3D printed frame to hold the camera and laser fiber
Camera Calibration
Camera-Laser Calibration
Find dots in pattern using MSER
Identify dots by correspondences with template training image
Identify dot correspondences using window matching with dots in expected location based on translation of center of laser pattern
Build lookup table relating (x,y) pixel coordinates for each dot across images at varying distances
Fit polynomial to relationship between x pixel coordinate and distance for each dot in laser pattern
3D Reconstruction
Find dots in testing image using MSER
Identify dot correspondences with template training image based on lookup table
Calculate distance using relationship between x pixel coordinate and distance for corresponding dot in training images
Real world z coordinate = distance
Compute real world x and y coordinates using real world z coordinate based on focal length and principal point
Finding Dots in Realistic Setting
No other light sources besides laser
Compare SIFT and MSER
Evaluate by counting correctly found dots, false positives, and false negatives
SIFT finds more dots but also has more false positives than MSER
SIFT:
MSER:
Dependencies
Access to laser & camera <html><span style=“color:green;”>✔</span></html>
Access to lab space for storage/testing <html><span style=“color:green;”>✔</span></html>
Obtaining Tae Soo Kim's prior work with 3D reconstruction using endoscope <html><span style=“color:green;”>✔</span></html>
Reliable fixation for camera and laser <html><span style=“color:green;”>✔</span></html>
Development of testing setup <html><span style=“color:green;”>✔</span></html>
SLAM depends on calibration
Effect on milestones: 3D reconstruction code will be difficult to develop without accurate calibration
Plan for resolving: Seek guidance from Dr. Reiter if problems arise, calibration almost complete
Not completed due to insufficient accuracy of calibration
Milestones and Status
Milestone name: Rigid Fixation of Camera and Laser
Planned Date: 2/25/17
Expected Date: 4/12/17
Completed: 4/14/17
Milestone name: Develop Calibration Code
Planned Date: 3/11/17
Expected Date: 4/22/17
Completed: 4/20/17
Milestone name: Collect Laser Pattern Data
Planned Date: 3/4/17
Expected Date: 4/12/17
Completed: 4/14/17
Milestone name: Create Testing Setup for Calibration
Planned Date: 3/11/17
Expected Date: 4/15/17
Completed: 4/15/17
Milestone name: Develop Testing Scripts for Calibration Code
Planned Date: 3/11/17
Expected Date: 4/15/17
Completed: 4/15/17
Milestone name: Collect Data to Test Calibration Accuracy
Planned Date: 3/18/17
Expected Date: 4/22/17
Completed: 4/22/17
Milestone name: Identify Best Dot Detection Method (ADDED SINCE CHECKPOINT)
Planned Date; 5/13/17
Expected Date: 5/13/17
Completed: 5/13/17
Milestone name: Develop SLAM Code
Milestone name: Plan Testing for SLAM
Milestone name: Collect Data for Testing SLAM
Reports and presentations
Project Plan
Project Background Reading
Project Checkpoint
Paper Seminar Presentations
Project Final Presentation
Project Final Report
Code
Project Bibliography
Reading List:
Tae Soo Kim’s reports from CIS 2 project (and maybe code and MS thesis)
-
-
Gamini Dissanayake, M.W.M., Newman, P., Clark, S., Durrant-Whyte, H.F., and Csorba, M. “A Solution to the Simultaneous Localization and Map Building (SLAM) Problem.” IEEE Transactions on Robotics and Automation 17.3. (2001): 229-241. Print.
Moutney, P., Stoyanov, D., Davison, A., and Yang, G.Z. “Simultaneous Stereoscope Localization and Soft-Tissue Mapping for Minimal Invasive Surgery.” Medical Image Computing and Computer-Assisted Intervention 4190. (2006): 347-354. Print.
M. Chan, W. Lin, J Qu, “Miniaturized three-dimensional endoscopic imaging system based on active stereovision,” in Applied Optics. 42(10), 1888-1898 (2003)
N. T. Clancy, D. Stoyanov, L. Maier-Hein, A. Groch, G. Yang, D. Elson, “Spectrally encoded fiber-based structured lighting probe for intraoperative 3D imaging,” in Biomedical Optics Express. 2(11), 3119-3128 (2011).
Other Resources and Project Files
