======Software for an Intra-Operative "Kinect"====== **Last updated: 5/14/2017 10pm** ======Summary====== We are working to develop the software for a system to do 3D reconstruction based on a structured light approach using a flexible endoscope. We are currently working with a basic camera in place of an endoscope to allow us to focus on developing the appropriate software, but making sure to make the software portable so it can be used in a system with a flexible endoscope. * **Students:** Shohini Ghosh, Elli Tian * **Mentors:** Dr. Austin Reiter, Dr. Russell Taylor ======Background, Specific Aims, and Significance====== Video-guided minimally invasive surgery can reduce adverse effects for patients through the use of an endoscope that is inserted through a small incision instead of open surgery. However, endoscopy is limited in that the surgeon only sees a geometrically distorted 2D image of the surgical site. It would be desirable for the surgeon to have more accurate information on the surface topology of the surgical site from a 3D imaging system as this can guide diagnosis and surgery. For example, a collapsed airway that can be difficult to see in a 2D image could be clearly visualized using a 3D reconstruction of the space. Stereo reconstruction has been used to do reconstruction in endoscopy, but these systems may not perform as well on human tissue with few features. A structured light approach to 3D reconstruction can address this issue, as the projected light can create feature points on tissue where there were previously no distinguishable features. By using a small camera and laser fiber that projects a pseudo-random pattern of green dots, we hope to develop software that will allow for precise and accurate 3D reconstruction of a variety of objects. Our goal is to make this software portable and flexible, so that it can eventually be adapted for real-time use during surgery with the laser fiber inserted down the working channel of a flexible endoscope. Our primary aims for the project include: 1. Real-time 3D reconstruction based on structured light approach. This reconstruction will allow surgeon to understand the shape of the space in which they are operating. 2. Real-time camera tracking simultaneously with reconstruction. This will indicate to the surgeon where their endoscope is relative to what they see through the endoscope. ======Deliverables====== * **Minimum:** - Rigid fixation method of camera to laser (4/12/17) - Template for laser pattern in appropriate coordinates relative to camera (4/12/17) - Code to compute depth map based on camera's field of view (4/22/17) - Assess accuracy using setup with objects at known positions relative to camera, need at least millimeter accuracy * **Expected:** - Code to create 3D reconstruction of simple objects and track camera movement relative to static scene (5/6/17) - Assess accuracy using simple objects with precisely known 3D shapes, need at least millimeter accuracy * **Maximum:** - Code to create 3D reconstruction of complex objects and track camera movement relative to static scene (5/13/17) - Assess accuracy using complex objects with precisely known 3D shapes, need at least millimeter accuracy ======Technical Approach====== ===3D Reconstruction=== __Mechanical Setup__ * Camera: Chameleon 1.3 MP Color USB 2.0 * Lens: Fujinon varifocal lens (model YV2.8x2.8SA-2) * Laser fiber manufactured by 3Dintegrated, emits a 520nm light at 50mW with approximately 4 dots per square centimeter * 3D printed frame to hold the camera and laser fiber {{:courses:446:2017:446-2017-13:screen_shot_2017-05-14_at_8.18.47_pm.png?nolink&300|}} __Camera Calibration__ * Checkerboard calibration (http://www.vision.caltech.edu/bouguetj/calib_doc/) __Camera-Laser Calibration__ * Collected training images at known distances {{:courses:446:2017:446-2017-13:screen_shot_2017-05-14_at_8.16.29_pm.png?nolink&300|}} * Find dots in pattern using MSER * Identify dots by correspondences with template training image * Identify dot correspondences using window matching with dots in expected location based on translation of center of laser pattern {{:courses:446:2017:446-2017-13:screen_shot_2017-05-12_at_2.48.51_am.png?nolink&300|}} * Build lookup table relating (x,y) pixel coordinates for each dot across images at varying distances * Fit polynomial to relationship between x pixel coordinate and distance for each dot in laser pattern {{:courses:446:2017:446-2017-13:screen_shot_2017-05-14_at_9.48.09_pm.png?nolink&300|}} __3D Reconstruction__ * Find dots in testing image using MSER * Identify dot correspondences with template training image based on lookup table * Calculate distance using relationship between x pixel coordinate and distance for corresponding dot in training images * Real world z coordinate = distance * Compute real world x and y coordinates using real world z coordinate based on focal length and principal point {{:courses:446:2017:446-2017-13:screen_shot_2017-05-14_at_9.48.16_pm.png?nolink&300|}} {{:courses:446:2017:446-2017-13:screen_shot_2017-05-14_at_9.48.28_pm.png?nolink&300|}} {{:courses:446:2017:446-2017-13:screen_shot_2017-05-14_at_9.48.34_pm.png?nolink&300|}} ===Finding Dots in Realistic Setting=== * Simulate realistic setting with ham {{:courses:446:2017:446-2017-13:screen_shot_2017-05-14_at_9.48.42_pm.png?nolink&300|}} * No other light sources besides laser * Compare SIFT and MSER * Evaluate by counting correctly found dots, false positives, and false negatives * SIFT finds more dots but also has more false positives than MSER SIFT: {{:courses:446:2017:446-2017-13:screen_shot_2017-05-14_at_10.04.12_pm.png?nolink&300|}} MSER: {{:courses:446:2017:446-2017-13:screen_shot_2017-05-14_at_10.04.28_pm.png?nolink&300|}} ======Dependencies====== - Access to laser & camera ✔ - Access to lab space for storage/testing ✔ - Obtaining Tae Soo Kim's prior work with 3D reconstruction using endoscope ✔ - Reliable fixation for camera and laser ✔ - Development of testing setup ✔ - SLAM depends on calibration - Effect on milestones: 3D reconstruction code will be difficult to develop without accurate calibration - Plan for resolving: Seek guidance from Dr. Reiter if problems arise, calibration almost complete - **Not completed due to insufficient accuracy of calibration** ======Milestones and Status ====== - Milestone name: Rigid Fixation of Camera and Laser * Planned Date: 2/25/17 * Expected Date: 4/12/17 * Completed: 4/14/17 - Milestone name: Develop Calibration Code * Planned Date: 3/11/17 * Expected Date: 4/22/17 * Completed: 4/20/17 - Milestone name: Collect Laser Pattern Data * Planned Date: 3/4/17 * Expected Date: 4/12/17 * Completed: 4/14/17 - Milestone name: Create Testing Setup for Calibration * Planned Date: 3/11/17 * Expected Date: 4/15/17 * Completed: 4/15/17 - Milestone name: Develop Testing Scripts for Calibration Code * Planned Date: 3/11/17 * Expected Date: 4/15/17 * Completed: 4/15/17 - Milestone name: Collect Data to Test Calibration Accuracy * Planned Date: 3/18/17 * Expected Date: 4/22/17 * Completed: 4/22/17 - Milestone name: Identify Best Dot Detection Method (ADDED SINCE CHECKPOINT) * Planned Date; 5/13/17 * Expected Date: 5/13/17 * Completed: 5/13/17 - Milestone name: Develop SLAM Code * Planned Date: 4/15/17 * Expected Date: 5/17/17 * Status: Not pursued due to calibration inaccuracy - Milestone name: Plan Testing for SLAM * Planned Date: 4/1/17 * Expected Date: 4/27/17 * Status: Not pursued due to calibration inaccuracy - Milestone name: Collect Data for Testing SLAM * Planned Date: 4/15/17 * Expected Date: 5/13/17 * Status: Not pursued due to calibration inaccuracy ======Reports and presentations====== * Project Plan * {{:courses:446:2017:446-2017-13:project_proposal_presentation_1_.pdf|Project plan presentation}} * {{:courses:446:2017:446-2017-13:projectproposal.pdf|Project plan proposal}} * Project Background Reading * See Bibliography below for links. * Project Checkpoint * {{:courses:446:2017:446-2017-13:checkpoint_presentation.pdf|Project checkpoint presentation}} * Paper Seminar Presentations * {{:courses:446:2017:446-2017-13:seminar_report.pdf|Elli seminar report}} * {{:courses:446:2017:446-2017-13:seminar_presentation.pdf|Elli seminar presentation}} * {{:courses:446:2017:446-2017-13:shohiniseminarwriteup1.pdf|Shohini seminar report}} * {{:courses:446:2017:446-2017-13:shohini_seminar_presentation.pdf|Shohini seminar presentation}} * Project Final Presentation * {{:courses:446:2017:446-2017-13:cis2poster.pdf|Final Poster}} * Project Final Report * {{:courses:446:2017:446-2017-13:finalreport_1_.pdf|Final Report}} * Code * https://bitbucket.org/etian1/3dkinect ======Project Bibliography======= **Reading List:** - Tae Soo Kim’s reports from CIS 2 project (and maybe code and MS thesis) - Patent application by Tae Soo Kim and Drs. Taylor and Reiter: http://www.freepatentsonline.com/y2016/0143509.html - SLAM algorithm: https://github.com/simondlevy/BreezySLAM - Gamini Dissanayake, M.W.M., Newman, P., Clark, S., Durrant-Whyte, H.F., and Csorba, M. “A Solution to the Simultaneous Localization and Map Building (SLAM) Problem.” IEEE Transactions on Robotics and Automation 17.3. (2001): 229-241. Print. - Moutney, P., Stoyanov, D., Davison, A., and Yang, G.Z. “Simultaneous Stereoscope Localization and Soft-Tissue Mapping for Minimal Invasive Surgery.” Medical Image Computing and Computer-Assisted Intervention 4190. (2006): 347-354. Print. - M. Chan, W. Lin, J Qu, “Miniaturized three-dimensional endoscopic imaging system based on active stereovision,” in Applied Optics. 42(10), 1888-1898 (2003) - N. T. Clancy, D. Stoyanov, L. Maier-Hein, A. Groch, G. Yang, D. Elson, “Spectrally encoded fiber-based structured lighting probe for intraoperative 3D imaging,” in Biomedical Optics Express. 2(11), 3119-3128 (2011). ======Other Resources and Project Files====== All code developed for this project can be found in the BitBucket repository at https://bitbucket.org/etian1/3dkinect.