Last updated: May 1, 2017

Stereo camera have not been widely used in da Vinci system. We want to provide useful stereo camera tools to DVRK. We mainly want to complete two tools: hand-to-eye calibration between robot and stereo camera and then registration known and unknown surface to the robot.

• Students: Mengze Xu and Joonghyun (Peter) Ahn
• Mentor(s): Preetham and Anton

You may want to include a picture or two here.

Current method for registering unknown surface to the Patient Side Manipulator (PSM) of Research Kit of da Vinci Surgical System (dVRK) involves manually manipulating the tool tip of the PSM to touch the desired surface. To replace the current registration process, we developed a software package that processes RGB images from a stereo camera system to register the phantom model to the PSM. New registration method will expedite the research and surgical processes that need the target surface registered to the robot.

Current surface registration method requires to be done every time the surface is moved, and touching the surface may be unfeasible in some cases (organ surfaces). Manual manipulation of the tool tip has low accuracy especially on elastic surfaces. Thus, a new non-tactile surface registration method is vital for the success and accuracy of the dVRK related procedure.

• Minimum: (Expected by March 20) [Completed by March 31]
  1. deliverable 1 Codes for hand-to-eye calibration
  2. deliverable 2 Result of error testing

• Expected: (Expected by April 10) [In Progress]
  1. deliverable 1 Codes for detecting predefined surfaces
  2. deliverable 2 Result of error testing

• Maximum: (Expected by May 1) [Not Started]
  1. deliverable 1 Codes for detecting unknown surfaces
  2. deliverable 2 Result of error testing

AruCo marker is used to help estimate the pose of the camera to the world. DVRK will provide pose of the PSM robot. Data is collected through ROS and processed in Matlab. Kronecker Product Method is used to solve AX=YB calibration problem. Edge detection will be used to detect deformed surface.

Opaque Phantom
PrimeSense Camera

1. Milestone name: Result of hand-eye calibration
   • Planned Date: Mar 20
   • Completed Date: Mar 31
   • Status: Completed. Accuracy result in millimeters
2. Milestone name: Predefined Surface Detection Program
   • Planned Date: April 10
   • Expected Date: May 1
   • Status: Completed. Accuracy result in millimeters.
3. Milestone name: Developing our own Stereo Camera System using two R200 Cameras
   • Planned Date: April 10
   • Completed Date: April 24
   • Status: Completed. (Several adjustments to the hardware setup; final setup is two cameras located perpendicular to each other pointing 45 degrees downward toward the phantom)

• Project Plan
  • Project plan presentation
  • Project plan proposal
• Project Background Reading
  • See Bibliography below for links.
• Project Checkpoint
  • Project checkpoint presentation
• Paper Seminar Presentations
  • Paper Seminar Review (Mengze)
  • Paper Seminar Presentation (Mengze)
  • Paper Seminar Review (Peter)
  • Paper Seminar Presentation (Peter)
• Project Final Presentation
  • PDF of Poster
• Project Final Report
  • Final Report
    • Without Mentor Evaluations. They are on the way.
  • All the codes in bitbucket repository.

[1] Azad P., Asfour T., Dillmann R. “Stereo-Based vs. Monocular 6-DoF Pose Estimation Using Point Features: A Quantitative Comparison.” In: Dillmann R., Beyerer J., Stiller C., Zöllner J.M., Gindele T. (eds) Autonome Mobile Systeme 2009. Informatik aktuell. Springer, Berlin, Heidelberg

[2] P. Azad. Visual Perception for Manipulation and Imitation in Humanoid Robots. PhD thesis, Universit ̈at Karlsruhe (TH), Karlsruhe, Germany, 2008.

[3] P. Azad, T. Asfour, and R. Dillmann. “Combining Harris Interest Points and the SIFT Descriptor for Fast Scale-Invariant Object Recognition.” In IEEE/RSJ In- ternational Conference on Intelligent Robots and Systems (IROS), St. Louis, USA, 2009.

[4] Mili Shah, Roger D Eastman, and Tsai Hong. An overview of robot-sensor calibration methods for evaluation of perception systems. In: Proceedings of the Workshop on Performance Metrics for Intelligent Systems. ACM, 2012, pp. 1520. isbn: 1450311261.

[5] http://robotics.stackexchange.com/questions/7163/hand-eye-calibration

[6] https://hal.inria.fr/inria-00590087/file/DornaikaHoraud-tra.pdf

[1] https://www.uco.es/investiga/grupos/ava/node/26

[2] http://wiki.ros.org/aruco