Team member: Haochen Wei

Mentor: Peter Kazanzides, Dayeon Kim

Last updated: May 11st

We propose a solution to provide an endoscopic surgery training simulator for training rookie surgeons and solve the problem of limited training sources.

The simulator will locate the endoscopic tools via stereo camera and render the model of tools on the screen. The tools are overlaid on the virtual simulation environment to create a training setup. The whole system behaves just like “Augment Reality”, but the output is on a 2D screen just like the regular endoscopic surgery.

The simulator can locate the tools at millimeter level accuracy and can maintain the performance of at least 30FPS constantly within a normal computer.

• Students: Haochen Wei
• Mentor(s): Peter Kazanzides; Dayeon Kim

Laparoscopic surgery has many advantages over traditional open surgery. During laparoscopic surgery, an endoscope was inserted through a keyhole on the patient’s abdominal wall. The real-time image captured by the endoscope is displayed on a monitor, and based on the visual guidance, the surgeon manipulates laparoscopic instruments to perform the procedure.

Due to the difficulty, it required lots of training from a trainee to a qualified surgeon. Recently several Augmented Reality based surgical simulator has been developed, but most of them only considered several specific types of procedure and ignore the robot-assist surgery system integration.

We use the Augments reality technic to create a simulation environment to train the surgeon. The simulator will locate the position and pose of the tools and will overlay tools on the virtual anatomy structure. Furthermore, Da Vinci also required some assistants to hold tools near the bed, and their technic is like regular endoscopic surgery, thus the system can be combined with Da Vinci System to use the existed anatomy simulator for bedside assistant training. And can be integrated with the Da Vinci System build-in simulator to train the whole team together.

• Minimum:
  1. Point clouds of hand-held (and robotic) instruments from sensorized phantom. 3/25
  2. 2D video from the perspective of the endoscope output to the standalone monitor.4/8

• Expected:
  1. Overlay the patient’s 3D model with the obtained point cloud. (Proof of concept). 4/22
  2. Collision detection between the tools and virtual organ model.4/28

1. Point clouds of hand-held (and robotic) instruments from sensorized phantom:

Passive method: Using 4 cameras fixed at corners of the phantom, 2 for each side. Then using the conventional triangulation method to obtain the point cloud. P: Cheap; High Accuracy; C: Might encounter problems with thin tools.

Active method: Using 2 Time of Flight RGBD cameras to obtain the point cloud, one 1for each side. P: Easy for obtaining point clouds C: Low accuracy on the close range; Relatively expensive

The current decision is to use one existing stereo camera system alone. Take advantage of the existing API and the constrain from epipolar geometry to increase the accuracy and lower the cost. For the current feature detection algorism, extra 2 cameras won't take more advantages. Due to the supply chain problem and time constrain of CIS2 project, the ToF camera will not be considered now and the team will focus on using the traditional cameras only.

2.2D video from the perspective of the endoscope output to the stand-alone monitor

If the endoscope is fixed on the da Vinci Robot: Forward Kinematics provided from robot. Registration between robot and phantom is needed. Render the point cloud accordingly.

If the endoscope is hand-held: AR-tag attached to the endoscope to localize the position and pose of endoscope. Render the point cloud accordingly.

3. Overlay the patient’s 3D model with the obtained point cloud. (Proof of concept)

Place some virtual cube/cylinders within the virtual space inside the phantom. Overlay the point cloud on it.

Might pursue rendering a 3d model directly using the unity engine.

4.Collision detection between the tools and virtual organ model.

Detect how many points are located within the virtual cube/cylinders. If the number of points exceeds a certain threshold, then report the collision. A possible challenge is the quality of the point cloud.

1. Milestone name: Finish comparison, Obtain, mount and calibrate all the cameras.
   • Planned Date: 3/11
   • Expected Date: 3/11
   • Status: Completed at 3/11
2. Milestone name: Finishing improving the point cloud data.
   • Planned Date: 3/25
   • Expected Date: 3/25
   • Status: Completed at 3/27
3. Milestone name: Achieve Real time rendering (minimal deliverable)
   • Planned Date: 4/8
   • Expected Date: 4/8
   • Status: Completed at 4/18
4. Milestone name: Collision detection
   • Planned Date: 4/28
   • Expected Date: 4/28
   • Status: Done at 4/29

cis_2_project_poster_group20.pdf

• Project Plan
  • project_plan_mixedreality_wh.pptx
  • project_proposal.pdf
• Project Background Reading
  • See Bibliography below for links.
• Project Checkpoint
  • project_plan_mixedreality_checkpoint.pptx
• Paper Seminar Presentations
  • cis2_background_reading_team20.pdf
  • seminar_mixedreality_ppt.pdf
• Project Final Presentation
  • cis_2_project_poster_group20.pdf
• Project Final Report
  • cis2_team20_finalreport.pdf

Qian L., Zhang X., Deguet A., Kazanzides P. (2019) ARAMIS: Augmented Reality Assistance for Minimally Invasive Surgery Using a Head-Mounted Display. In: Shen D. et al. (eds) Medical Image Computing and Computer Assisted Intervention – MICCAI 2019.https://doi.org/10.1007/978-3-030-32254-0_9

Azimi, Ehsan et al. “An Interactive Mixed Reality Platform for Bedside Surgical Procedures.” MICCAI (2020).https://intuitivecomputing.jhu.edu/publications/2020-miccai-azimi.pdf

Zollhöfer, Michael & Stotko, Patrick & Görlitz, Andreas & Theobalt, Christian & Nießner, Matthias & Klein, Reinhard & Kolb, Andreas. (2018). State of the Art on 3D Reconstruction with RGB‐D Cameras. Computer Graphics Forum. 37. 625-652. 10.1111/cgf.13386.

Han, X., Laga, H. Bennamoun, M. Image-Based 3D Object Reconstruction: State-of-the-Art and Trends in the Deep Learning Era. IEEE Transactions On Pattern Analysis And Machine Intelligence. 1578-1604 (2021,5), http://dx.doi.org/10.1109/TPAMI.2019.2954885

Dickey, R. M., Srikishen, N., Lipshultz, L. I., Spiess, P. E., Carrion, R. E., & Hakky, T. S. (2016). Augmented reality assisted surgery: a urologic training tool. Asian journal of andrology, 18(5), 732–734. https://doi.org/10.4103/1008-682X.166436

R. Frikha, R. Ejbali and M. Zaied, “Handling occlusion in Augmented Reality surgical training based instrument tracking,” 2016 IEEE/ACS 13th International Conference of Computer Systems and Applications (AICCSA), 2016, pp. 1-5, doi: 10.1109/AICCSA.2016.7945729.

Fritz, T., Stachel, N. & Braun, B. (2019). Evidence in surgical training – a review. Innovative Surgical Sciences, 4(1), 7-13. https://doi.org/10.1515/iss-2018-0026

Cano A.M., Gayá F., Lamata P., Sánchez-González P., Gómez E.J. (2008) Laparoscopic Tool Tracking Method for Augmented Reality Surgical Applications. In: Bello F., Edwards P.J.E. (eds) Biomedical Simulation. ISBMS 2008. Lecture Notes in Computer Science, vol 5104. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-70521-5_21

Following google drive contain every document of this project https://drive.google.com/drive/folders/1k3WnxH9BptmxdAbfzAm9MQ4JeIgQkHth?usp=sharing