Last updated: March.1st, 23:30

Development of context situational awareness method using Signed Distance Field(SDF) with different feedback modalities.

• Students: Hisashi Ishida, Juan Barragan Antonio
• Mentor(s): Max Li, Adnan Munawar, Prof. Misha Kazhdan, Prof. Russ Taylor

Medical Background
Mastoidectomy is a common procedure to treat an infection in mastoid air cells. The surgeon will cut the ear remove the hollow bone by drilling the temporal bone. This procedure can be highly challenging because it requires high precision to preserve the critical structures such as semicircular canals, sigmoid sinus, and facial nerves. Damaging those important structures can result in subsequent fatal complications, which vary from temporal or permanent loss of ringing in the ear or dizziness, partial or total loss of hearing, facial nerve palsy, or even death of the patients.

Technical Background
Previous work from Hopkins has developed a virtual simulator, AMBF, for volumetric drilling. This simulation leverages segmented CT images to create an anatomically accurate drilling simulator environment which the users observe via a stereoscopic display. This simulator is particularly powerful because it allows users to practice surgical procedures while also generating data for surgical computer vision algorithms.

Currently, the simulation environment does not provide safety cues related to the distance between the drill and critical anatomies. Although a warning message is provided when the user collides with an anatomy, this is not enough feedback to teach the train how to avoid such a dangerous situation in a real procedure. Secondly, the simulator lacks the capability of providing haptic feedback to secure the patient safety. We believe that improving these two aspects will result in improved safety for the patient and reduce the workload of the surgeons.

Aim 1: Implement SDF method for AMBF to monitor the distance between the tool and the static anatomy in real-time. Significance: Provide fast and realtime monitoring method for all the anatomy and the tool.

Aim 2: Propose SDF based feedback with different modalities.
Significance: Provide additional safety for a VR guided surgery and reduce workload in the surgery.

Aim 3: Expand our method to adapt SDF method for the time-varying volume.
Significance: Provide useful guidance for drilling the temporal bone

Aim 4: Write a conference paper.
Significance: Share quantitative assessment results of our proposed SDF based feedback with the medical and engineering community

• Minimum: (March 16, 2022)
  1. Creating an AMBF plugin to calculate SDFs for static objects in the volumetric drilling simulation.

• Expected: (April 18, 2022)
  1. Using the calculated SDFs to improve the user situational awareness via visual cues.
  2. Using the calculated SDFs to improve the user situational awareness via haptic feedback.

• Maximum: (May 5, 2022)
  1. Internal user study.
  2. Optimizing algorithms to calculate SDFs of objects changing over time in real-time, e.g., the drilling volume.

This project will be divided into three different phases where each one of the phases addresses a different objective. Phase 1 will be focusing on integrating SDF functions into the drilling simulation for objects that do not change their volume. Phase 2 will be focusing on using the calculated SDF to improve the situational awareness of the user via haptic or visual feedback. The last phase of the project will be concerned with optimizing the calculation of SDF to perform them in real-time.

The goal for this phase is to implement SDF calculation functions for objects whose volume is not changing over time, i.e., anatomies that are not being drilled on. Provided that static objects' SDFs do not change, this calculation can be performed once as an initialization step at the beginning of the simulation. Then, the calculated SDFs can be stored in a look-up table to be used while the user is interacting with the simulator. The diagram below summarizes the different components of the proposed system.

The implementation of the SDF will be based on the method proposed by Saito and Toriwaki, 1994[2]. This method was chosen because it allows for parallelization of the calculations and works with volumes represented as voxel grids. These functions will be implemented in c++ and compiled as a shared library. Then, the library will be added to the simulation utilizing the AMBF plugins capability.

The goal for this phase is to implement improved situational awareness using different feedback modalities: Visual feedback and Haptic feedback. For visual feedback, our plan is to overlay a warning message saying when the drill gets too close to the important anatomy. Using the stored look-up table of SDF for each structure, we can monitor the distance between the drill and the anatomy in real-time.

For haptic feedback, we are adopting the virtual fixture method[3]. Our proposal is to provide forbidden regional virtual fixtures to avoid collision with the important anatomy. We will compare the proposed method with the mesh-based VF[4] for the effectiveness of the method.

More details for the SDF implementation will be added for finishing Phase 1 of the project. Finishing phase 1 will provide us with information about how fast the algorithms are running in the simulation and how much faster they need to be for optimal performance in real-time. This information will guide the modification of the algorithms for phase 3.

describe dependencies and effect on milestones and deliverables if not met

• Access to a Linux workstation: needed by 02/18/2022 to avoid delay in all milestones.
• Access to haptic device: needed by 03/23/2022 to avoid delay in the haptic feedback development and user study.
• IRB approval: Needed by 03/31/2022 to avoid delays in the user study.
• VR device: Needed by 03/21/2022 to avoid delay in the user study. In case of not acquiring the VR headset the user study can also be performed on a computer screen.

1. Milestone name: Offline EDT calculation
   • Planned Date: March 16
   • Expected Date: March 16
   • Status: In progres
2. Milestone name: Situational awareness interface
   • Planned Date: April 18
   • Expected Date: April 18
   • Status: Not started
3. Milestone name: Online EDT calculation
   • Planned Date: May 5
   • Expected Date: May 5
   • Status: Not started

• Project Plan
  • Project plan presentation
  • Project plan proposal
• Project Background Reading
  • See Bibliography below for links.
• Project Checkpoint
  • Project checkpoint presentation
• Paper Seminar Presentations
  • here provide links to all seminar presentations
• Project Final Presentation
  • PDF of Poster
• Project Final Report
  • Final Report
  • links to any appendices or other material

[1] A. Munawar et al., “Virtual reality for synergistic surgical training and data generation,” Computer Methods in Biomechanics and Biomedical Engineering: Imaging & Visualization, vol. 0, no. 0, pp. 1–9, Nov. 2021, doi: 10.1080/21681163.2021.1999331.​

[2] T. Saito and J.-I. Toriwaki, “New algorithms for euclidean distance transformation of an n-dimensional digitized picture with applications,” Pattern Recognition, vol. 27, no. 11, pp. 1551–1565, Nov. 1994, doi: 10.1016/0031-3203(94)90133-3.​

[3] S. A. Bowyer, B. L. Davies and F. Rodriguez y Baena, “Active Constraints/Virtual Fixtures: A Survey,” in IEEE Transactions on Robotics, vol. 30, no. 1, pp. 138-157, Feb. 2014, doi: 10.1109/TRO.2013.2283410.​

[4] Li, Z., Gordon, A., Looi, T., Drake, J., Forrest, C., & Taylor, R. H. (2020, October). Anatomical mesh-based virtual fixtures for surgical robots. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (pp. 3267-3273). IEEE.

Here give list of other project files (e.g., source code) associated with the project. If these are online give a link to an appropriate external repository or to uploaded media files under this name space (2022-01).