Last updated: May.17th, 13:20

Development of context situational awareness method using Signed Distance Field(SDF) with different feedback modalities.

• Students: Hisashi Ishida, Juan Barragan Antonio
• Mentor(s): Max Li, Adnan Munawar, Prof. Misha Kazhdan, Prof. Russ Taylor

Medical Background
Mastoidectomy is a common procedure to treat an infection in mastoid air cells. The surgeon will cut the ear remove the hollow bone by drilling the temporal bone. This procedure can be highly challenging because it requires high precision to preserve the critical structures such as semicircular canals, sigmoid sinus, and facial nerves. Damaging those important structures can result in subsequent fatal complications, which vary from temporal or permanent loss of ringing in the ear or dizziness, partial or total loss of hearing, facial nerve palsy, or even death of the patients.

Technical Background
Previous work from Hopkins has developed a virtual simulator, AMBF[1], for volumetric drilling. This simulation leverages segmented CT images to create an anatomically accurate drilling simulator environment which the users observe via a stereoscopic display. This simulator is particularly powerful because it allows users to practice surgical procedures while also generating data for surgical computer vision algorithms.

Currently, the simulation environment does not provide safety cues related to the distance between the drill and critical anatomies. Although a warning message is provided when the user collides with an anatomy, this is not enough feedback to teach the train how to avoid such a dangerous situation in a real procedure. Secondly, the simulator lacks the capability of providing haptic feedback to secure the patient safety. We believe that improving these two aspects will result in improved safety for the patient and reduce the workload of the surgeons.

Aim 1: Implement SDF method for AMBF to monitor the distance between the tool and the static anatomy in real-time. Significance: Provide fast and realtime monitoring method for all the anatomy and the tool.

Aim 2: Propose SDF based feedback with different modalities.
Significance: Provide additional safety for a VR guided surgery and reduce workload in the surgery.

Aim 3: Expand our method to adapt SDF method for the time-varying volume.
Significance: Provide useful guidance for drilling the temporal bone

Aim 4: Write a conference paper.
Significance: Share quantitative assessment results of our proposed SDF based feedback with the medical and engineering community

• Minimum: (March 16, 2022)
  1. Creating an AMBF plugin to calculate SDFs for static objects in the volumetric drilling simulation.

• Expected: (April 18, 2022)
  1. Using the calculated SDFs to improve the user situational awareness via visual cues.
  2. Using the calculated SDFs to improve the user situational awareness via haptic feedback.

• Maximum: (May 5, 2022)
  1. Internal user study.
  2. Optimizing algorithms to calculate SDFs of objects changing over time in real-time, e.g., the drilling volume.

To improve the situational awareness and safety, it is necessary to calculate the distance between the drill tip and the anatomies in real-time. This information can be then utilize to prevent the user from drilling a critical anatomy or just to improve the user's awareness of close critical anatomical structures as he performs the procedure. This distance could be naively calculated by iterating over the voxel representation of the environment after each drill update. This however is a computationally inefficient algorithm that can result in high delays in the system.

As an alternative, signed distance fields (SDF) can be calculated for each of the anatomies in the simulation. A SDF function will generate a new voxel map in which the voxel's value is the minimum distance between that specific voxel location and the the reference anatomy. After obtaining a SDF for each of the different anatomies in the simulation, the calculation of the minimum distance between the drill tool tip and a specific anatomy can be speed up significantly. The procedure to obtain this distance would only require to query the SDF in the voxel where the drill is currently located. In this regard, SDF functions need to be calculated for all static anatomies at the beginning of the simulation to provided feedback to the user.

To achieved the goals of project, the work will be divided into three different phases each one with an unique and specific objective. Phase 1 will be focusing on integrating SDF functions into the drilling simulation for objects that do not change their volume. Phase 2 will be focusing on using the calculated SDF to improve the situational awareness of the user via haptic or visual feedback. The last phase of the project will be concerned with optimizing the calculation of SDF to perform them in real-time.

The goal for this phase is to implement SDF calculation functions for static objects whose volume is not changing over time, i.e., critical anatomies that are not being drilled on. For this phase, it will be assumed that the critical anatomies' SDF will be constant through out all the simulation, and therefore, the SDF calculation can be performed only once at initialization. Then, the calculated SDFs can be stored in a look-up table to be used while the user is interacting with the simulator. The diagram below summarizes the different components of the proposed system.

The implementation of the SDF will be based on the method proposed by Saito and Toriwaki, 1994\cite{saito_new_1994}. This method was chosen because it allows for parallelization of the calculations and works with volumes represented as voxel grids. These functions will be implemented in c++ and compiled as a shared library. Then, the library will be added to the simulation utilizing the AMBF plugins capability.

This phase aims to implement helpful feedback with different modalities for surgeons using the SDF look-table calculated in Phase 1. We are currently planning to implement two different feedbacks: visual warning and haptic feedback.

For visual warning, a warning message will be overlayed to the screen to provide an intuitive warning to the surgeons. This warning will be initialized when the drill passes a certain threshold using the SDF look-up table. Consequently, our method could provide situational awareness before damaging the important anatomy.

For haptic feedback we are adopting the virtual fixture method, specifically the forbidden regional VF[3], to avoid collision with the anatomy using the gradient field of the SDF. Virtual fixtures are reported to be very effective in improving both the accuracy and the safety during the skull drilling experiment[4]

Technical details of the phase 3, will be specified as soon as the phase 1 is finished. The reason for that is that the speed at which the SDF can be calculated is currently unknown. The speed at which SDFs can be calculated on the static case will determine the modifications required for real-time calculations.

describe dependencies and effect on milestones and deliverables if not met

• Access to a Linux workstation: needed by 02/18/2022 to avoid delay in all milestones.
• Access to haptic device: needed by 03/23/2022 to avoid delay in the haptic feedback development and user study.
• IRB approval: Needed by 03/31/2022 to avoid delays in the user study.
• VR device: Needed by 03/21/2022 to avoid delay in the user study. In case of not acquiring the VR headset the user study can also be performed on a computer screen.

1. Milestone name: Offline EDT calculation
   • Planned Date: March 16
   • Expected Date: March 16
   • Status: In progres
2. Milestone name: Situational awareness interface
   • Planned Date: April 18
   • Expected Date: April 18
   • Status: Not started
3. Milestone name: Online EDT calculation
   • Planned Date: May 5
   • Expected Date: May 5
   • Status: Not started

• Project Plan
  • Project plan presentation
  • Project plan proposal
• Project Background Reading
  • Project Background Reading
  • See Bibliography below for links.
• Project Checkpoint
  • Project checkpoint presentation
• Paper Background Presentations
  • Paper Background Presentations
  • Virtual Reality for Synergistic Surgical Training and Data generation
  • Image-Guided Mastoidectomy with a Cooperatively Controlled ENT Microsurgery Robot

• Project Final written report
  • Final Report

• Project Final Presentation
  • Final Presentation
• Project Final Poster
  • Final Poster
  • links to any appendices or other material
• Code and documentation:
  • https://github.com/jabarragann/volumetric_drilling/tree/devel

[1] A. Munawar et al., “Virtual reality for synergistic surgical training and data generation,” Computer Methods in Biomechanics and Biomedical Engineering: Imaging & Visualization, vol. 0, no. 0, pp. 1–9, Nov. 2021, doi: 10.1080/21681163.2021.1999331.​

[2] T. Saito and J.-I. Toriwaki, “New algorithms for euclidean distance transformation of an n-dimensional digitized picture with applications,” Pattern Recognition, vol. 27, no. 11, pp. 1551–1565, Nov. 1994, doi: 10.1016/0031-3203(94)90133-3.​

[3] S. A. Bowyer, B. L. Davies and F. Rodriguez y Baena, “Active Constraints/Virtual Fixtures: A Survey,” in IEEE Transactions on Robotics, vol. 30, no. 1, pp. 138-157, Feb. 2014, doi: 10.1109/TRO.2013.2283410.​

[4] Li, Z., Gordon, A., Looi, T., Drake, J., Forrest, C., & Taylor, R. H. (2020, October). Anatomical mesh-based virtual fixtures for surgical robots. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (pp. 3267-3273). IEEE.

Here give list of other project files (e.g., source code) associated with the project. If these are online give a link to an appropriate external repository or to uploaded media files under this name space (2022-01).