Last updated: 02/23/2023

This project aims to create a navigation software that can guide the user to maneuver the continuum manipulator (CM), a flexible surgical drill. It shall provide guidance in: (1) the CM tip pinpointing location of the penetration site (2) aligning pivot orientation and in-plane rotation of the CM shaft before penetration (3) after the CM entering the patient body, achieving the proper depth of penetration (4) amount and direction of the CM’s deformation during tissue removal. To reduce the physical and mental load enforced onto the surgeon, this project also aims to develop the software for the Microsoft HoloLens 2, a state-of-the-art optical see-through head-mounted display. Such device shall enable the user to receive all necessary navigation information while focusing on the surgical site.

• Students: Mr. Nick Zhang
• Mentors: Mr. Andreas keller, Prof. Alejandro Martin-Gomez, Prof. Mehran Armand

* The wiki page only provides a high-level overview of the project. Researchers furthering this project development are advised to first read all files from the “Project Documentation Series” folder provided on OneDrive.

* Project software user manual can be found in the UIS document.

* Demo video of using the project software.

Osteonecrosis (also called avascular necrosis) refers to the death of bone tissue due to disrupted blood supply. When this condition affecting the hip, it causes the femoral joint to collapse which leads to patient experiencing pain and loss of mobility. For treating osteonecrosis of the hip in its late stage, a total hip replacement (also called total hip arthroplasty) is often performed, where the compromised bone and tissue is removed and replaced with prosthetic parts. However, the traditional total hip replacement involves an invasive and risky procedure. In turn, to increase blood flow, stop joint destruction, and prevent osteonecrosis reaching its late stage, the surgical procedure core decompression is needed.

Core decompression involves surgical drilling the area of dead bone tissue near the femoral joint. The conventional core decompression procedure utilizes a rigid drill inserted multiple times at different angles to remove dead tissue. However, because of the varying anatomic structures in each patient, this approach using a rigid drill limits access and removal of the lesion region and may lead to unnecessary damage to the surrounding healthy tissues. To address these complications, researchers at Johns Hopkins University and Beihang University proposed to perform the core decompression procedure with a continuum manipulator (CM), a bendable drill only needed to be inserted once into the patient. Performing the procedure using a flexible drilling tool may allow the surgeon to effectively remove all and only the necrotic bone tissue. Nevertheless, currently, maneuvering the CM presents challenges to the surgeon user in the operation room. The conventional fluoroscopic navigation information is displayed on an external monitor in the operation room. Thus, the surgeon needs to split attention between the surgical site and the monitor, experiencing increased mental and physical load. In addition, surgeon needs to accurate place and align the CM before insertion, as well as control the CM’s deformation.

Several research groups have proposed tool-mounted navigation systems by mounting a compact display onto a surgical tool holder [9,10]. Particularly, Schutz et al. proposed a guidance visualization technique “Circle Display” for reducing cognitive load during tool navigation and achieving comparable accuracy to conventional navigation method. Besides, other research groups have proposed guidance techniques rendered on head-mounted displayed [11]. Overall, these works focus on guiding the user maneuver a rigid surgical tool and are unable to account for the deformation of a flexible tool.

This project aims to create a navigation software that can guide the user to maneuver the CM. It shall provide guidance in: (1) the CM tip pinpointing location of the penetration site (2) aligning pivot orientation and in-plane rotation of the CM shaft before penetration (3) after the CM entering the patient body, achieving the proper depth of penetration (4) amount and direction of the CM’s deformation during tissue removal. To reduce the physical and mental load enforced onto the surgeon, this project also aims to develop the software for the Microsoft HoloLens 2, a state-of-the-art optical see-through head-mounted display. Such device shall enable the user to receive all necessary navigation information while focusing on the surgical site.

• Minimum: (Expected by March 28th)
  1. A fully functional HoloLens applicaiton that can guide the user to maneuver the CM (completed).
  2. OneDrive folder containing source code and documentation (completed).

• Expected: (Expected by April 27th)
  1. Implement an interactive tutorial for users to learn how to use the application (completed)
  2. A phantom evaluation using a virtual CM model and virtual anatomic structures (completed).
  3. Provide analysis on user's performance to evaluate the accuracy that can be achieved (completed).

• Maximum: (Expected by May 17th)
  1. Implement additional guidance strategy (completed).
  2. Evaluate the additional guidance strategy (not started).

• Interface Diagram:

• Spatial Relationship Diagram:

• Mock handheld surgical drill:

The mock handheld surgical drill was designed to simulate the real handheld CM unit made with low-cost materials.

• Experiemental Setup:

The experiemental setup simulates a real surgical scene.

• Software Interface Diagram:

• Navigation GUI presented to user during drilling procedure:

• Ciscle display: Pivot-orientation and in-plane rotation guidance as proposed by Schutz et al.;
• Guidance paradigm: Based on user's hand maneuver;
• CM visualization: Provide visualization of CM tip locaiton within the anatomic structure.

We designed a two-stage hands-on demo that can showcase our guidance system during the CIS-II poster session. In addition, as an initial evaluation of our guidance system, some data was collected from our internal testing using our system in a controlled experiment setting.

• Training: Prior to using the complete guidance paradigms, users are asked to complete a 7-step tutorial that teaches them how to interact with the experimental setup and each important UI element from the guidance paradigms.

• User testing the navigation GUI during poster session: After the training stage, the navigation GUI is presented for the users to try out on the experimental setup.

• Internal testing results: We performed one trial of the drilling procedure using Paradigm 2 and collected data on pivot-orientation error over time, in-plane rotation error over time, and CM tip trajectory on the cross-sectional anatomic structure image. Overall, more data is required to provide sound justification for evaluating the effectiveness of the proposed guidance paradigms. Due to constraint on project time, data from only one trial was provided. However, data showed that user was able to achieve good accuracy from following the guidance paradigm.

• Using real handheld CM unit;

• Perform procedure on sawbone phantoms;

• In-situ visualization;

• VIsualization of entire CM body;

• User study.

The only outstanding dependency is the marker tracking library using the HoloLens 2 sensors which had not been released by one of the project mentors during the completion of this project. Therefore, we chose to move forward with the contingency plan that utilizes the Polaris Vicra tracking system.

1. Milestone 1: Preliminary Design Review
   • Planned Date: 03/01/2023
   • Expected Date: 03/01/2023
   • Status: Completed
2. Milestone 2: Application deployment and documentation complete
   • Planned Date: 03/27/2023
   • Expected Date: 05/08/2023
   • Status: Completed
3. Milestone 3: Phantom evaluation performed; results obtained
   • Planned Date: 04/13/2023
   • Expected Date: 04/13/2023
   • Status: Completed
4. Milestone 4: Maximum deliverable
   • Planned Date: 04/27/2023
   • Expected Date: 05/04/2023
   • Status: Not started
5. Milestone 5: Final report and presentation complete
   • Planned Date: 05/04/2023
   • Expected Date: 05/08/2023
   • Status: Completed

• Project Plan
  • Project plan presentation
  • Project plan proposal
• Project Background Reading
  • Background Reading Presentation
  • Background Reading Report
• Project Checkpoint
  • Project checkpoint presentation
• Project Poster
  • Project Poster
• Project Documentation Series: (Readers are advised to read the following three documents in the below listed order.)
  • Final Project Report
  • High-level design requirements: Software Requirements Specification (SRS)
  • Low-level design specifications: User Interfaces Specification (UIS)
• Source code and design files: Stored on OneDrive
• Demo video:
  • Video
• Mentor Report

* Weekly 1-on-1 meeting with mentor Alejandro

* Weekly lab meeting

* Source code and documentation storage: OneDrive folder

* Personal Google Drive backup

* Discord, Email

• [1] J. H. Herndon and O. E. Aufranc, “Avascular necrosis of the femoral head in the adult. A review of its incidence in a variety of conditions,” Clin Orthop Relat Res, vol. 86, pp. 43-62, Jul-Aug 1972.

• [2] F. Mwale, H. Wang, A. J. Johnson, M. A. Mont, and J. Antoniou, “Abnormal vascular endothelial growth factor expression in mesenchymal stem cells from both osteonecrotic and osteoarthritic hips,” Bull NYU Hosp Jt Dis, vol. 69 Suppl 1, pp. S56-61, 2011.

• [3] K. J. Bozic, D. Zurakowski, and T. S. Thornhill, “Survivorship analysis of hips treated with core decompression for nontraumatic osteonecrosis of the femoral head,” J Bone Joint Surg Am, vol. 81, pp. 200-9, Feb 1999.

• [4] P. Hernigou, A. Habibi, D. Bachir, and F. Galacteros, “The natural history of asymptomatic osteonecrosis of the femoral head in adults with sickle cell disease,” J Bone Joint Surg Am, vol. 88, pp. 2565-72, Dec 2006.

• [5] F. Alambeigi et al., “A Curved-Drilling Approach in Core Decompression of the Femoral Head Osteonecrosis Using a Continuum Manipulator,” in IEEE Robotics and Automation Letters, vol. 2, no. 3, pp. 1480-1487, July 2017, doi: 10.1109/LRA.2017.2668469

• [6] C. Gao et al., “Fluoroscopic Navigation for a Surgical Robotic System Including a Continuum Manipulator,” in IEEE Transactions on Biomedical Engineering, vol. 69, no. 1, pp. 453-464, Jan. 2022, doi: 10.1109/TBME.2021.3097631.

• [7] J. H. Ma, S. Sefati, R. H. Taylor and M. Armand, “An Active Steering Hand-Held Robotic System for Minimally Invasive Orthopaedic Surgery Using a Continuum Manipulator,” in IEEE Robotics and Automation Letters, vol. 6, no. 2, pp. 1622-1629, April 2021, doi: 10.1109/LRA.2021.3059634.

• [8] A. Martin-Gomez et al., “STTAR: Surgical Tool Tracking using Off-the-Shelf Augmented Reality Head-Mounted Displays,” in IEEE Transactions on Visualization and Computer Graphics, doi: 10.1109/TVCG.2023.3238309.

Appropriate project-related files will be disclosed at the discretion of the project mentors.