Table of Contents

Content Generation for HMD Based Medical Training

Last updated: 2/26/2018 9:20pm

Summary

The goal of our project is to create a smart training module that provides automated feedback for basic medical procedures using the Hololens.

Relevance/Importance:

A recent study done by the BMJ has shown that medical error is the third leading cause of death in the United States, with an estimated 251,000 deaths in 2013. The article cites “inadequate skill” as one of the primary reasons for this medical error.1 Traditional non-interactive instruction has not properly prepared medical and paramedical students to perform life-saving procedures. There is a need for an interactive training system that allows the user to efficiently learn and perfect the medical skill before ever performing on a real patient. This project hopes to address this need, and will be working towards the goal of eliminating the need for a medical professional instructor, which would greatly save resources.

Background:

This project builds upon a previous study on the use of the Microsoft Hololens, in training nursing and university students, staff and faculty on: Needle Chest Decompression and Initiating Direct Line I.V. During the training procedure, the participants in that study were split into two groups. For both tasks one group used the Hololens while the other group didn’t. On the Hololens, users saw text and images telling the user what to do during each step or each procedure.

Deliverables

Technical Approach

Surgical Tool Tracking

The goal of tool tracking is to detect the position and orientation of the tool (in the case of direct line IV, a needle). Previous works have attempted to do this in ultrasound images with optical flow and Hough transform. Our first attempt will be to try to adapt their methods to fit the Hololens: First, detect the marker on the manekin using ARToolkit and limit the scope of the image to within a specified distance of the marker. Then, use canny detection to find edges and hough line transformation to find lines. Next, filter lines based off length (the needle is about 2-3 inches long), so exceptionally long or short lines are likely not the needle. Finally, threshold using HSV color-space (since it is more robust to brightness edge). Preferably, we will eventually add sparse optical flow instead of color-thresholding. This algorithm is rudimentary, so it can be done quickly to get a quick gauge of the Hololens capabilities and is quickly adaptable. If this algorithm ends up being not feasible, we will next try to adapt tool tracking convolutional neural networks used in pose estimation for minimally invasive surgery.

Adaptive Leveling System

The adaptive leveling system will be implemented by using a recommender which uses a score to determine the amount of guidance the user should receive during training. The scoring metric the recommender will be using are time and accuracy. Time can be measured using system time. Accuracy can be determined with the aid of the tool tracking. At specific intervals, the user will check the placement of the tool. The tool tracking software will determine the pose error of the tool based on its goal position and then combine this data with the time of the task to create a cost function for each step.

The recommender will use these scores to determine which level the user is at, and what type of feedback they need. As the user gains more experience there will be less of the overlays and the text so the user does not use the Hololens as a crutch.

Dependencies

Dependency Solution/Backup Date/Progress
Microsoft Hololens Get access to Dr. Huang and/or Dr. Kazanzides labs Completed; 15 February
Code for modules on previous study Get GitHub access to code by Ehsan Completed; 18 February
Surgical Training Mannequins Get access to Dr. Kazanzides’s pod in LCSR Completed; 19 February
Hololens Streaming Application Get GitHub access to code Completed; 23 February
Surgical Training Content Work closely with Allan and Prateek; Already have for first two procedures In progress throughout semester
Ventriculostomy Training Model Order. Backup: use Dr. Molina’s model Completed; 15 March
Structure Sensor for 3D images None Needed Completed; 2 April
iPad None Needed Completed; 2 April

Milestones and Status

  1. Feb 26, 2018: Project Proposal (Completed)
  2. Mar 6, 2018: Live video feed from HoloLens (Completed)
  3. Mar 6, 2018: Interfacing specifications completed (Completed)
  4. Mar 16, 2018: UI added to remote system (Completed)
  5. Apr 6, 2018: 3D overlays imported as static objects in scene (Completed)
  6. Apr 13, 2018: Scoring system integrated (Almost Done) → Waiting on tool tracking
  7. Apr 13, 2018: Software for tip position and planar orientation (Almost done)
  8. Apr 20, 2018: Complete testing for initial tool tracking and overlays
  9. Apr 27, 2018: Difficulty levels completed
  10. May 4, 2018: 3D overlays dynamic objects to provide feedback
  11. May 4, 2018: 3D orientation of tool tracking completed
  12. May 11, 2018: Scoring updated based on new tool tracking
  13. May 11, 2018 Overlays updated based on new tool tracking

Reports and presentations

* Paper Presentation

Project Bibliography

  1. Ayvali, Elif and Jaydev P. Desai. “Optical Flow-Based Tracking of Needles and Needle-Tip Localization Using Circular Hough Transform in Ultrasound Images.” Ann Biomed Eng. 43(8): 1828–1840, 2015
  2. Herron, Jennifer. “Augmented Reality in Medical Education and Training.” Journal of Electronic Resources in Medical Libraries, 13:2, 51-55, 2016.
  3. Pratt, Philip, Mathew Ives, Graham Lawton, Jonathan Simmons, Nasko Radev, Liana Spyropoulou, and Dimitri Amiras. “Through the HoloLens looking glass: augmented reality for extremity reconstruction surgery using 3D vascular models with perforating vessels.” European Radiology Experiment. 2:2, 2018
  4. Qiu, Wu and Mingyue Ding. “Needle Segmentation Using 3D Quick Randomized Hough Transform.” First International Conference on Intelligent Networks and Intelligent Systems
  5. E. Azimi et al., “Evaluation of Optical See-Through Head-Mounted Displays in Training for Critical Care and Trauma,” In Virtual Reality (VR), IEEE, 2018. (accepted)
  6. Wright, T., de Ribaupierre, S., & Eagleson, R. (2017). Design and evaluation of an augmented reality simulator using leap motion. Healthcare Technology Letters, 4-5. DOI: 10.1049/htl.2017.0070
  7. R. Dockter, R. Sweet and T. Kowalewski, “A fast, low-cost, computer vision approach for tracking surgical tools,” 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems, Chicago, IL, 2014, pp. 1984-1989. doi: 10.1109/IROS.2014.6942826
  8. Qian, L., Barthel, A., Johnson, A., Osgood, G., Kazanzides, P., Navab, N., & Fuerst, B. (2017). Comparison of optical see-through head-mounted displays for surgical interventions with object-anchored 2D-display. International Journal of Computer Assisted Radiology and Surgery, 1-10. DOI: 10.1007/s11548-017-1564-y

Other Resources and Project Files