Contact Us
CiiS Lab
Johns Hopkins University
112 Hackerman Hall
3400 N. Charles Street
Baltimore, MD 21218
Directions
Lab Director
Russell Taylor
127 Hackerman Hall
rht@jhu.edu
Last updated: 2/26/2018 9:20pm
The goal of our project is to create a smart training module that provides automated feedback for basic medical procedures using the Hololens.
A recent study done by the BMJ has shown that medical error is the third leading cause of death in the United States, with an estimated 251,000 deaths in 2013. The article cites “inadequate skill” as one of the primary reasons for this medical error.1 Traditional non-interactive instruction has not properly prepared medical and paramedical students to perform life-saving procedures. There is a need for an interactive training system that allows the user to efficiently learn and perfect the medical skill before ever performing on a real patient. This project hopes to address this need, and will be working towards the goal of eliminating the need for a medical professional instructor, which would greatly save resources.
This project builds upon a previous study on the use of the Microsoft Hololens, in training nursing and university students, staff and faculty on: Needle Chest Decompression and Initiating Direct Line I.V. During the training procedure, the participants in that study were split into two groups. For both tasks one group used the Hololens while the other group didn’t. On the Hololens, users saw text and images telling the user what to do during each step or each procedure.
The goal of tool tracking is to detect the position and orientation of the tool (in the case of direct line IV, a needle). Previous works have attempted to do this in ultrasound images with optical flow and Hough transform. Our first attempt will be to try to adapt their methods to fit the Hololens: First, detect the marker on the manekin using ARToolkit and limit the scope of the image to within a specified distance of the marker. Then, use canny detection to find edges and hough line transformation to find lines. Next, filter lines based off length (the needle is about 2-3 inches long), so exceptionally long or short lines are likely not the needle. Finally, threshold using HSV color-space (since it is more robust to brightness edge). Preferably, we will eventually add sparse optical flow instead of color-thresholding. This algorithm is rudimentary, so it can be done quickly to get a quick gauge of the Hololens capabilities and is quickly adaptable. If this algorithm ends up being not feasible, we will next try to adapt tool tracking convolutional neural networks used in pose estimation for minimally invasive surgery.
The adaptive leveling system will be implemented by using a recommender which uses a score to determine the amount of guidance the user should receive during training. The scoring metric the recommender will be using are time and accuracy. Time can be measured using system time. Accuracy can be determined with the aid of the tool tracking. At specific intervals, the user will check the placement of the tool. The tool tracking software will determine the pose error of the tool based on its goal position and then combine this data with the time of the task to create a cost function for each step.
The recommender will use these scores to determine which level the user is at, and what type of feedback they need. As the user gains more experience there will be less of the overlays and the text so the user does not use the Hololens as a crutch.
| Dependency | Solution/Backup | Date/Progress |
|---|---|---|
| Microsoft Hololens | Get access to Dr. Huang and/or Dr. Kazanzides labs | Completed; 15 February |
| Code for modules on previous study | Get GitHub access to code by Ehsan | Completed; 18 February |
| Surgical Training Mannequins | Get access to Dr. Kazanzides’s pod in LCSR | Completed; 19 February |
| Hololens Streaming Application | Get GitHub access to code | Completed; 23 February |
| Surgical Training Content | Work closely with Allan and Prateek; Already have for first two procedures | In progress throughout semester |
| Ventriculostomy Training Model | Order. Backup: use Dr. Molina’s model | Completed; 15 March |
| Structure Sensor for 3D images | None Needed | Completed; 2 April |
| iPad | None Needed | Completed; 2 April |
* Paper Presentation
