Last updated: 05/10/22

Medical image display is important for diagnosis and surgery. However, current medical image display methods are inappropriate for doctors' observation. The increase of imaging data makes better visualizaion methods necessary.

In this project, we propose a human body anatomic information annotation framework using Augmented Reality technique with ultrasound imaging. We use unity Vuforia engine to track and save anatomic images combined with the pose and position information of the ultrasound probe. And use Microsoft HoloLens for augmented reality display and doctor-image interaction function. With the help of these tools, we can recur history medical images at the same body location at different visit time. With this system, doctors can easily observe the evolution of human body anatomical over time and so to make better diagnosis decision and assist treatment.

• Students: Guanyu Song
• Mentor(s): Alejandro Martin Gomez

Here we show a typical application scene provided by mentor Alejandro Martin Gomez.

Here is the a summary of our work flow.

}}

* Background: As medical images make more and more influences in diagnosis and treatment, there are greater and greater demand for better medical image display. CAMP lab has done excellent work on augmented reality application in medical field. Now, we propose a plan to combine real-time ultrasound imaging and augmented reality display for recurring medical images captured at different time point. It will help improve the experience for doctors to make diagnosis and treatment.

* Relevance: Traditional approaches use printed notes with anatomical atlas to indicate the position of these landmarks. This task can be time-consuming and imprecise. Here we use optical markers and unity Vuforia engines combined with HoloLens for annotation.

* Goal: Design an augmented reality application to annotate anatomical landmarks in medical scenarios that require identifications during different recurring appointments. With this system, doctors can save the coordinates and images whenever needed, and recall these datas when needed.

- Minimum(due on 4/15): Finishing Tracking and Saving function. Provide a working demo capable to track the tool pose and display visual augmentations of their pose with respect to the patient’s body. Provide a git repository containing a readme file (include instructions on how to replicate, the system and its dependencies.) and the source code to replicate the project.

• Save the pose and position of the tool with respect to the patient’s body. Provide c# code uploaded on google docs and github.
• Record images with desired pose coordinates and show the recorded images during future visits. Provide image files with orientation and position information.

- Expected(due on 4/30): Testing on a human body phantom. In addition to the minimum, it is expected to include a 3D scanned replica of human body phantom and a 3D printed ultrasound probe.

• Make 3D print of ultrasound probe model and use 3D scanner to create a virtual replica of the phantom. Provide 3D reconstruction file.
• Evaluate the accuracy of the tracking technique. Provide a report containing the results of the tracking accuracy

- Maximum(5.1-): Adding body tracking function. Automatically identify the patient’s body pose. Visualize the human body anatomy evolution with HoloLens display. Provide source code for the tracking of patient’s body pose.

1. Locate optical markers onto the patient’s body to identify the patient’s pose (alternative method including body tracking is also an option).
2. Save the pose and position as well as images data for recurring.
3. Display the position and history of the anatomical landmarks on top of the patient’s body with augmented reality technique.

• Project Plan
  • project_plan_.pdf| Project plan presentation}}
  • cis_ii_plan_proposal.pdf|Project plan proposal}}
• Project Background Reading
  • See Bibliography below for links.
• Project Checkpoint
  • checkpoint.pdf| Project checkpoint presentation}}
• Paper Seminar Presentations
  • 04-26_backgroud_reading_presentation.pdf
  • link to the paper discussed:https://doi.org/10.1007/s11548-021-02526-7
• Project Final Presentation
  • cis_project_poster.pdf|PDF of Poster}}
• Project Final Report
  • 05-08_cis_ii_final_report.pdf|Final Report}}
  • links to any appendices or other material

* here list references and reading material references:

1. INFOGRAPHIC: SKIN CANCER BODY MOLE MAP: https://www.aad.org/public/diseases/skin-cancer/find/at-risk/mole-map
2. CWRU takes the stage at Microsoft’s Build conference to show how HoloLens can transform learning: https://case.edu/hololens/
3. Nguyen, T., Plishker, W., Matisoff, A. et al. HoloUS: Augmented reality visualization of live ultrasound images using HoloLens for ultrasound-guided procedures. Int J CARS 17, 385–391 (2022). https://doi.org/10.1007/s11548-021-02526-7
4. Matthew G. Hanna, Ishtiaque Ahmed, Jeffrey Nine, Shyam Prajapati, Liron Pantanowitz; Augmented Reality Technology Using Microsoft HoloLens in Anatomic Pathology. Arch Pathol Lab Med 1 May 2018; 142 (5): 638–644. doi: https://doi.org/10.5858/arpa.2017-0189-OA
5. Farahani, N., Post, R., Duboy, J., Ahmed, I., Kolowitz, B. J., Krinchai, T., Monaco, S. E., Fine, J. L., Hartman, D. J., & Pantanowitz, L. (2016). Exploring virtual reality technology and the Oculus Rift for the examination of digital pathology slides. Journal of pathology informatics, 7, 22. https://doi.org/10.4103/2153-3539.181766
6. M. Krichenbauer, G. Yamamoto, T. Taketom, C. Sandor and H. Kato, “Augmented Reality versus Virtual Reality for 3D Object Manipulation,” in IEEE Transactions on Visualization and Computer Graphics, vol. 24, no. 2, pp. 1038-1048, 1 Feb. 2018, doi: 10.1109/TVCG.2017.2658570.
7. Farahani, N., Post, R., Duboy, J., Ahmed, I., Kolowitz, B. J., Krinchai, T., Monaco, S. E., Fine, J. L., Hartman, D. J., & Pantanowitz, L. (2016). Exploring virtual reality technology and the Oculus Rift for the examination of digital pathology slides. Journal of pathology informatics, 7, 22. https://doi.org/10.4103/2153-3539.181766

Here give list of other project files (e.g., source code) associated with the project. If these are online give a link to an appropriate external repository or to uploaded media files under this name space (2022-01).

project git: https://github.com/stefaniesong/Ultrasound-AR-tool-tracking-system

marker generator: https://www.brosvision.com/ar-marker-generator/