We are creating a virtual reality environment with patient models and an interactable C-arm for recreating internal fixation of pelvic fractures. To achieve this, DeepDRR provides a fast and realistic simulation of dynamic digital radiography(DDR) from CT scans shown in the Head-Mounted Display.

• Students: Han Zhang, Zixuan Liu, Liam Wang
• Mentor(s): Benjamin Killeen,Mathias Unberath

To identify an accurate injury history of pelvic injuries, a compressive manual stress examination usually be applied to the patient, which will cause severe pain in the pelvic area with any attempted movements. Although the percutaneous pelvic fracture surgery have shown multiple prestigious advantageous than traditional open surgery, it has inadequate visualization because of the minimal invasive surgery. (Mosheiff & Routt. 2020)

(Unberath at el. 2018) proposed a framework, DeepDRR, that can generate realistic DRRs from 3D CT with more accurate anatomical landmark detection and localization of robot end-effectors than conventional naive DRRs.

By using a virtual C-arm model in the VR environment to train residents who had few experience observing C-arm procedures, they showed an overall significant improvement in C-Arm placement with regards to angular accuracy (mean ~2 degree improvement), and total procedure time (mean 11 minutes less time) for interventional spine procedures. (Allen et al. 2022)

We intend to develop a virtual reality (VR) training environment for percutaneous pelvic fracture surgery to overcome the challenges of traditional C-arm fluoroscopy training, including ionizing radiation exposure and the complexity of human anatomy. The VR simulation will allow clinicians to train without radiation exposure, collect detailed operation data, and test new surgical techniques or protocols. The proposed project consists of three goals, including developing a foundation for a realistic and immersive training environment, integrating interactive C-arm and surgical tools, and developing a sophisticated simulation of K-wire and screw insertion interactions. The solution aims to reduce training costs, enhance patient outcomes, and increase accessibility to C-arm fluoroscopy training, providing orthopedic surgeons with a safe and effective learning platform.

• Minimum: (Expected by 02/28)
  1. Annotated CT data that includes landmarks and K-wire paths.[Completed]
  2. Project proposal document.[Completed]
  3. A documented and unit-tested python DeepDRR server package and Unity project that can display live X-ray images from a CT model corresponding to different angles of the camera.[Completed]
  4. Static Unity prefabs containing C-arm and tool models.[Completed]
  5. A Unity prefab that can be imported and is capable of reporting the live K-wire position and orientation in the simulated X-ray images, along with corresponding documentation and unit tests.[Completed]

• Expected: (Expected by 03/23)
  1. Documented MonoBehavior script components that allow the user to adjust the C-Arm model in VR. A set of manual test procedures for ensuring working VR C-Arm movement interactions.[Completed]
  2. Documented MonoBehavior script components that allow the user to operate the tools smoothly and realistically. A set of manual test procedures for ensuring working VR interactions. Documentation for VR interaction interfaces.[Completed]
  3. A documented guide to extending our simulation environment for future C-arm devices, use-cases, and techniques.[Completed]

• Maximum: (Expected by 05/05)
  1. Documented MonoBehavior script components in the project that can restrict the movement of the K-wire insertion in different stages. A series of test cases to verify the behavior functions as defined.[Completed]
  2. Documented MonoBehaviour script components that provide VR controller vibration haptics to help the user recognize and adjust their movements during the K-wire insertion process.[Completed]
  3. GUI for the user headset. Documentation and instructions for using the GUI.[Completed]
  4. A paper analyzing the effectiveness of virtual pelvic surgery training. [Incomplete, Expect: Summer]
  5. A dataset of surgical training procedures.[Incomplete, Expect: Summer]
  6. A Unity-based multiplayer environment that the mentor can provide real-time guidance to user on the surgical simulations.[Incomplete, Expect: Summer]

Current Demo Video

We will use the Unity (https://unity.com/) game engine to develop our virtual reality environment to simulate the surgical procedure. We plan to distinguish our virtual surgical environment from previous work by focusing on more robust and realistic VR interactivity using the hand controllers. For this reason, Unity was chosen for our development platform rather than SlicerVR or AMBF, as we determined that Unity has the best support for easily creating complex VR interactions and user interfaces.

Since we are using Unity OpenXR and XRTool Kit, our virtual environment will be compatible with any VR headset as long as it follows the protocol of OpenXR and XRTool Kit. For development, we will use the HTC Vive Pro and Meta Quest 2 headset with the corresponding controllers.

We will use DeepDRR to provide surgeons with realistic simulated x-ray images from the virtual C-arm, which can be placed at an arbitrary position and orientation relative to the virtual patient. To generate the radiographs, DeepDRR requires a 3D CT volume and information about the relative location/intrinsic camera properties of the fluoroscopy device. Additionally, the locations and density volumes of any surgical tools which need to be included in the simulated radiographs must be provided. Given this information, the DeepDRR program can output a simulated radiograph.

DeepDRR is a Python package for Linux computers with an NVIDIA CUDA-capable GPU with >11 GB of memory. To connect the DeepDRR program to our virtual reality simulated environment Unity program, we will implement a socket-based DeepDRR server to allow the Unity program to request radiographs from DeepDRR in real-time. The DeepDRR server will have the flexibility to run on the same computer as the Unity VR application, or on a separate computer connected via Ethernet or WiFi to the computer running the Unity application. The Unity VR application will run on a computer running Windows, so the DeepDRR server will have the capability to run inside a Windows Subsystem for Linux virtual machine or Linux Docker Container on the same computer.

We will use the ZeroMQ networking library (https://zeromq.org/) for socket communication over TCP due to its robust pub-sub messaging patterns that will allow the server to be easily extended to serve multiple clients in the future. We will define the network packet serialization data structures using the Cap’n Proto data interchange format (https://capnproto.org/) which can automatically generate both C# (Unity) and Python code to serialize/deserialize data structures from binary buffers.

When the user requests a radiograph in the Unity application, the Unity app will send a data structure to the DeepDRR server with information about the position and orientation of the C-arm, patient, and surgical tools, as well as C-arm camera intrinsics. The DeepDRR server will process the request and reply with a DRR image which the Unity application will decode and display on a virtual monitor next to the patient.

Pelvic fractures internal fixation tool models

The pelvic fractures internal fixation tool models consist of K-wire sets, surgical screw sets, surgical drills, and surgical driver kits. The K-wire sets come in two lengths (300mm and 450mm) and a diameter of 2.8mm, which is commonly used for percutaneous fixation of pelvic fractures. The surgical screw sets come in a diameter of 6.5mm, thread lengths of 12mm and 32mm, and their length ranging from 30mm to 130mm. These screw sets are used for providing internal fixation after inserting the k-wire. The k-wire and screws have been modeled with proper thread patterns. The surgical drill and driver kits are designed to be used and fit with both the K-wire sets and surgical screw sets precisely. The surgical drill model is designed by 3DSMAX and the rest are designed by Solidworks with high accuracy and real scale to provide an immersive experience for trainees. C-arm Model and Kinematics

To generate a simulated X-ray from DeepDRR and to track the rotation and translation of K-wires, we use homogeneous frame transformations to calculate the kinematics between the Unity room and DeepDRR world, as shown in the accompanying image. This process enables us to project the simulated X-ray images accurately and place the virtual K-wires in the correct location, orientation, and depth within the pelvic anatomy. By incorporating this technique, we can provide trainees with a more realistic and immersive training environment that accurately simulates the surgical procedure.

K-wire simulation

We will implement a basic Kirschner-wire (k-wire) interaction simulation with different types of tissue to make the wire insertion process more intuitive for surgeons who are familiar with the real procedure. At a minimum, the simulation must be sufficiently realistic such that surgeons feel that their performance in the virtual environment is at least somewhat representative of their skills in real procedures. After discussion with our advisors, we believe that while the experience of actually inserting k-wires will be quite different in our simulation compared to the real world (e.g. no force feedback/less accurate tissue interaction), the skills of adjusting the C-arm and spatial placement of k-wires based off of the x-ray images should translate well to our simulation. As a maximum deliverable, we plan to investigate more sophisticated k-wire/tissue interaction simulation methods if time allows.

Our system's user interface has undergone several modifications and optimizations, and it now consists of three main parts. The first part is a large screen located at the center of the operating room that primarily displays a user guide, informing users about the functionalities of the various buttons on the VR controller. The second part is a large screen situated in front of the operating table, which is primarily used to display real-time digital radiography (DRR) images. If the DRR images have not yet been fully loaded, this screen will appear red. The third part (shown in Figure 2) is a control menu located on the left side of the operating table, which is primarily used to manage the interactive features related to surgical operations.

Our multiplayer system uses the same ZMQ connection as the other networked modules. When a surgical tool is moved locally by a player, the Unity client publishes a stream of timestamped pose updates to the ZMQ proxy. Each Unity client constantly listens for packets containing timestamped pose updates for objects in the surgical simulation. When a packet is received, the client updates the position of the corresponding object in the local scene. To smooth out position updates, the Unity client attempts to show the position of each object at a small fixed time offset in the past, (e.g. 200 milliseconds) and uses linear interpolation or extrapolation to blend between the last received updates. The multiplayer function has not yet been completed, and we will continue to work on it this summer. The below figure is a multiplayer prototype at the current stage.

The figure presented herein delineates the dependencies for our project. All requisite dependencies have been procured in accordance with initial expectations. Furthermore, we have submitted an Institutional Review Board (IRB) application, with the anticipation of approval prior to May 30th. This will facilitate the execution of the user study, which is slated for commencement during the summer months.

Relative to the checkpoint status, we have revised the project milestones. All initial minimum, expected, and maximum deliverables have been successfully accomplished. At present, our focus has shifted to achieving objectives that surpass the original maximum expectations: the user study, the implementation of multiplayer functionality, and the collection of data from the user study. We anticipate the completion of these additional deliverables over the course of the summer months.

• Project Plan
  • Project plan presentation
  • Project plan proposal
• Project Background Reading
  • Background review report
  • Background Presentation
    • Paper 1: Allen et al. Development and evaluation of an open source virtual reality C Arm simulator
    • Paper 2: Unberath et al. DeepDRR – A Catalyst for Machine Learning in Fluoroscopy-guided Procedures
    • Paper 3: Brazil et al. Development of Unity Simulator for Epidural Insertion Training for Replacing Current Lumbar Puncture Simulators
• Project Checkpoint
  • Project checkpoint presentation
• Paper Seminar Presentations
  • Background review report
  • Background Presentation
• Project Final Poster and Teaser
  • Poster Teaser
  • Poster
• Project Final Report
  • Final Report
• Project Mentor Report
  • Mentor Report
• Documentation
  • Software Requirement Specifications
  • Interface Specification
  • API Documentation

1.Unberath, M., Zaech, J.-N., Lee, S. C., Bier, B., Fotouhi, J., Armand, M., & Navab, N. (2018). DeepDRR – A Catalyst for Machine Learning in Fluoroscopy-guided Procedures. arXiv. https://doi.org/10.48550/ARXIV.1803.08606

2.Daniel R. Allen, Collin Clarke, Terry M. Peters & Elvis C.S Chen (2022) Development and evaluation of an open-source virtual reality C-Arm simulator, Computer Methods in Biomechanics and Biomedical Engineering: Imaging & Visualization, DOI: 10.1080/21681163.2022.2152374

3.Rommens, P. M., Graafen, M., Arand, C., Mehling, I., Hofmann, A., & Wagner, D. (2020). Minimal-invasive stabilization of anterior pelvic ring fractures with retrograde transpubic screws. Injury, 51(2), 340-346. Würfl, T., Hoffmann, M., Christlein, V., Breininger, K., Huang, Y., Unberath, M., & Maier, A. K. (2018). Deep learning computed tomography: Learning projection-domain weights from image domain in limited angle problems. IEEE transactions on medical imaging, 37(6), 1454-1463.

4.Rami Mosheiff, Chip Routt.Percutaneous fixation of pelvic fractures. orthoinfo - aaos. OrthoInfo. (n.d.). Retrieved February 8, 2023, from https://orthoinfo.aaos.org/en/treatment/internal-fixation-for-fractures

5.Wang, Z.-h. and Li, K.-n. (2019), Regional Injury Classification and Treatment of Open Pelvic Fractures. Orthop Surg, 11: 1064-1071. https://doi.org/10.1111/os.12554

6.M. Unberath et al., “DeepDRR – A Catalyst for Machine Learning in Fluoroscopy-guided Procedures,” arXiv.org, https://arxiv.org/abs/1803.08606v1 (accessed Mar. 13, 2023).

7.A. L. Brazil, A. Conci, E. Clua, L. K. Bittencourt, L. B. Baruque, and N. da Silva Conci, “Haptic forces and gamification on epidural anesthesia skill gain,” Entertainment Computing, vol. 25, pp. 1–13, Mar. 2018, doi: 10.1016/j.entcom.2017.10.002.

8.J. Moo-Young, T. M. Weber, B. Kapralos, A. Quevedo, and F. Alam, “Development of Unity Simulator for Epidural Insertion Training for Replacing Current Lumbar Puncture Simulators,” Cureus, vol. 13, no. 2, p. e13409, doi: 10.7759/cureus.13409.

• Github Organization: https://github.com/PelvisVR
• Client-side repository: https://git.lcsr.jhu.edu/pelvisvr/vr_surgical_room
• Server-side repository: https://github.com/PelvisVR/deepdrr_zmq
• API Documentation: https://pelvisvr.github.io/deepdrr_zmq/deepdrrzmq.html