Last updated: 2023/5/15

This project seeks to implement a well-documented package that can be easily used to perform 2D/3D fluoroscopic image registration tasks.

• Students: Jiaming “Jeremy” Zhang, Zhangcong She
• Mentor(s): Benjamin Killeen, Prof. Mathias Unberath

In minimally invasive surgery, clinicians use intraoperative fluoroscopy to overcome the occlusion and ascertain the poses of anatomy, surgical instruments, or artificial implants.

2D/3D registration is the process that estimates the pose of the 3D objects, such as the CT, based on 2D images, such as the X-ray. The advantage of 2D/3D registration is that it doesn’t require fiducials.

To perform 2D/3D registration, the coordinates of the landmarks have to be determined in 2D x-ray images and 3D CT scans. These two steps are done by two well-developed packages, namely SyntheX and xReg. SyntheX uses Trans-Unet to detect the 2D landmarks in pixel coordinates. xReg takes the intensity information and coordinates of both 2D and 3D landmarks as inputs and computes the projection matrix accordingly. The projection matrix is computed based on the following optimization problem.

The following figure demonstrates the typical 2D/3D registration scenario.

Our project aims to develop a pipeline that automatically performs the 2D/3D registration process between X-ray images and CT scans during operation. To be specific, we need to integrate the image acquisition process, data synthesizing process, landmark detection, and online registration into one sole software with a user-friendly GUI. In addition, we also seek to visualize the data with novel projective paradigms on HoloLens.

More importantly, we want to increase the interpretability and readability of our code so that other developers can add more features in a simple way by following the development guidance we provide.

1. Different packages don’t have proper ways to communicate. Transferring data is a big headache.
2. Sometimes are developed under different environments and not compatible with each other.
3. The packages are poorly documented and therefore hard to maintain and expand
4. A fully integrated, user-friendly and automatic pipeline is critical for clinical application

• Minimum:
  1. Documentation for SyntheX, provide applicable interfaces
  2. A script for Automatic Data Acquisition
  3. A well-documented program integrating previous works

• Expected:
  1. A fully automatic pipeline integrating all components
  2. A view-rendering application for projective visualization
  3. A report for Validating our application on cadaveric images
• Maximum:
  1. Integration with mixed reality visualization of relevant anatomy

https://github.com/shez12/xregi

Here is the Documentation for our code implementation readme.pdf

Other important documentation can be found on Github:

1. Collaboration Specification https://github.com/jeremyzz830/CISII-Repo/blob/main/Docs/collaboration_documentation.md
2. Data Specification https://github.com/jeremyzz830/CISII-Repo/blob/main/Docs/data_specification.md
3. SyntheX Tutorial https://github.com/jeremyzz830/CISII-Repo/blob/main/Docs/synthex_tutorial.md

1. Initially, we would implement a script to retrieve the X-ray image from Loop-X to local device in the real-time.
2. Then, the landmark detection of X-ray image would be done by applying SyntheX.
3. While, in preoperative period, we would apply Total segmentator to perform segmentation and manually annotate the CT scan.
4. Finally, processed X-ray image and CT scan would be used as input of xReg(Regi2D3D) to get transformation matrix from CT scan frame to X-ray frame.

We have completed several submodules in our project, which were designed based on the program architecture shown above.

2D Landmarks Detection

• SyntheX provides a well-trained Trans-Unet model to automatically detect landmarks on X-ray images.

Annotation for Given CTs

• Segmentation is done automatically using TotalSegmentator.
• Professional personnel have manually annotated 3D landmarks, and our mentor has provided the annotation results.

Solving Registration Problem

• Loading X-ray images, CT scans, and their corresponding landmarks.
• Using xReg to solve the registration problem.

Because SyntheX and Xreg were developed by different authors, the output of SyntheX cannot be directly used by Xreg, and the data retrieved from Loop-X may not be in a format that is readable by SyntheX. To address this, we are using a Python-based data exchange panel to transfer data into the required format.

• Read and write HDF5
• Read X-ray DICOM Image
• Read 2D landmarks in Pandas Dataframe

In our experiments on pelvic registration, we utilized sequential X-ray images and high-lighted computed landmarks on one example image (as shown in Figure below). Overall, the landmark detection process worked well, with most of the landmarks being located correctly. In most cases, through manual inspection, the total number of detected landmarks is slightly less than the total number of visible landmarks. We have tried several threshold values until the inference result is satisfactory. Figure below shows how the number of detected landmarks varies along with the threshold values.

Figure below shows the result of the registration solver of xReg. The intensity features are projected onto the X-ray images to help the user verify whether the registration result is sufficiently correct. The figure proves that our workflow performs well under the experiment setups. To obtain correct results, we have tried different combinations of the key parameters used in the optimization process. Besides, we have found out that xReg only takes L./R.IOF, L./R.ASIS, and L./R.FH as its optimizing targets. Among the key parameters involved, the extrinsic parameter of the camera affects the result most significantly. Bad assignment of the extrinsics could lead to a poor initialization for the bound of the searching area, which eventually causes failure or completely wrong results. Again, as our objective is to provide an integration of xReg, the quality of the output of the package will not be discussed here. Nevertheless, with the package implemented, the performance of xReg and other potential methods can be compared and evaluated by a controlled dataset with known ground-truth.

Once the registration problem has been computed, we utilize the estimated registration pose to render the pelvis with respect to the projective coordinate frame. The result of this process is shown in the figure below. The successful rendering of the pelvis with respect to the projective coordinate frame demonstrates the versatility of our approach.

In conclusion, our key contributions in this project includes:

1. We have successfully built a new unified and modularized architecture in Python that provides developer interfaces for future modularization and development.
2. By integrating SyntheX and xReg into our program, we have demonstrated our program's compatibility and effectiveness, and have highlighted the potential for its use in a wide range of medical imaging applications.
3. The package can be easily installed and utilized with simple terminal commands, making it accessible to users of different levels.

To ensure that users can effectively leverage the package to accelerate their research, we have created detailed documentation for both SyntheX and XREGI. This documentation provides users with the necessary resources to effectively use the package.

In the future, we will keep maintaining XREGI and make it more versatile. We seek to incorporating our implementation with robotics medical imaging platforms to realize real-time image acquisition and registration. Moreover, we plan to employ the registration result derived by XREGI in mixed reality area, such as the use of HoloLens, to facilitate its clinical applications in intraoperative processes. Additionally, we plan to submit our work to a peer-reviewed conference or journal to further validate the effectiveness and potential of our package.

1. Milestone name: Documented and well-organized repository for SyntheX
   • Planned Date: Mar 6
   • Expected Date: Mar 11
   • Status: Complete
2. Milestone name: PyPi package
   • Planned Date: Mar 20
   • Expected Date: Apr 11
   • Status: In progress
3. Milestone name: Well-designed, documented, and automatic pipeline
   • Planned Date: Apr 3
   • Expected Date: Apr 16
   • Status: complete
4. Milestone name: Well-analyzed report about validating program
   • Planned Date: Apr 15
   • Expected Date: Apr 25
   • Status: complete
5. Milestone name: Simulation app executable in HMD
   • Planned Date: May 15
   • Expected Date: May 20
   • Status: hold

• Project Plan
  • presentation_plan.pptx
  • cis2_project_proposal.pdf
• Project Background Reading
  • See the Bibliography below for links.
  • background_presentation.pdf
  • cis2_background_report.pdf
• Project Checkpoint
  • checkpoint_presentation.pdf
• final checkpoint
  • team07_final_checkpoint.pptx
• Project Final Presentation
  • poster.pdf
• Project Final Report
  • final report
  • mentor report

1. Grupp, Robert \& Hegeman, Rachel \& Murphy, Ryan \& Alexander, Clayton \& Otake, Yoshito \& McArthur, Benjamin \& Taylor, Russell. (2019). Pose Estimation of Periacetabular Osteotomy Fragments with Intraoperative X-Ray Navigation.
2. Grupp, R.B., Unberath, M., Gao, C. et al. Automatic annotation of hip anatomy in fluoroscopy for robust and efficient 2D/3D registration. Int J CARS 15, 759–769 (2020). https://doi.org/10.1007/s11548-020-02162-7
3. R. B. Grupp et al., “Pose Estimation of Periacetabular Osteotomy Fragments With Intraoperative X-Ray Navigation,” in IEEE Transactions on Biomedical Engineering, vol. 67, no. 2, pp. 441-452, Feb. 2020, doi: 10.1109/TBME.2019.2915165.
4. C. Gao et al., “Fiducial-Free 2D/3D Registration for Robot-Assisted Femoroplasty,” in IEEE Transactions on Medical Robotics and Bionics, vol. 2, no. 3, pp. 437-446, Aug. 2020, doi: 10.1109/TMRB.2020.3012460.
5. Gao C, Farvardin A, Grupp RB, Bakhtiarinejad M, Ma L, Thies M, Unberath M, Taylor RH, Armand M. Fiducial-Free 2D/3D Registration for Robot-Assisted Femoroplasty. IEEE Trans Med Robot Bionics. 2020 Aug;2(3):437-446. doi: 10.1109/tmrb.2020.3012460. Epub 2020 Jul 28. PMID: 33763632; PMCID: PMC7982989.

1. Gao, C., “SyntheX: Scaling Up Learning-based X-ray Image Analysis Through In Silico Experiments”, arXiv e-prints, 2022. doi:10.48550/arXiv.2206.06127.
2. Arcadelab, “Arcadelab/synthex,” GitHub. [Online]. Available: https://github.com/arcadelab/SyntheX. [Accessed: 21-Feb-2023].
3. C. Gao, “Fluoroscopic navigation for robot-assisted orthopedic surgery,” dissertation, 2022.
4. P. Markelj, D. Tomaževič, B. Likar, and F. Pernuš, “A review of 3D/2D registration methods for image-guided interventions,” Medical Image Analysis, vol. 16, no. 3, pp. 642–661, 2012.
5. R. B. Grupp, M. Unberath, C. Gao, R. A. Hegeman, R. J. Murphy, C. P. Alexander, Y. Otake, B. A. McArthur, M. Armand, and R. H. Taylor, “Automatic annotation of hip anatomy in fluoroscopy for robust and efficient 2D/3D registration,” International Journal of Computer Assisted Radiology and Surgery, vol. 15, no. 5, pp. 759–769, 2020.
6. Y. Otake, M. Armand, R. S. Armiger, M. D. Kutzer, E. Basafa, P. Kazanzides, and R. H. Taylor, “Intraoperative image-based multiview 2D/3D registration for image-guided orthopaedic surgery: Incorporation of fiducial-based C-arm tracking and GPU-acceleration,” IEEE Transactions on Medical Imaging, vol. 31, no. 4, pp. 948–962, 2012.
7. rg2/xreg,GitHub.[Online]. Available: https://github.com/rg2/xreg
8. Y. Otakeet al., “Robust patella motion tracking using intensity-based 2D-3D registration on dynamic bi-plane fluoroscopy: Toward quantitative assessment in MPFL reconstruction surgery,”Proc. SPIE, vol. 9786, 2016,Art. no. 97 860B

Here give list of other project files (e.g., source code) associated with the project. If these are online give a link to an appropriate external repository or to uploaded media files under this name space (2023-07).