This project aims to design and construct a co-robotic autonomous system for acquiring Ultrasound images during a Mammogram.

• Students: Julian Brown, Kevin Wang, Yuxin(Ethan) Chen
• Mentor(s): Dr. Emad Boctor, Dr. Web Stayman, Dr. Russell Taylor, Yixuan Wu

Mammography is an important test for reducing breast cancer mortality, but it is not without its problems. Over 40 million screenings are performed each year and around 6 million of those tests result in the patient being called back to the office for further screening. With only around 350,000 patients diagnosed with breast cancer per year, this means there is a lot of wasted time and money both on the patient's side and doctor's side.

This project aims to add Ultrasound imaging to the Mammogram screening procedure in order to lower the percentage of patients that require further testing. To do this, a robotic system is being designed that will autonomous move across the patient's breasts after the X-Ray images are taken and acquire Ultrasound images of any regions of concern that may be present.

• Minimum: (April 8)
  1. Calibration of the robotic arm.
  2. Setup of the imaging validation and tests.
  3. Simple motion planning to land the ultrasound probe on a specific location on the compression plate. Offline (open loop) control, directing ultrasound probe to lesion from target.

• Expected: (April 22)
  1. A real-time interface to acquire camera and ultrasound images.
  2. With the developed real-time interface, demonstrate automatic ultrasound acquisition of a region of interest (whole lesion) with a known location with respect to the tracking tag on the compression plate.
  3. Documentation

• Maximum: (May 6)
  1. Demonstrate dynamic and real-time adaptation as the tracking tags move.
  2. Force control of the robot arm to ensure safety.
  3. Integration and registration with a second modality (Mammography, CT, or preoperative 3D ultrasound).

Calibration

There are four main parts for calibration, which are UR5 calibration, camera calibration, hand eye calibration and ultrasound probe calibration.

For UR5 calibration, since each UR5 has been calibrated at the factory, the UR5 calibration information has been extracted from the UR5 manipulator. The kinematic chain is generated and used to calculate the forward kinematics and inverse kinematics.

For camera calibration, two point grey chameleon cameras have been calibrated to get rid of the distortion on the image edges. Point grey camera driver is installed and used to connect point grey camera to ROS. Then camera calibrator in ROS is applied to calibrate the cameras by using an 8*6 checkerboard with 25mm squares. Then the rectification matrix and camera matrix are exported and applied through the image_proc ROS package to generate the rectified camera images.

For hand eye calibration, it will be a hand-in-eye problem. AprilTag will be used as the marker. AprilTag ROS package will be used to get the transformation from camera coordinate to the marker coordinate. The UR5 robot will be moved to different positions and look at AprilTag. The transformation from UR5 base coordinate to UR5 end coordinate and the transformation from camera coordinate to AprilTag marker coordinates will be recorded at each position. Then the transformation from UR5 end coordinate to camera coordinate will be solved by establishing an AX=XB problem.

For ultrasound probe calibration, a cross-wire phantom fixed in a water tank will be used. The transformation from UR5 end coordinate to US probe coordinate to ultrasound probe coordinate will be calculated by solving a BXp problem.

After calibration, those transformations will be validated by collecting independent data and plug those transformations back to check their accuracy.

Robot Motion Planning

Regarding robot motion, there are three movements that the robot must perform. First, the robot is moved from any position, to its initial position via hand over hand control. Next, the robot must navigate to a specific lesion site. Finally, the robot must perform a “wobble” motion at the lesion site to acquire volumetric US data. The approach that will be taken to achieve these movements is outlined in the figure below.

In the green box, is the plan to achieve the expected deliverables of motion. First, hand over and control will be integrated into the robot workflow. Next, an offline robot motion planning algorithm for motion to the lesion site and the “wobble” motion will be developed simultaneously as they have no dependency upon each other. When these two prongs are complete, they will be integrated into the expected motion deliverable. In the maximum deliverable outlined in the blue box, these motions will be extended to account for more realistic scenarios; motion planning will incorporate real time adjustments to account for patient movement, and the “wobble” motion will incorporate force sensing to account for realistic US acquisition. The yellow boxes detail the testing protocol that will be used. Expected deliverables will be validated using a ROS virtual simulation then using a physical simulation consisting of a phantom in between acoustic gel coated plates. The maximum deliverable will follow an identical testing protocol, with the addition of a moving phantom and standard TPX plates without the acoustic gel coating.

Integration and registration with a second modality

To integrate and register with a second modality, the first step would be to find some reference features that can be detected in both ultrasound and second modality. Then step two would be to use some segmentation software to extract the feature points from both ultrasound and second modality. Step three would be to use the iterative closest point (ICP) method to find the transformation from the ultrasound image to the image from the second modality.

• Milestone name: Camera Calibration
  • Planned Date: Feb 27
  • Expected Date: Feb 27
  • Status: Completed

• Milestone name: UR5 Calibration
  • Planned Date: Feb 27
  • Expected Date: Feb 27
  • Status: Completed

• Milestone name: Hand Eye Calibration
  • Planned Date: March 4
  • Expected Date: March 4
  • Status: Completed

• Milestone name: Hand over Hand Control Integration
  • Planned Date: March 11
  • Expected Date: March 11
  • Status: Removed

• Milestone name: Offline Motion Planning to Lesion Site
  • Planned Date: April 15
  • Expected Date: April 12
  • Status: Completed

• Milestone name: “Wobble Motion” at Lesion Site
  • Planned Date: April 15
  • Expected Date: April 15
  • Status: Completed

• Milestone name: Ultrasound Probe Calibration
  • Planned Date: April 20
  • Expected Date: April 20
  • Status: Completed

• Milestone name: Adaptive Motion Planning
  • Planned Date: April 29
  • Expected Date: April 29
  • Status: Completed

• Milestone name: Force Sensitive “Wobble”
  • Planned Date: April 29
  • Expected Date: April 29
  • Status: Completed

• Milestone name: US - CT Registration
  • Planned Date: April 29
  • Expected Date: April 29
  • Status: Continue on summer

• Project Plan
  • Plan Presentation
  • Project Proposal
• Project Background Reading
  • See Bibliography below for links.
  • Background Reading Presentation
  • Background Reading Report
  • Background Reading Paper - Phantom with Multiple Active Points for Ultrasound Calibration
• Project Checkpoint
  • Project checkpoint presentation
• Project Final Presentation
  • PDF of Poster
• Project Final Report
  • Final Report

• Wendie, A., Berg, M. D., & Jeffrey, D. (2008). Combined Screening with Ultrasound and mammography compared to mammography alone in women at elevated risk of breast cancer: results of the first-year screen in ACRIN 6666. JAMA, 299(18), 2151-2163.
• Buist, D. S., Porter, P. L., Lehman, C., Taplin, S. H., & White, E. (2004). Factors contributing to mammography failure in women aged 40–49 years. Journal of the National Cancer Institute, 96(19), 1432-1440.
• LaValle, S. M. (2006). Planning algorithms. Cambridge university press.
• Gilboy, K. M., Wu, Y., Wood, B. J., Boctor, E. M., & Taylor, R. H. (2020). Dual-Robotic Ultrasound System for In Vivo Prostate Tomography. In Medical Ultrasound, and Preterm, Perinatal and Paediatric Image Analysis (pp. 161-170). Springer, Cham.
• Aalamifar, F. (2016). Co-robotic ultrasound tomography: a new paradigm for quantitative ultrasound imaging (Doctoral dissertation, Johns Hopkins University).
• Zhang, H. K., Cheng, A., Kim, Y., Ma, Q., Chirikjian, G. S., & Boctor, E. M. (2018). Phantom with multiple active points for ultrasound calibration. Journal of Medical Imaging, 5(4), 045001. PDF

Here give list of other project files (e.g., source code) associated with the project. If these are online give a link to an appropriate external repository or to uploaded media files under this name space (2022-01).

• Github Repository: https://github.com/KevinWang905/Co-Robotic-Ultrasound-Mammography
• Google Drive: https://drive.google.com/drive/folders/1B_MSVh_tHmRteHqT1s-PsmOTkKbvaFqT?usp=sharing