Trace: • project-18

Dual-robotic arm prostate ultrasound tomography

Last updated: 5/10/2023 and 15：34

Summary

Students: Yunpu Zhang, Zhenghao Li, Ziyi Wang
Mentor(s): Yixuan Wu, Dr. Mohammad Salehizadeh, Dr. Russ Taylor, Dr. Emad Boctor

We will use a dual robotic system with aligned abdominal and TRUS probe for prostate cancer detection. What we need to do is to achieve perfect alignment of the dual-robotic arms to ensure the needs of ultrasound tomography. Moreover, we will make the dual robotic arm safer when moving. In detail, We will determine each transformation of the dual robotic arm system through hand-eye calibration and base-to-base calibration, and then do a motion planning for the robotic arm. After that, we will manually guide the abdominal probe to the correct initial position by applying the hand-over-hand control, then we will constrain the force, velocity and acceleration for the robots, as well as a virtual fixture to limit the workspace of it, to avoid the collision between two robotic arms. Finally, we will implement a real-time demo to show the automaticity and safety of the framework.

Background, Specific Aims, and Significance

The detection of prostate cancer is an important mission for physicians, as it is the second most common non-skin cancer and the fifth leading cause of cancer death in men worldwide in 2018. However, many detection methods have drawbacks. PSA, or prostate-specific antigen, which is measured through a blood test, can be helpful in detecting prostate cancer, but it is not a perfect test, and there can be false positives or false negatives; multi-parametric prostate MRI is a useful detection method, but the equipment is expensive so the penetration rate is insufficient. So we need another method that provides practical and effective detection results, and Ultrasound computed tomography (USCT) technology can be a choice.

USCT has several advantages over traditional ultrasound tomography. It is real-time, non-invasive and the images are of high resolution. Also, USCT provides quantitative ultrasound transmission: like the sound velocity and sound attenuation, and the quantitative data could help to detect the stage of cancer.

However, the physiological structure of the prostate is deep, so it is difficult to obtain high-resolution images by traditional methods. Therefore, we use the dual-robotic system, the transrectal ultrasound (TRUS) probe extends into the human body and the probe is very close to the Prostate, receiving the ultrasound inside the patient's body. It can reduce the acoustic impedance mismatch and could get high-quality images.

Deliverables

Minimum: (02/10 - 04/20)
1. A demonstration of proper dual-robotic setup and equipment connectivity. (Completed)
2. Robot base-to-base registration and end-effector-to-ultrasound calibration. (Completed)
3. Real-time motion planning of dual arms for ultrasound image acquisition where two ultrasound transducers are automatically aligned. (Completed)

Expected: (02/25 - 04/30)
1. Integration of virtual fixture to limit robot arms’ workspace to ensure the safety for in vivo imaging (for Abdominal probe) (Completed)
2. Enhanced path planning to achieve smoother and steadier motion. (Completed)

Maximum: (05/01 - 05/08)
1. A real-time demo on a speed of sound phantom in preparation for the validation of ultrasound tomography reconstruction methods. (Completed)
2. A real-time demo on a realistic pelvis phantom to show the automaticity and safety of the framework (Completed)
3. Integration of virtual fixture for the TRUS probe (In Progress)

Technical Approach

Robot Arm Calibration & Motion Planning

All the variables we used in this section are defined below.

The system is shown in Figure below:

In the calibration part, we could obtain BT and BA from forward kinematics. FA is the rotational transformation of the robot end-effector to the force sensor, which is also a known SE(3) matrix. The first goal here is to apply BXp calibration to obtain XT and XA respectively: we would solve a BXp problem for each UR5 arm. After that, we would use Point Cloud Registration to obtain the transformation matrix between the two bases: X_TA

The BXp problem could be solved with

In the motion planning part, we would use the position of the TRUS probe to calculate the velocity that the Abdominal probe should have to accomplish the desired motion. The velocity is defined as:

V_pose and V_contact are defined in figure below.

For brevity, we would not expand the calculation for v_pose and v_contact in this short technical summary of approaches, but the calculation for them could be found in [4].

Hand Guidance & Virtual Fixture & Smooth Motion

Hand Guidance is to move the robot to correct initial position with human hand, more information for this could be found in [4]. Virtual fixture should be a familiar concept; for smooth motion, it could also be defined as adding constraints to the robot, like maximum velocity, acceleration, etc.

The virtual fixture could be implemented with a number of different approaches. To limit the work space, we would like to accomplish this with RVIZ and MoveIt. Basically, we will have the robot in those simulation program imitating the behavior of the real robot. Then, we could add mesh files to illustrate obstacles or walls in the real world to the simulation. We would also implement a function to utilize our representation in the simulated world to determine if there is an obstacle in the path of a robot motion. Before trying to move the robot to any new position, the function would find out if the path is clean. The robot should only move when there is a clean path.

Another method of virtual fixture would be adding geometry constraints to the robot based on optimized constrained control. At this early stage, we are mostly considering about keeping the tip of the abdominal probe in a line. More specifically, when we do motion planning, we would move the probe from node to node. When the robot is executing the movement, we want its tip of the abdominal probe to stay in a line.[8]. From this paper, if we have a line L(s) = L0+l*s, where L0 is a point on the line and l is a unit vector indicating the direction, the system needs to correct the offset from the current position to the closest point on the line. To accomplish this goal, we first need a rotation matrix transforming the line to the world coordinate.

The goal for the maximum part is to have an automatic image acquisition system. We would first use our dual robotic motion planning program to help validate an ultrasound image reconstruction algorithm from our mentor Yixuan. Then, we need to combine our program with the reconstruction algorithm to perform a demo on realistic phantoms to show the automaticity and safety of the framework. The phantom could be found here [7]. To do the virtual fixture of the TRUS probe, we would need to find out is the probe at correct orientation based on the image it obtains could apply computer vision techniques to resolve the information and correct the position of the TRUS probe.

where l' is a random unit vector not aligned with l.

Then, if we approximate a cylinder around the line with radius epsilon by a polygon with n vertices centered at origin, and we have:

Finally, we could set the H and h as follow. Again, more information could be found in [8]:

The idea of virtual fixture could be visualized in Figure below. We would limit the workspace by virtual walls, as well as moving the tip of the abdominal probe in a line. (green dashed line in Figure) The constraints are mainly three types in our plan: force, velocity, and acceleration, but before any constraints, we need to know the value for those properties at any time. The force part would come from the force measured from sensor. The acceleration part is also quite straightforward: the derivative of the velocity. The velocity part is calculated in section above.

After having those values, we could add constraints on them. For instance, we could limit the maximum acceleration; we could also set the speed inversely proportional to the distance from the abdominal probe to the phantom. Also, the translation part of the UTA ∈ SE3 is not a constant value in expected, we need to calculate the speed and the location of the abdominal probe at any time considering the force, velocity and acceleration contacted with the phantom.

Automatic Image Acquisition System

To do the virtual fixture of the TRUS probe, we would need to find out is the probe at correct orientation based on the image it obtain. We could apply computer vision techniques to resolve the information and correct the position of the TRUS probe.

Dependencies

Milestones and Status

Milestone name: Literature Review
- Planned Date: 02/10
- Expected Date: 02/17
- Status: Done
Milestone name: Familiar with CISST/SAW libraries and UR5 robot
- Planned Date: 02/16
- Expected Date: 02/23
- Status: Done
Milestone name: Install Ubuntu 16.04 and ROS1 on lab laptop
- Planned Date: 02/16
- Expected Date: 02/17
- Status: Done
Milestone name: Understand Previous Code
- Planned Date: 02/16
- Expected Date: 03/09
- Status: Done
Milestone name: Read Virtual Fixture and Path Planning literature
- Planned Date: 02/25
- Expected Date: 04/09
- Status: Done
Milestone name: Dual Robot installation
- Planned Date: 02/25
- Expected Date: 03/05
- Status: Done
Milestone name: UR5 Calibrations (end-effector to ultrasound probe)
- Planned Date: 03/03
- Expected Date: 03/17
- Status: Done
Milestone name: Base-to-Base Calibration
- Planned Date: 03/17
- Expected Date: 03/27
- Status: N/A
Milestone name: Real Time Motion Planning for Dual Arms
- Planned Date: 03/27
- Expected Date: 04/05
- Status: Done
Milestone name: Virtual Fixture
- Planned Date: 04/05
- Expected Date: 04/30
- Status: Done
Milestone name: Enhanced motion control and path planning
- Planned Date: 04/05
- Expected Date: 04/30
- Status: Done
Milestone name: Real time demo
- Planned Date: 05/01
- Expected Date: 05/08
- Status: Done

{{ :courses:456:2023:projects:456-2023-18:cis_ii_final_report_2_.pdf |}}

Reports and presentations

Project Plan
- Project plan presentation
- Project plan proposal{{ :courses:456:2023:projects:456-2023-18:checkpoint_presentation.pptx |}}
Project Background Reading cis_ii_final_report_2_.pdf
Project Checkpoint
- Project checkpoint presentation
Paper Seminar Presentations
- Poster Teaser
Project Final Presentation
- PDF of Poster
Project Final Report
- Final Report
  - links to any appendices or other material
- Design Document https://www.overleaf.com/project/642a4b7024296e65f94343d5
- Mentor Report: cis_ii_2023_project_mentors_report_team18.pdf
- Phantom User Manual: Phantom User Manual

Project Bibliography

References

[1]Gilboy, Kevin Michael. ROBOTIC ULTRASOUND TOMOGRAPHY AND COLLABORATIVE CONTROL. 2020,

https://drive.google.com/file/d/19cGn_ZnCLbUESI9dta3l0s91Qdg-DwQl/view?usp=sharing.

[2]Ting-Yun Fang, Weiqi Wang. Co-Robotic Ultrasound Imaging System. 2017,

https://drive.google.com/file/d/16PvOlSm7f2qzwKBoGVO7q6Qlb2Tdtrwy/view?usp=sharing.

[3]Zerdine, Z.-Skin. Multi-Modality Pelvic Phantom. https://www.cirsinc.com/wp-content/uploads/2019/04/048A-DS-120418.pdf.

[4]Gilboy, Kevin M., et al. Dual-Robotic Ultrasound System for In Vivo Prostate Tomography. 2020, https://link.springer.com/chapter/10.1007/978-3-030-60334-2_16.

[5]Zhang, Haichong K., et al. ‘Phantom with Multiple Active Points for Ultrasound Calibration’. J. Med. Imag. 5(4), 045001 (2018), Doi: 10. 1117/1. JMI. 5. 4. 045001., 2022b, https://pubmed.ncbi.nlm.nih.gov/30525061/.

[6]Bray, F., et al. Global Cancer Statistics 2018: GLOBOCAN Estimates of Incidence and Mortality Worldwide for 36 Cancers in 185 Countries. Vol. 68(6), 2018, pp. 394–424.

[7]Horn, Berthold K. P., et al. ‘Closed-Form Solution of Absolute Orientation Using Orthonormal Matrices’. Journal of The Optical Society of America A-Optics Image Science and Vision, vol. 5, 1988, pp. 1127–1135.

[8]Seifabadi, R. ‘Correlation of Ultrasound Tomography to MRI and Pathology for the Detection of Prostate Cancer’. Medical Imaging 2019: Ultrasonic Imaging and Tomography International Society for Optics and Photonics, vol. 10955, 2019, p. 109550C.

[9]Aalamifar, F. ‘Co-Robotic Ultrasound Tomography: A New Paradigm for Quantitative Ultrasound Imaging’. Ph. D. Thesis, Johns Hopkins University, Oct. 2016.

Reading material