Explosions and similar incidents generate fragmented debris that can become lodged in the body of a patient. By way of the venous system or direct penetration, particles such as thrombi, bullet fragments, and shrapnel can become trapped in the heart and disrupt cardiac function. Injuries of this nature are common in both civilian and military populations. Symptomatic foreign bodies are conventionally removed through open heart surgery, which comes with numerous risks and long recovery times.

To circumvent the disadvantages of open surgery, we propose a minimally invasive surgical approach for the retrieval of foreign bodies from a beating heart. By reducing risks such as bacterial mediastinitis, hemolysis, and bone fracture associated with median sternotomy and cardiopulmonary bypass, a minimally invasive approach can potentially improve the management of cardiac foreign bodies, postoperative recovery, and procedure times. Because minimally invasive surgery introduces or amplifies technical challenges due to limited workspace and dexterity, the proposed solution includes a dexterous robotic end effector with the ability to navigate autonomously under the guidance of 3D ultrasound. Streaming images from the ultrasound system are processed in real time to track the foreign body, robot, and anatomical features of interest.

A system for minimally invasive evacuation of foreign bodies from a beating heart is illustrated in Figure 1. Its primary components include

• an ultrasound system, including a 3D transesophageal echocardiography (TEE) probe,
• a robot that can be commanded via both manual operation and image guidance,
• a workstation computer tasked with ultrasound data processing and robot control, and
• a beating heart phantom used in preliminary studies (Figure 2b).

The beating heart phantom is a specially developed multimodality phantom compatible with X-ray, ultrasound, and MR imaging. A full replica of a human heart, it is made of PVA (Polyvinyl Acetate) and is fixed inside a Plexiglas water tank. Two servo-actuated pneumatic pistons pump water into and out of the phantom to create the deformable effect of a heartbeat and blood flow.

In preliminary studies, tracking of the robot end-effector and foreign body in the ultrasound volumes is performed using a modified 3D normalized cross correlation (NCC) method. Using NCC, the region of interest is first selected interactively in the first image frame. This provides a template for the algorithm to automatically track the foreign body in subsequent frames. In each frame, the position of maximum correlation value, R_max, is assumed to be the position of the foreign body. A higher correlation value corresponds to a close template match, so an inadequate match, i.e. R_max < R_mean - c * R_stdev, is rejected by the system. By removing 18.5% of ultrasound frames, a tracking accuracy of 2.3 mm RMS can be achieved.

The advantage of this approach is that the simple implementation is amenable to real-time performance, while working around the weaknesses of NCC. More advanced techniques will be explored in the future that allow for tracking of complex shapes in the presence of noise and image artifacts.

The movement of a foreign body in the heart is complex; it influenced by pulmonary motion, heartbeats, blood flow, and interactions with tissues. Whereas pulmonary and cardiac motions are relatively predictable, interaction with heart structures and interruption of blood flow around the foreign body would require a complex, real-time model of the heart and fluid dynamics. In order to design a system to retrieve such projectiles, we would like to understand and quantify its motion.

We selected a 3.2-mm steel ball to act as a foreign body (due to its likeness to a clinical trauma case) and inserted it into the right ventricle of the beating heart phantom. With the phantom activated with a preprogrammed heartbeat, 3D ultrasound images of the moving foreign body were acquired at 20 frames per second and streamed to a workstation computer over TCP/IP.

NCC-based tracking was robust against a dynamic background, the moving heart wall. Figure 3 displays ultrasound images of the foreign body in the phantom, while Figure 4 shows the motion traces of the foreign body in three axes.

As can be inferred from Figure 4, though there is some correspondence between the foreign body motion and heartbeat events, but there is a large component of arbitrary motion as well. The frequency spectrum of the motion is shown in Figure 5. Significant spectral power at both lower and higher frequencies suggests that the motion is complex. The foreign body reaches a maximum speed and acceleration of approximately 343.5 mm/s and 7 800.0 mm/s² respectively over a 40.2 mm range. Speeds have previously been reported in the range of up to 300 mm/s for the heart wall and 200 mm/s for the mitral valve.

These figures and behaviors are challenging for existing surgical robots to handle, thus rendering infeasible a retrieval approach based on direct pursuit. The robot experiments described below reiterate this observation.

First, a registration is computed between the robot and ultrasound probe coordinate systems. Foreign body trajectories obtained from the characterization study are used to guide the robot along the same path. The robot end effector is imaged under 3D ultrasound at 20 fps as it pursues the virtual foreign body. The end effector is then tracked to determine the path traveled. We evaluate the feasibility and efficacy of the proposed system by comparing the robot and foreign body path pairs—a close match would suggest that the robot can follow and thus capture the target.

An excerpt of the foreign body and robot tool tip motion traces are shown together for comparison, separated into x-, y-, and z-axes. Because the robot used in these experiments is incapable of reaching the speeds attained by the foreign body, analysis is performed retrospectively by slowing down the virtual target. This setup allows us to develop the robotic system and track the end effector. An alternate experiment would employ a simulated robot that runs on real-time input data.

Different slowdown factors were tried over several experimental runs. In the 9x slowdown case (Figure 6), the foreign body becomes effectively a series of near-static targets along a path in 3D space. The RMS error between the recorded foreign body and robot positions is 2.1 mm (9x reduction) and 1.6 mm (20x). Given that surgical robots are not generally fast, capturing a foreign body in real time by direct pursuit appears to be a difficult, and perhaps unsafe, task. Via continued experiments, we advocate the investigation of an indirect retrieval approach.

We note that previous work in this area focus entirely on discrete sets of static targets. The two preliminary experiments are summarized in the video below. The red marker represents the tracked foreign body, while the blue line indicates the robot position as it tracks the foreign body.

Foreign body and robot tracked in ultrasound volumes. The beating heart phantom moves in the background.

It was observed from the characterization experiments that the foreign body tends to visit only a subsection of the ventricle with regularity. We thus propose an alternative retrieval approach: The real-time modified 3D NCC-based tracking method described above is used to build a spatial probability map of the foreign body, based on its past locations. A visualization of this is shown in Figure 7. Warmer (red) regions indicate where the fragment is more likely to be located.

The apparent spatial locality of the foreign body suggests the viability of a more relaxed retrieval approach based on aiming the dexterous robotic end effector at a “waiting” position of high probability. This helps reduce difficult-to-achieve robot performance requirements. On the average across datasets, the fragment spends up to 50.5% of the time in the most visited location.

In vitro simulation of the clinical scenario provides valuable insights on the behavior of a foreign body in a beating heart, suggesting that its motion is fast, abrupt, and often unpredictable. It demonstrates that for a system to be capable of securing such a projectile, the robot must be agile and dexterous, and the capturing mechanism must be carefully investigated as well. A few alternatives under consideration include suction and various forms of grippers.

Although the phantom study simplifies the real clinical scenario, the ability to achieve promising results using simple techniques encourages the continued development in this direction. Future milestones for this work will involve handling increasingly realistic situations. Among these tasks are online registration and tracking, robust localization algorithms for complex shapes, improved visualization and tracking for different foreign body types, and well defined behaviors under error conditions. Of particular interest is real-time computation of virtual fixtures based on observed tissue motion, allowing the system adjust to the dynamic environments inherent in surgery. These virtual fixtures will be used to protect tissue from inadvertent damage by the robot.

This project is a collaboration between ERC-CISST - Johns Hopkins University, Philips Research North America, and Johns Hopkins Medical Institutions.

• Graduate Students
  • Paul Thienphrapa
• Clinicians
  • Kaushik Mandal
• Faculty
  • Aleksandra Popovic (Philips Research North America)
  • Russell H. Taylor

• Philips Research North America
• Johns Hopkins University internal funds

• Computer Integrated Interventional Systems Laboratory
• Philips Research North America

• P. Thienphrapa, A. Popovic, and R. Taylor, “Guidance of a high dexterity robot under 3D ultrasound for minimally invasive retrieval of foreign bodies from a beating heart,” IEEE Int. Conf. on Robotics and Automation (ICRA), Hong Kong, China, May 2014.
• P. Thienphrapa, A. Popovic, and R. Taylor, “3D ultrasound-guided retrieval of foreign bodies from a beating heart using a dexterous surgical robot,” Hamlyn Symposium on Medical Robotics, London, UK, Jun. 2013.
• P. Thienphrapa, B. Ramachandran, H. Elhawary, A. Popovic, and R. Taylor, “Intraoperative analysis of locations for 3D ultrasound-guided capture of foreign bodies from a beating heart,” Hamlyn Symposium on Medical Robotics, London, UK, Jul. 2012.
• P. Thienphrapa, B. Ramachandran, R. Taylor, and A. Popovic, “A system for 3D ultrasound-guided robotic retrieval of foreign bodies from a beating heart,” IEEE RAS/EMBS Int. Conf. on Biomedical Robotics and Biomechatronics (BioRob), Rome, Italy, Jun 2012.
• P. Thienphrapa, B. Ramachandran, H. Elhawary, R. Taylor, and A. Popovic, “Multiple capture locations for 3D ultrasound-guided robotic retrieval of moving bodies from a beating heart,” SPIE Medical Imaging, San Diego, CA, Feb 2012.
• B. Ramachandran, P. Thienphrapa, A. Jain, and A. Popovic, “Tracking using 3D ultrasound for guiding cardiac interventions,” Int. Conf. on Biomedical Engineering, Manipal, India, Dec 2011.
• P. Thienphrapa, H. Elhawary, B. Ramachandran, D. Stanton, and A. Popovic, “Tracking and characterization of fragments in a beating heart using 3D ultrasound for interventional guidance,” Medical Image Computing and Computer Assisted Intervention Conference (MICCAI), Toronto, Canada, Sep 2011.