===== 3D Ultrasound-Guided Robot =====
{{ :research:3dus:3dus_thumb.png?180 }}
==== Problem ====
Explosions and similar incidents generate fragmented debris that can become lodged
in the body of a patient. By way of the venous system or direct penetration, particles
such as thrombi, bullet fragments, and shrapnel can become trapped in the heart
and disrupt cardiac function. Injuries of this nature are common in both civilian
and military populations. Symptomatic foreign bodies are conventionally removed
through open heart surgery, which comes with numerous risks and long recovery times.
==== Solution ====
To circumvent the disadvantages of open surgery, we propose a minimally invasive
surgical approach for the retrieval of foreign bodies from a beating heart. By reducing
risks such as bacterial mediastinitis, hemolysis, and bone fracture associated with
median sternotomy and cardiopulmonary bypass, a minimally invasive approach can
potentially improve the management of cardiac foreign bodies, postoperative recovery,
and procedure times. Because minimally invasive surgery introduces or amplifies
technical challenges due to limited workspace and dexterity, the proposed solution
includes a dexterous robotic end effector with the ability to navigate autonomously
under the guidance of 3D ultrasound. Streaming images from the ultrasound system
are processed in real time to track the foreign body, robot, and anatomical features
of interest.
{{ :research:3dus:3dus_system2.png?600 |Figure 1. Prototype surgical system}}
A system for minimally invasive evacuation of foreign bodies from a beating heart
is illustrated in Figure 1. Its primary components include
* an ultrasound system, including a 3D transesophageal echocardiography (TEE) probe,
* a robot that can be commanded via both manual operation and image guidance,
* a workstation computer tasked with ultrasound data processing and robot control, and
* a beating heart phantom used in preliminary studies (Figure 2b).
The beating heart phantom is a specially developed multimodality phantom compatible
with X-ray, ultrasound, and MR imaging. A full replica of a human heart, it is made
of PVA (Polyvinyl Acetate) and is fixed inside a Plexiglas water tank. Two servo-actuated
pneumatic pistons pump water into and out of the phantom to create the deformable
effect of a heartbeat and blood flow.
{{ :research:3dus:3dus_phantom.png?480 |Figure 2. (a) Experimental setup. (b) Heart phantom with TEE probe shown.}}
==== Methods ====
In preliminary studies, tracking of the robot end-effector and foreign body in the
ultrasound volumes is performed using a modified 3D normalized cross correlation
(NCC) method. Using NCC, the region of interest is first selected interactively
in the first image frame. This provides a template for the algorithm to automatically
track the foreign body in subsequent frames. In each frame, the position of maximum
correlation value, //Rmax//, is assumed to be the position of the foreign
body. A higher correlation value corresponds to a close template match, so an inadequate
match, i.e. //Rmax// < //Rmean// - c * //Rstdev//,
is rejected by the system. By removing 18.5% of ultrasound frames, a tracking accuracy
of 2.3 mm RMS can be achieved.
The advantage of this approach is that the simple implementation is amenable to
real-time performance, while working around the weaknesses of NCC. More advanced
techniques will be explored in the future that allow for tracking of complex shapes
in the presence of noise and image artifacts.
==== Preliminary Studies ====
=== Characterization of foreign body motion ===
The movement of a foreign body in the heart is complex; it influenced by pulmonary
motion, heartbeats, blood flow, and interactions with tissues. Whereas pulmonary
and cardiac motions are relatively predictable, interaction with heart structures
and interruption of blood flow around the foreign body would require a complex,
real-time model of the heart and fluid dynamics. In order to design a system to
retrieve such projectiles, we would like to understand and quantify its motion.
We selected a 3.2-mm steel ball to act as a foreign body (due to its likeness to
a clinical trauma case) and inserted it into the right ventricle of the beating
heart phantom. With the phantom activated with a preprogrammed heartbeat, 3D ultrasound
images of the moving foreign body were acquired at 20 frames per second and streamed
to a workstation computer over TCP/IP.
{{:research:3dus:3dus_slices.png?360|Figure 3. Ultrasound images showing the foreign body (outlined) in the heart phantom in two orthogonal slices, and in 3D.}}{{:research:3dus:3dus_3dus.png?198}}
NCC-based tracking was robust against a dynamic background, the moving heart wall.
Figure 3 displays ultrasound images of the foreign body in the phantom, while Figure
4 shows the motion traces of the foreign body in three axes.
{{ :research:3dus:3dus_fbtrace.png?480 |Figure 4. Motion traces of a foreign body in a beating heart phantom. Vertical lines correspond to rejected frames.}}
{{ :research:3dus:3dus_fbfreq.png?600 |Figure 5. Frequency components of the foreign body motion in three axes.}}
As can be inferred from Figure 4, though there is some correspondence between the
foreign body motion and heartbeat events, but there is a large component of arbitrary
motion as well. The frequency spectrum of the motion is shown in Figure 5. Significant
spectral power at both lower and higher frequencies suggests that the motion is
complex. The foreign body reaches a maximum speed and acceleration of approximately
343.5 mm/s and 7 800.0 mm/s2 respectively over a 40.2 mm range. Speeds
have previously been reported in the range of up to 300 mm/s for the heart wall
and 200 mm/s for the mitral valve.
These figures and behaviors are challenging for existing surgical robots to handle,
thus rendering infeasible a retrieval approach based on direct pursuit. The robot
experiments described below reiterate this observation.
=== Robot guidance experiments ===
First, a registration is computed between the robot and ultrasound probe coordinate
systems. Foreign body trajectories obtained from the characterization study are
used to guide the robot along the same path. The robot end effector is imaged under
3D ultrasound at 20 fps as it pursues the virtual foreign body. The end effector
is then tracked to determine the path traveled. We evaluate the feasibility and
efficacy of the proposed system by comparing the robot and foreign body path pairs---a
close match would suggest that the robot can follow and thus capture the target.
{{ :research:3dus:3dus_fbrobot.png?600 |Figure 6. Comparison of foreign body and robot tool tip motion traces.}}
An excerpt of the foreign body and robot tool tip motion traces are shown together
for comparison, separated into x-, y-, and z-axes. Because the robot used in these
experiments is incapable of reaching the speeds attained by the foreign body, analysis
is performed retrospectively by slowing down the virtual target. This setup allows
us to develop the robotic system and track the end effector. An alternate experiment
would employ a simulated robot that runs on real-time input data.
Different slowdown factors were tried over several experimental runs. In the 9x
slowdown case (Figure 6), the foreign body becomes effectively a series of near-static
targets along a path in 3D space. The RMS error between the recorded foreign body
and robot positions is 2.1 mm (9x reduction) and 1.6 mm (20x). Given that surgical
robots are not generally fast, capturing a foreign body in real time by direct pursuit
appears to be a difficult, and perhaps unsafe, task. Via continued experiments, we
advocate the investigation of an indirect retrieval approach.
We note that previous work in this area focus entirely on discrete sets of static
targets. The two preliminary experiments are summarized in the video below. The
red marker represents the tracked foreign body, while the blue line indicates the
robot position as it tracks the foreign body.
{{ youtube>ex68ixW9QyU | Foreign body and robot tracked in ultrasound volumes. The beating heart phantom moves in the background. }}
=== Spatial probability of the foreign body ===
It was observed from the characterization experiments that the foreign body tends
to visit only a subsection of the ventricle with regularity. We thus propose an
alternative retrieval approach: The real-time modified 3D NCC-based tracking method
described above is used to build a spatial probability map of the foreign body,
based on its past locations. A visualization of this is shown in Figure 7. Warmer
(red) regions indicate where the fragment is more likely to be located.
The apparent spatial locality of the foreign body suggests the viability of a more
relaxed retrieval approach based on aiming the dexterous robotic end effector at
a "waiting" position of high probability. This helps reduce difficult-to-achieve
robot performance requirements. On the average across datasets, the fragment spends
up to 50.5% of the time in the most visited location.
{{ :research:3dus:3dus_probmap.png?600 |Figure 7. Probability map of the foreign body position. Left and center: Coronal and sagittal slices. Right: 3D view.}}
==== Future Plans ====
//In vitro// simulation of the clinical scenario provides valuable insights on the
behavior of a foreign body in a beating heart, suggesting that its motion is fast,
abrupt, and often unpredictable. It demonstrates that for a system to be capable
of securing such a projectile, the robot must be agile and dexterous, and the capturing
mechanism must be carefully investigated as well. A few alternatives under consideration
include suction and various forms of grippers.
Although the phantom study simplifies the real clinical scenario, the ability to
achieve promising results using simple techniques encourages the continued development
in this direction. Future milestones for this work will involve handling increasingly
realistic situations. Among these tasks are online registration and tracking, robust
localization algorithms for complex shapes, improved visualization and tracking
for different foreign body types, and well defined behaviors under error conditions.
Of particular interest is real-time computation of virtual fixtures based on observed
tissue motion, allowing the system adjust to the dynamic environments inherent in
surgery. These virtual fixtures will be used to protect tissue from inadvertent
damage by the robot.
This project is a collaboration between ERC-CISST - Johns Hopkins University, Philips
Research North America, and Johns Hopkins Medical Institutions.
===== Project Personnel =====
* Graduate Students
* Paul Thienphrapa
* Clinicians
* Kaushik Mandal
* Faculty
* Aleksandra Popovic (Philips Research North America)
* [[http://www.cs.jhu.edu/~rht/|Russell H. Taylor]]
===== Funding =====
* Philips Research North America
* Johns Hopkins University internal funds
===== Affiliated Labs =====
* [[start|Computer Integrated Interventional Systems Laboratory]]
* [[http://www.research.philips.com/|Philips Research North America]]
===== Publications =====
* P. Thienphrapa, A. Popovic, and R. Taylor, “{{:research:3dus:icra2014.pdf|Guidance of a high dexterity robot under 3D ultrasound for minimally invasive retrieval of foreign bodies from a beating heart}},” //IEEE Int. Conf. on Robotics and Automation (ICRA)//, Hong Kong, China, May 2014.
* P. Thienphrapa, A. Popovic, and R. Taylor, “{{:research:3dus:hamlyn2013.pdf|3D ultrasound-guided retrieval of foreign bodies from a beating heart using a dexterous surgical robot}},” //Hamlyn Symposium on Medical Robotics//, London, UK, Jun. 2013.
* P. Thienphrapa, B. Ramachandran, H. Elhawary, A. Popovic, and R. Taylor, “{{:research:3dus:hamlyn2012.pdf|Intraoperative analysis of locations for 3D ultrasound-guided capture of foreign bodies from a beating heart}},” //Hamlyn Symposium on Medical Robotics//, London, UK, Jul. 2012.
* P. Thienphrapa, B. Ramachandran, R. Taylor, and A. Popovic, “{{:research:3dus:biorob2012.pdf|A system for 3D ultrasound-guided robotic retrieval of foreign bodies from a beating heart}},” //IEEE RAS/EMBS Int. Conf. on Biomedical Robotics and Biomechatronics (BioRob)//, Rome, Italy, Jun 2012.
* P. Thienphrapa, B. Ramachandran, H. Elhawary, R. Taylor, and A. Popovic, “{{:research:3dus:spie2012.pdf|Multiple capture locations for 3D ultrasound-guided robotic retrieval of moving bodies from a beating heart}},” //SPIE Medical Imaging//, San Diego, CA, Feb 2012.
* B. Ramachandran, P. Thienphrapa, A. Jain, and A. Popovic, “Tracking using 3D ultrasound for guiding cardiac interventions,” //Int. Conf. on Biomedical Engineering//, Manipal, India, Dec 2011.
* P. Thienphrapa, H. Elhawary, B. Ramachandran, D. Stanton, and A. Popovic, “{{:research:3dus:miccai2011.pdf|Tracking and characterization of fragments in a beating heart using 3D ultrasound for interventional guidance}},” //Medical Image Computing and Computer Assisted Intervention Conference (MICCAI)//, Toronto, Canada, Sep 2011.