Last updated: 5/14/2013 and 20:21

All project documents can be found on this wiki, as well as at https://www.dropbox.com/sh/4axyopck81c926j/_7sdX8nzu6.

Our project aims at improving the existing interface for the da Vinci robot console and make it ready to be used clinically for Ultrasound guided surgery. Surgical collaborators will be recruited for clinical testing to determine the effectiveness of the interface in minimizing the difference in success of liver surgeries between novice and expert surgeons. Further improvements will be incorporated into the interface to allow the user to manipulate 3D lesions and fiducials within the da Vinci console itself.

• Students: Vineeta Khatuja, Andrew Wang, Tifany Yung
• Mentor(s): Michael Choti MD MBA, Theodoros Katsichtis, Colin Lea, Russell Taylor PhD

Computer-assisted laparoscopic procedures are becoming increasingly common. Laparoscopies allow for minimally invasive procedures in the abdominal cavity using a few small incisions as opposed to one large one in an open surgery. These laparoscopies can be augmented with ultrasonography which provides real-time imaging to update the preoperative data. Laparoscopic ultrasound can greatly improve the efficacy of many operations including increasing the accuracy of biopsies and ablations as well as scanning and detecting lesions on the liver, kidney and many other organs.

A robotic platform offers the surgeon an array of advantages compared to conventional open and laparoscopic techniques. Stereoscopic 3D vision, 7 degrees of motion and physiologic tremor attenuation allow the surgeon to be more precise and competent.

Real time ultrasonography is commonly used in modern open, laparoscopic and robotic surgery. Ultrasound data guide the surgical decision making process intraoperatively when localizing a lesion and determining its position relative to other structures.

We propose the development and utilization of a novel da Vinci interface, integrating and displaying live intraoperative US as well as additional preoperative data (CT, MRI, X-ray and DICOM) to the surgeon. Displaying all the necessary data to the surgeon through the Da Vinci master console can potentially increase the ergonomy and efficacy of robotic procedures improving patient outcome.

1. Inputs
   • Video grabbers provide video feed to the laptop and interface from two sources:
     1. da Vinci operative field camera
     2. ultrasound machine
2. Surgical WorkStation
   • This is the laptop on which the interface was developed. It is responsible for processing the input data and perform necessary calculations.
   • An HDMI cable connects this to the da Vinci to project the interface to the bottom half of the console display.
3. Surgical Console
   • This is the da Vinci master-side console used by the surgeon while performing the surgical procedure.

TilePro is a software feature on the da Vinci. The feature partitions the visual space which displays the interface that the surgeon sees on the console.

Out implementation of TilePro takes uses the two slots available in the da Vinci console:

1. Upper slot : Displays the the 3D live video of the operative field using the Camera 1 and Camera 2 inputs.
2. Lower Slot : Displays the live ultrasound video, image browser and other related items.

• Minimum:
  1. Acquire software dependencies, Mock OR access. - Complete.
  2. Propose budget and acquire funding for da Vinci usage ($50/hr) from Dr. Taylor. - Complete.
  3. Preoperative image viewer: implement saving and browsing of US images on the console. - Complete.
  4. Preliminary nonclinical study design in mock OR. - Complete.
  5. Masters as Mice implementation. - Complete.
• Expected:
  1. Clinical study design. - Complete.
  2. Submit and have IRB approved. - Complete.
  3. Contact surgical collaborators for clinical study. - In progress.
  4. Schedule and confirm participating surgeons for study. - In progress.
  5. Perform test and deployment of software in the mock OR. - Complete.
  6. Changes to software after nonclinical testing. - In progress.
• Maximum:
  1. Complete clinical study of software at the hospital with actual patients. - Not expected.
  2. Features/changes requested from clinical tests. - Not expected.
  3. Tool to measure lesion size on the ultrasound image. - Not expected.
  4. Use 3D fiducials to show previously viewed areas. - Not expected.
  5. Build 3D model of anatomical structures from CT. - Not expected.
  6. Provide flexible view manipulation of 3D models. - Not expected.
• Possible follow-ups:
  1. Speech-to-text intrap notes.
  2. 3D representation of the US probe in the UI.
  3. Enable 3D fiducial placement on the 3D models, not just the US images.

The interface was designed to allow the maximum, but user-friendly, display of data within the available space in the console display while preserving the aspect ratio of the videos and images to prevent image distortion. The layout was kept simple to allow for easier incorporation of new features later on, as well as to avoid diverting the surgeon's attention with flashy widgets. A dark-toned color scheme was also chosen for this reason.

The interface was implemented so that events are triggered by sliding the mouse over the desired tabs and images, rather than using a mouse click, as mentioned later. This was implemented in order to avoid fatiguing the surgeon's further during the procedure.

The interface described below is contained in the bottom half of the da Vinci console display inside the TilePro; the top half is reserved for the da Vinci camera feed. On the right side of the interface is an image browser used to view saved intraoperative ultrasound and stereo images, as well as preoperative images. The left side holds a video viewer for the live ultrasound feed, in addition to two image viewers to allow the surgeon to view a saved ultrasound image with its corresponding stereo image taken at the same time. Alternatively, this left side of the interface can display an enlarged preoperative image. The former layout appears if the surgeon choose to view an intraoperatively-saved ultrasound; the latter takes its place if the surgeon instead chooses a preoperative image to view. All of these features (Figure 1) are described in detail under the Software development subsection.

• Software:

1. QT framework The interface is built using the QT application framework, a cross platform application framework that is widely used for developing graphical user interfaces of application software. QT uses a layout-based approach to widget management, in which widgets are arranged in the optimal positions in windows based on simple layout rules, leading to a consistent and organized look and feel. QT provides a signal and slot mechanism to communicate between objects. A signal, emitted when a particular event occurs, triggers a slot, which is a function that is called in response to a particular signal. QT widgets have many predefined signals, but widget subclasses can always be made in order to add custom signals to them. Pre-defined slots can be used, or new slots can be defined.
2. Masters as Mice The da Vinci tool manipulators at the master console were chosen as the mouse controller to be used in the interface since the surgeon would not have to leave the console or look away, as would be the case if the laptop mouse were to be used instead. Synthetic QT mouse events are generated by the movement and clicking of the Master Right Tool Manipulator (MRTM), the right-hand side tool manipulator at the master's side (Figure 2). These synthetic events are then posted to the main QT application and handled accordingly by the program, thus triggering events in the interface such as a change in tab or image selection.The interface was designed such that pressing and holding the left clutch pedal at the master console and then pinching both tool manipulators simultaneously triggers the Masters as Mice mode, thus activating the mouse within the interface. Masters as Mice is released by pinching both manipulators simultaneously again. This feature was implemented using callback mechanism. Callback functions are called when da Vinci events like putting the head into the console, pressing or releasing tool manipulators, and clicking occur and are detected by the da Vinci system.These callback functions set the state of the system using state variables, which is then used to decide if the masters as mice mode is activated or not.The MRTM move callback retrieves the 3D coordinates of the MRTM tool tip position. These points are projected onto a 2D plane by discarding the z-coordinates to get a 2D point position. A QT signal is then emitted, which is connected to a QT slot which creates a new QMouseMove event and updates the cursor position.
3. Image Browser Two separate tabs are created for preoperative and Ultrasound image browsing.Hovering the mouse cursor over the images in the image browser changes the current selected image in the image viewer. The image browser is implemented using the QTabWidget.It was essential to have an intuitive and easy scroll option. For this reason the scrolling method chosen is similar to the Kinetic scrolling feature as in mobile phones where tap - drag - release is used to scroll. We have used the pinch - drag -realease model.(Pinch means clicking the da Vinci right master tool manipulator).The scrollbar is implemented using the QScrollBar. The value fiield of the QScrollBar is responsible for which part of the scrollbar is currently visible. The 2D positon of the mouse cursor in the interface is noted at the time of pinch and release, the scroll value is incremted by taking the difference between the y-positons of the two positions.
4. Image Viewer There are two image viewers. The Ultrasound image viewer shows the ultrasound and the live operative field images saved , which is based on the current selected image in the image browser.Preoperative image viewer shows the current selected preoperative image in the image browser.The image viewer is implemented using the QWidget and QLabel which is used to display the images.
5. Video ViewerA video viewer, which is used to display the live ultrasound feed mentioned above, was implemented and placed on the left of the interface using a QWidget as the medium with which to display it. Also as mentioned above, the video viewer is only visible while viewing the intraoperatively-saved ultrasound images. The cisst SVL library is used to capture the video and display it on the QWidget.

• Non-Clinical Testing:
  • Once the baseline interface is complete (3D lesion mapping is removed, color scheme and interface component arrangement improved), our team members Andrew and Tifany will perform non-clinical tests on the interface. These tests aim to assess the stability and compatibility of the baseline interface on the Hopkins da Vinci system. The tests will be performed using the da Vinci robot in the Hackerman mock OR on a gelatin phantom liver with pseudo-lesions. We will perform the tasks of identifying and locating lesions using the LapUS interface. Since the interface has not been tested for more than a period of 10 minutes, our goal is to use the interface for a more extensive period of time to check for any ossibility of the UI crashing during use and to discover any possible glitches or bugs.
  • Once the enhanced interface is complete (3D organ model manipulation, 3D lesion mapping properly implemented, etc.), another stability and functionality test will be performed to discover potentials for crashes or glitches.
• Clinical Testing:
  • A clinical study will then be performed to test the utility and functionality of these added features. Since we plan to have surgeons test the interface during procedures on actual patients, we will require an IRB approval. A questionnaire will be issued to the surgeons participating in the study that will determine their satisfaction and comfort with interface. A supplementary questionnaire will also be issued to determine the surgeons’ opinions of the LapUS interface with respect to having just the da Vinci live camera feed.

1. Windows computer compatible with software dependencies.
   • Required to install software dependencies and begin work on the interface.
   • Status: resolved.
2. QT Creator IDE (application framework for GUI development)
   • Open source: download from http://qt.digia.com/Product/Developer-Tools/.
   • Status: resolved.
3. CISST (libraries for development of computer-assisted intervention systems)
   • Available for download on the ERC CISST website: https://trac.lcsr.jhu.edu/cisst.
   • Status: resolved.
4. Intuitive API (source code for control of the Da Vinci surgical robot)
   • Requires signed non-disclosure agreement between Intuitive Surgical and each group member.
   • Status: resolved
5. Video grabber
   • Install onto laptop to allow video connection with da Vinci console.
   • Status: resolved.
6. Budget proposal for project funds to use Mock OR.
   • Mock OR costs $50/hr to use.
   • Status: resolved.
7. IRB proposal submission.
   • IRB proposal is needed from the Homewood IRB to conduct clinical studies.
   • Status: resolved.
8. Liver phantom
   • Gelatin phantom with pseudolesions recommended by Dr. Choti.
   • Cellulose and agarose/glycerol lesion imitations embedded in collagen phantom.
   • Status: resolved.
9. Surgical/clinical collaborators
   • Needed as subjects and to assist with clinical testing of the new integrated interface.
   • Status: in progress.
10. Mock OR and Da Vinci robot access
    • Permission must be granted by Dr. Taylor to each member.
    • Status: resolved.

1. Obtain Mock OR access.
   • Planned Date: 2/22/13
   • Expected Date: 2/21/13
   • Status: Complete.
2. Set up software dependencies for GUI development.
   • Planned Date: 2/22/13
   • Expected Date: 3/29/13
   • Status: Complete.
3. US images save and browse on console.
   • Planned Date: 3/8/13
   • Expected Date: 3/29/13
   • Status: Complete.
4. Testing/debugging plan.
   • Planned Date: 3/1/13
   • Expected Date: not expected by end of semester.
   • Status: In progress.
5. Contact surgical collaborators for clinical study
   • Planned Date: 3/1/13
   • Expected Date: not expected by end of semester.
   • Status: In progress.
6. Have participating surgeons scheduled and confirmed.
   • Planned Date: 3/8/13
   • Expected Date: not expected by end of semester.
   • Status: In progress.
7. IRB for clinical study
   • Planned Date: 4/1/13
   • Expected Date: 4/30/13
   • Status: Complete.
8. Complete baseline interface testing in the Mock OR.
   • Planned Date: 3/22/13
   • Expected Date: 4/25/13
   • Status: Complete.
9. Masters as Mice.
   • Planned Date: 4/5/13
   • Expected Date: 4/25/13
   • Status: Complete.
10. Ability to manipulate a 3D model of a lesion or organ
    • Planned Date: 4/19/13
    • Expected Date: not expected.
    • Status: Not started.
11. Use 3D fiducials to show previously viewed areas
    • Planned Date: 5/3/13
    • Expected Date: not expected.
    • Status: Not started.
12. Clinical study design (task experiments, user feedback survey).
    • Planned Date: 4/19/13
    • Expected Date: 2/22/13
    • Status: Complete.
13. Complete clinical testing of the software at the hospital.
    • Planned Date: 5/3/13
    • Expected Date: not expected by end of semester.
    • Status: Not started.

• Project Plan
  • Project plan presentation
  • proposal_document.pdf
• Project Background Reading
  • See Bibliography below for links.
• Project Checkpoint
  • checkpointpresentation.pptx
  • checkpointpdf.pdf
• Paper Seminar Presentations
  • Vineeta
    • Vineeta Presentation
    • Vineeta Paper Critical Review
    • Vineeta Paper-I
    • Vineeta Paper-II
  • Tifany
    • rcs1471_paper_tifany.pdf
    • seminar_paper_tifany.pdf
    • seminar_presentation_tifany.ppt
  • Andrew
    • maximizing_console_surgeon_independence_by_using_fourth_arm_and_tile_pro.pdf
    • robotic_ultrasound_probe_for_tumor_identification_in_robotic.pdf
    • data_integration_during_robotic_ultrasound_paper_seminar.pdf
    • paper_presentation.pptx
• Project Miniature Checkpoint
  • minicheckpoint.pptx
• Project Poster Teaser
  • posterteaser_final.pptx
• Project Final Presentation Poster
  • finalposter_last.pptx
• Project Final Report
  • finalreport_last.pdf
  • Please see Other Resources and Project Files for LapUS source code.

• Bartosz F. Kaczmarek, S. S., Firas Petros, Quoc-Dien Trinh, Navneet Mander, Roger Chen, Mani Menon, Craig G. Rogers (2012). “Robotic ultrasound probe for tumor identification in robotic partial nephrectomy: Initial series and outcomes.” International Journal of Urology.
• Caitlin M. Schneider, B. P. D. P., MD; Russell H. Taylor, PhD; Gregory W. Dachs II, MS; Christopher J. Hasser, PhD; Simon P. DiMaio, PhD; Michael A. Choti, MD, MBA, FACS Surgical Technique: Robot-assisted laparoscopic ultrasonography for hepatic surgery.
• Caitlin M. Schneider, G. W. D. I., Christopher J. Hasser, Michael A. Choti, Simon P. DiMaio, Russell H. Taylor Robot-Assisted Laparoscopic Ultrasound, Johns Hopkins University; Johns Hopkins Medicine; Intuitive Surgical, Inc.
• Craig G. Rogers, M. R. L., MD; Akshay Bhandari, MD; Louis Spencer Krane, MD; Daniel Eun, MD; Manish N. Patel, MD; Ronald Boris, MD; Alok Shrivastava, MD; Mani Menon, MD (2009). “Maximizing Console Surgeon Independence during Robot-Assisted Renal Surgery by Using the Fourth Arm and TilePro.” Journal of Endourolgy 23(1): 115-121.
• Francesco Volonté, N. C. B., François Pugin, Joël Spaltenstein, Boris Schiltz, Minoa Jung, Monika Hagen, Osman Ratib, Philippe Morel (2012). “Augmented reality to the rescue of the minimally invasive surgeon. The usefulness of the interposition of stereoscopic images in the Da Vinci robotic console.” The International Journal of Medical Robotics and Computer Assisted Surgery.
• Joshua Leven, D. B., Rajesh Kumar, Gary Zhang, Steve Blumenkranz, Xiangtian (Donald) Dai, Mike Awad, Gregory D. Hager, Mike Marohn, Mike Choti, Chris Hasser, Russell H. Taylor DaVinci Canvas: A Telerobotic Surgical System with Integrated, Robot-Assisted, Laparoscopic Ultrasound Capability, The Johns Hopkins University; Intuitive Surgical, Inc.

• Budget proposal budgetproposal.pdf
• Clinical testing protocol protocol_draft2.docx
• Confidential: IRB proposal confidential_eforma_na_00082654.doc
• Project source code project9_lapus.zip
• All project documentsalldocs.zip