Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
courses:446:2013:446-2013-09:446-2013-09 [2013/05/11 00:40]
awang29@johnshopkins.edu [Tile Pro]
courses:446:2013:446-2013-09:446-2013-09 [2019/08/07 12:01] (current)
Line 1: Line 1:
 ======Data Integration during Robotic Ultrasound-guided Surgery====== ======Data Integration during Robotic Ultrasound-guided Surgery======
-**Last updated: ​Date and time** +**Last updated: ​5/​14/​2013 ​and 20:21**
  
 +All project documents can be found on this wiki, as well as at https://​www.dropbox.com/​sh/​4axyopck81c926j/​_7sdX8nzu6.
 ======Summary====== ======Summary======
 {{ :​courses:​446:​2013:​446-2013-09:​euro_uro.png?​nolink&​200|}} {{ :​courses:​446:​2013:​446-2013-09:​euro_uro.png?​nolink&​200|}}
Line 20: Line 20:
 We propose the development and utilization of a novel da Vinci interface, integrating and displaying live intraoperative US as well as additional preoperative data (CT, MRI, X-ray and DICOM) to the surgeon. Displaying all the necessary data to the surgeon through the Da Vinci master console can potentially increase the ergonomy and efficacy of robotic procedures improving patient outcome. ​ We propose the development and utilization of a novel da Vinci interface, integrating and displaying live intraoperative US as well as additional preoperative data (CT, MRI, X-ray and DICOM) to the surgeon. Displaying all the necessary data to the surgeon through the Da Vinci master console can potentially increase the ergonomy and efficacy of robotic procedures improving patient outcome. ​
  
-======Current ​System=======+======Hardware ​System=======
     - Inputs     - Inputs
-       ​* ​There are three inputs ​to the surgical workstation. +       ​* ​Video grabbers provide video feed to the laptop and interface from two sources: 
-          - Camera 1 +          - da Vinci operative field camera 
-          - Camera 2 +          - ultrasound machine
-          - Live Ultrasound+
     - Surgical WorkStation     - Surgical WorkStation
-       * It is responsible for processing the input data and perform necessary calculations.+       ​* ​This is the laptop on which the interface was developed. ​It is responsible for processing the input data and perform necessary calculations
 +       * An HDMI cable connects this to the da Vinci to project the interface to the bottom half of the console display.
     - Surgical Console     - Surgical Console
-       * This is the console ​seen by the surgeon while performing the surgical procedure. +       * This is the da Vinci master-side ​console ​used by the surgeon while performing the surgical procedure.
-{{ :​courses:​446:​2013:​446-2013-09:​block_diagram.png?​nolink |}} +
- +
 ======Tile Pro====== ======Tile Pro======
 **TilePro** is a software feature on the da Vinci. The feature partitions the visual space which displays the interface that the surgeon sees on the console. ​ **TilePro** is a software feature on the da Vinci. The feature partitions the visual space which displays the interface that the surgeon sees on the console. ​
Line 69: Line 66:
     - Enable 3D fiducial placement on the 3D models, not just the US images.     - Enable 3D fiducial placement on the 3D models, not just the US images.
 ======Technical Approach====== ======Technical Approach======
 +The interface was designed to allow the maximum, but user-friendly,​ display of data within the available space in the console display while preserving the aspect ratio of the videos and images to prevent image distortion. The layout was kept simple to allow for easier incorporation of new features later on, as well as to avoid diverting the surgeon'​s attention with flashy widgets. A dark-toned color scheme was also chosen for this reason.
 +
 +The interface was implemented so that events are triggered by sliding the mouse over the desired tabs and images, rather than using a mouse click, as mentioned later. This was implemented in order to avoid fatiguing the surgeon'​s further during the procedure.
 +
 +The interface described below is contained in the bottom half of the da Vinci console display inside the TilePro; the top half is reserved for the da Vinci camera feed. On the right side of the interface is an image browser used to view saved intraoperative ultrasound and stereo images, as well as preoperative images. The left side holds a video viewer for the live ultrasound feed, in addition to two image viewers to allow the surgeon to view a saved ultrasound image with its corresponding stereo image taken at the same time. Alternatively,​ this left side of the interface can display an enlarged preoperative image. The former layout appears if the surgeon choose to view an intraoperatively-saved ultrasound; the latter takes its place if the surgeon instead chooses a preoperative image to view. All of these features (Figure 1) are described in detail under the Software development subsection.
 +
 +{{:​courses:​446:​2013:​446-2013-09:​hardwaresetup.png?​600|}}
 +
   * **Software:​**   * **Software:​**
-    ​Remove ​the 3D lesion mapping tool+  ​**QT framework** The interface is built using the QT application framework, a cross platform application framework that is widely used for developing graphical user interfaces of application software. QT uses a layout-based approach to widget management, in which widgets are arranged in the optimal positions in windows based on simple layout rules, leading to a consistent and organized look and feel. QT provides a signal and slot mechanism to communicate between objects. A signal, emitted when a particular event occurs, triggers ​ a slot, which  is a function that is called in response to a particular signal. QT widgets have many predefined signals, but widget subclasses can always be made in order to add custom signals to them. Pre-defined slots can be used, or new slots can be defined.  
-      Understand ​the LapUs code and identify ​the code to be removed. +  ​- ​**Masters as Mice** The da Vinci tool manipulators at the master console were chosen as the mouse controller ​to be used in the interface since the surgeon would not have to leave the console or look away, as would be the case if the laptop mouse were to be used insteadSynthetic QT mouse events are generated by the movement and clicking of the Master Right Tool Manipulator (MRTM), the right-hand side tool manipulator at the master'​s side (Figure 2)These synthetic events are then posted to the main QT application and handled accordingly by the program, thus triggering events in the interface such as a change in tab or image selection.The interface was designed such that pressing and holding the left clutch pedal at the master console and then pinching both tool manipulators simultaneously triggers the Masters as Mice mode, thus activating the mouse within the interface. Masters as Mice is released by pinching both manipulators simultaneously again. This feature was implemented using callback mechanism. Callback functions are called when da Vinci events like putting the head into the console, pressing or releasing tool manipulators,​ and clicking occur and are detected by the da Vinci system.These callback functions set the state of the system using state variables, which is then used to decide if the masters as mice mode is activated or not.The MRTM move callback retrieves the 3D coordinates of the MRTM tool tip positionThese points are projected onto a 2D plane by discarding the z-coordinates to get a 2D point position. A QT signal is then emitted, which is connected to a QT slot which creates a new QMouseMove event and updates the cursor position
-    ​Real time operative ​tool measurement. +  ​- ​**Image Browser** ​ Two separate tabs are created for preoperative and Ultrasound image browsing.Hovering the mouse cursor over the images ​in the image browser changes the current selected image in the image viewer. The image browser is implemented using the QTabWidget.It was essential to have an intuitive ​and easy scroll optionFor this reason the scrolling method chosen is similar to the Kinetic scrolling feature as in mobile phones where  tap drag release is used to scroll. We have used the pinch - drag -realease model.(Pinch means clicking the da Vinci right master tool manipulator).The scrollbar is implemented using the QScrollBar. The value fiield of the QScrollBar is responsible for which part of the scrollbar is currently visible. The 2D positon of the mouse cursor in the interface ​is noted at the time of pinch and release, the scroll value is incremted by taking the difference between the y-positons of the two positions
-      * Use the daVinci API to get the operative field measurement ​tool. +  ​- ​**Image Viewer** There are two image viewers. The Ultrasound image viewer shows the ultrasound and the live operative field images saved , which is based on the current selected image in the image browser.Preoperative image viewer shows the current selected preoperative image in the image browser.The image viewer is implemented using the QWidget and QLabel which is used to display the images
-    ​Lesion measurement tool+  ​- ​**Video Viewer**A video viewer, which is used to display the live ultrasound feed mentioned above, was implemented ​and placed on the left of the interface using a QWidget as the medium ​with which to display it. Also as mentioned above, ​the video viewer is only visible while viewing the intraoperatively-saved ultrasound ​images. ​The cisst SVL library is used to capture the video and display it on the QWidget. 
-      Perform calibration between ​the US images and user interface. +
-    ​User-friendly ​interface. +
-      Less disruptive color scheme+
-      Save and browse capabilities ​with the US images, with lesion descriptions.+
   * **Non-Clinical Testing:​** ​   * **Non-Clinical Testing:​** ​
       * Once the baseline interface is complete (3D lesion mapping is removed, color scheme and interface component arrangement improved), our team members Andrew and Tifany will perform non-clinical tests on the interface. These tests aim to assess the stability and compatibility of the baseline interface on the Hopkins da Vinci system. The tests will be performed using the da Vinci robot in the Hackerman mock OR on a gelatin phantom liver with pseudo-lesions. We will perform the tasks of identifying and locating lesions using the LapUS interface. Since the interface has not been tested for more than a period of 10 minutes, our goal is to use the interface for a more extensive period of time to check for any ossibility of the UI crashing during use and to discover any possible glitches or bugs.       * Once the baseline interface is complete (3D lesion mapping is removed, color scheme and interface component arrangement improved), our team members Andrew and Tifany will perform non-clinical tests on the interface. These tests aim to assess the stability and compatibility of the baseline interface on the Hopkins da Vinci system. The tests will be performed using the da Vinci robot in the Hackerman mock OR on a gelatin phantom liver with pseudo-lesions. We will perform the tasks of identifying and locating lesions using the LapUS interface. Since the interface has not been tested for more than a period of 10 minutes, our goal is to use the interface for a more extensive period of time to check for any ossibility of the UI crashing during use and to discover any possible glitches or bugs.
Line 201: Line 203:
   * Project Poster Teaser   * Project Poster Teaser
     * {{:​courses:​446:​2013:​446-2013-09:​posterteaser_final.pptx|}}     * {{:​courses:​446:​2013:​446-2013-09:​posterteaser_final.pptx|}}
-  * Project Final Presentation Poster ​(in progress; to be uploaded prior to Monday poster session) +  * Project Final Presentation Poster 
-    * {{:​courses:​446:​2013:​446-2013-X:final_poster_pdf.pdf|PDF of Poster}} +    * {{:​courses:​446:​2013:​446-2013-09:finalposter_last.pptx|}} 
-  * Project Final Report ​(in progress; to be uploaded prior to Monday poster session) +  * Project Final Report 
-    * {{:​courses:​446:​2013:​446-2013-X:final_report.pdf|Final Report}} +    * {{:​courses:​446:​2013:​446-2013-09:finalreport_last.pdf|}} 
-    * links to any appendices or other material+    * Please see Other Resources and Project Files for LapUS source code.
 ======Project Bibliography======= ======Project Bibliography=======
   * Bartosz F. Kaczmarek, S. S., Firas Petros, Quoc-Dien Trinh, Navneet Mander, Roger Chen, Mani Menon, Craig G. Rogers (2012). "​Robotic ultrasound probe for tumor identification in robotic partial nephrectomy:​ Initial series and outcomes."​ International Journal of Urology.   * Bartosz F. Kaczmarek, S. S., Firas Petros, Quoc-Dien Trinh, Navneet Mander, Roger Chen, Mani Menon, Craig G. Rogers (2012). "​Robotic ultrasound probe for tumor identification in robotic partial nephrectomy:​ Initial series and outcomes."​ International Journal of Urology.
Line 214: Line 216:
   * Joshua Leven, D. B., Rajesh Kumar, Gary Zhang, Steve Blumenkranz,​ Xiangtian (Donald) Dai, Mike Awad, Gregory D. Hager, Mike Marohn, Mike Choti, Chris Hasser, Russell H. Taylor DaVinci Canvas: A Telerobotic Surgical System with Integrated, Robot-Assisted,​ Laparoscopic Ultrasound Capability, The Johns Hopkins University; Intuitive Surgical, Inc.   * Joshua Leven, D. B., Rajesh Kumar, Gary Zhang, Steve Blumenkranz,​ Xiangtian (Donald) Dai, Mike Awad, Gregory D. Hager, Mike Marohn, Mike Choti, Chris Hasser, Russell H. Taylor DaVinci Canvas: A Telerobotic Surgical System with Integrated, Robot-Assisted,​ Laparoscopic Ultrasound Capability, The Johns Hopkins University; Intuitive Surgical, Inc.
 ======Other Resources and Project Files======  ​ ======Other Resources and Project Files======  ​
-  * **IRB Protocol** {{:​courses:​446:​2013:​446-2013-09:​protocol_draft2.docx|}} +  * **Budget proposal** {{:​courses:​446:​2013:​446-2013-09:​budgetproposal.pdf|}} 
-  * **Budget Proposal** {{:​courses:​446:​2013:​446-2013-09:​budgetproposal.pdf|}} +  * **Clinical testing protocol** {{:​courses:​446:​2013:​446-2013-09:​protocol_draft2.docx|}} 
-  * **Project ​Code** {{:​courses:​446:​2013:​446-2013-09:​project9_lapus.zip|}}+  * **Confidential:​ IRB proposal** {{:​courses:​446:​2013:​446-2013-09:​confidential_eforma_na_00082654.doc|}} 
 +  * **Project ​source code** {{:​courses:​446:​2013:​446-2013-09:​project9_lapus.zip|}} 
 +  * **All project documents**{{:​courses:​446:​2013:​446-2013-09:​alldocs.zip|}}
courses/446/2013/446-2013-09/446-2013-09.1368247209.txt.gz · Last modified: 2019/08/07 12:05 (external edit)




ERC CISST    LCSR    WSE    JHU