Last updated: 5th February 2011 2:38pm

This project involves two entities - Intuitive Surgical Inc. (ISI) and ERC-CISST (Computer Integrated Surgical Systems and Technology) at Johns Hopkins University (JHU). We are developing a simulation sandbox using open source libraries which would allow developers to create new object models and applications using the daVinci System. Currently, we would be focusing our sandbox to work with the daVinci S Surgeon Console present in the Swirnow Mock Operating Room at JHU.

• Students: Anand Malpani
• Mentor(s): Anton Deguet (JHU), Prof. Taylor (JHU), Simon Dimaio (ISI), Ashwinram Suresh (ISI)

• Background
  • Currently, ISI have three generations of daVinci robotic systems viz. STANDARD, S (system in Hackerman 134 - Mock OR), and SI. Additionally, they have a Skills Simulator system which serves the purpose of reducing training time spent on the actual robot and provides the add-on features of repeatability and structured task environments.
  • Such a platform for simulating robotic surgical tasks could have wide applications in the fields of skill evaluation, surgical planning, task modeling, and so on. However, at present, the simulation tasks of the daVinci Skills Simulator are outsourced to Mimic Technologies Inc. . The Mimic Simulation engine performs the graphics and physics rendering of the environment as well as the interacting objects. This engine appears as a black-box talking to the master console of the daVinci Si System over an Ethernet and DVI connection (as seen below). The master manipulator kinematics are passed over from the console to the simulator core which performs the state management and sends the target kinematics of the virtual slaves (being simulated) to the Mimic engine. The engine in-return passes the 3D rendering of the scene over the DVI connection to the stereo viewer on the master console. Due to the black box nature of the Mimic engine the developers do not have access to the rendering pipeline, and, our project aims to overcome this problem.

• Aims
  • We want to create a simulation sandbox that uses open source dependencies (except the ISI API - available on an agreement with ISI - for talking to the robot). This platform would allow researchers to create their own simulations for a number of possible uses like training exercises, intra-operative path planning, testing learning and vision algorithms on a structured environment, etc.
  • ISI has already created a couple of exercises using H3DAPI, an open source Haptics library. At first, we would be testing their code on our daVinci Skills Simulator. Later, we would be creating the sandbox framework using the ISI-API, the CISST-SAW framework, and the H3D libraries.
  • The simulation system with our sandbox would look something like the figure below. (Notice: changes in the blocks as compared to the figure above showing the daVinci Skills Simulator system blocks).

• Significance
  • Once in place, this framework would:
    1.  Enable researchers to design their own environments (simulation tasks) using the graphics, physics models present in the sandbox
    2. Enable developers to create and add new object models to the existing library of models and develop newer applications using the same
    3. Allow simulating patient specific anatomical data to enable procedural planning - e.g. a surgeon can create a road-map of the procedure to be performed and look at any difficulty arising and perform the actual procedure in the operating room after reviewing the plan.
    4. Prove to be a rapid prototyping platform for new user interfaces, surgical procedures, image guidance methods, new instruments for the robot and so on.
    5. Be the testing ground for learning approaches like modeling a surgical task performance, performing tool tracking before applying them in the real world

• Minimum: (Expected by 14th April)
  1. Extend and develop the existing CISST-SAW component for interfacing with the ISI-BBAPI to include additional functions related to setup of the console for use as a stand alone simulator console.
  2. Create a simple virtual slaves “Core” for emulating the actual “Core” of the slave side of the robot. [NEW]
  3. Create a new CISST component for interfacing to the simulation library (H3D).
  4. Demonstrate the sandbox using a basic example with basic features.
  5. Demo the
• Expected: (Expected by 28th April)
  1. Extend the virtual slave “Core” to incorporate other features of the daVinci console e.g. clutching, swapping instruments, camera control, etc.
  2. Mimic the existing sandbox sample tasks from ISI to interface to the CISST components (modifying according to available features in the sandbox)
  3. Demonstrate the modified task developed by ISI using this extended sandbox on the daVinci S console in the Mock OR.
• Maximum: (Expected by 9th May) (Will be pursued after the CIS course during summers)
  1. Develop a new application involving new models using the extended simulation sandbox, perhaps demonstrating the implications of this framework.
  2.  Extend the sandbox for including features needed in the new applications developed.

The project aims to create a framework for slave side simulation of the daVinci system using open source dependencies (except for the ISI-BBAPI [4], which is used to communicate back and forth with the robot. The ISI-BBAPI was available to us due to an agreement between ISI and JHU. This is binary package so that people do not tamper with the robot.) ISI has already created a “Sandbox” containing some object models and a couple of training tasks using H3D[5], an open source scene graph API. Currently, the sandbox communicates directly to the Skills Simulator (which is a computer as shown above in the block diagram attached to the console) using the ISI-SIM-API and can be used only with the Skills Simulator, which is compatible with the specific Si consoles only. However, we would like to build this sandbox using the CISST libraries and the SAW framework [2, 3] as it would allow us to create a component based application which can have any master device on one end sending commands driving the simulated tools on the other end.

As seen in the figure above, the PC running the simulation sandbox would be communicating back and forth with the master console via the CISST-SAW wrapper of the ISI BB API via a network connection. Additionally, the sandbox would be rendering stereo simulation (based on the H3D library) on the stereo viewer of the console via DVI cables. The flow of information within the sandbox is shown below.

• H3DViewer: This is a GUI-based loader distributed along with the H3DAPI. It is able to load X3D files containing the scene graph information.
• X3D File(s): X3D is an xml-based file format to describe the different nodes within the scene-graph. H3DAPI has X3D nodes which can be used in the x3d file to specify their relationship with the other nodes in the graph. Nodes can have different functions like graphic models, collision models, rigid body models, etc. These nodes can be routed (connected) to each other to pass and update their field members as per changes in the scene.
• Graphics Nodes: These describe the different objects in the scene and their rendering properties. X3D files for models can be developed from CAD files using some conversion softwares. These X3D files can then be used in the definition of the graphics nodes.
• Python Script: This node is to load python scripts and pass variable and make function calls to the python script.
• H3DPhysics Nodes: These nodes are present in the physics toolkit distributed with the H3DAPI. Physics rendering is achieved using these nodes like collision models, rigid body models, joints, PID control, etc.
• Virtual Slaves node: This is the custom node that we have created for our sandbox. It is used to pass information about the desired joint positions to the PID nodes which then drive the simulation. This node is also a CISST component so that it can communicate with the other components for getting updated target joint values.

The following components have been developed for this project:

• cisstDaVinci(cdv)ReadWrite (mtsTaskPeriodic):
  • Existing cisst multitask component built using isi-bbapi to communicate to and from the robot
  • Added methods to it so as to disengage the real slaves from the masters during the use of the console as a simulator.
  • Provides interfaces to send master manipulator's cartesian positions and console user events.
• mtsTeleOperation (mtsTaskPeriodic):
  • Developed to bridge between any master and simulated slave component for information passing.
  • Requires interfaces to receive information from master(s)
  • Provides interfaces to send information to slave(s) (their desired cartesian positions)
  • Manages state of the system like “Clutched”, “CameraControl”, etc.
• mtsVirtualSlavesCore (mtsTaskPeriodic):
  • Developed to emulate the core on the slave side of the daVinci robot
  • Requires interfaces to receive desired slave cartesian pose
  • Provides interfaces to send desired slave joint positions
  • Performs inverse kinematics using the cisstRobot library and known DH-parameters
• VirtualSlaves (X3DChildNode, mtsComponent):
  • Developed this H3DNode as well as mtsComponent to pass information to and from the rendering engine
  • Requires interfaces to receive the desired slave joint positions
  • Instantiates all the above classes as well as the component manager to manage these components
  • “Routes” the target joint values to the PIDCollection nodes in H3D
• ISI Nodes (H3D Nodes):
  • Developed by Ashwin for the Sim Sandbox
  • SevenDOF: data structure for storing the seven joint values
  • PIDCollection, JointPID, PrismaticJointPID, RevoluteJointPID: PID controllers moving the joints to the target positions

describe dependencies and effect on milestones and deliverables if not met

1. Initial work from ISI on the simulation platform [DONE]
   • get access to the existing sample exercise written by Ashwin (ISI) and guide on using it
2. Access to the H3DAPI svn repository [DONE]
   • sending request to the admin for the same
   • awaiting access priveleges
   • in conversation with the admin people to get the svn repository check-out
   • struggling to get the svn checkout (weird errors occurring, only partial check-outs!!)
   • network security hasn't been able to resolve this either (works fine from outside hopkins) [RESOLVED]
3. Availability of daVinci in the mock OR [DONE]
   • scheduling it in time for experimentation
   • finding the budget or funding for the daVinci usage
4. H3D (Open Haptics Library) [DONE]
   • Registration and access to the H3DAPI
5. Workstation to host the simulator and related hardware and software [DONE]
   • Have received information regarding computer specs on which ISI has tested
   • co-ordinate with Prof. Taylor to use an existing workstation with processing power and graphics for the required capabilities
   • order for hardware has been placed (02/15) awaiting delivery (expected 02/19)
   • maintaining correspondence with ISI to make sure compatibility issues do not arise
6. cisst daVinci BB API wrapper in shape [DONE]
   • contact Anton so that he remembers to do this sooner
   • setup meeting to understand the component and its properties
7. Bug fix for cisstRobot by Simon Lenard (developer) [DONE]
8. Test 3D tilepro input to the console [PENDING]

• Below is a rough timeline for achieving the milestones mentioned in the first column and marked by the red dots.

1. Milestone name: Test ISI examples from existing sandbox on Skills Simulator
   • Planned Date: 19th Feb
   • Expected Date: 25th Feb
   • Status: Ongoing (compiling the H3D library on the simulator backpack, partial success (02/18), code revision issues (simulation works but model doesn't work perfectly)
2. Milestone name: Cleaning and updating the cisst daVinci isi-bbapi wrapper
   • Planned Date: 7th March
   • Completed Date: 21st March
   • Status: DONE
3. Milestone name: Extending the isi-bbapi component for simulation
   • Planned Date: 14th April
   • Expected Date: 23rd April
   • Status: DONE
4. Milestone name: Implementing the virtual slave “Core” and tele-operation CISST component with basic features
   • Planned Date: 14th April
   • Expected Date: 5th May
   • Status: IN PROGRESS
5. Milestone name: Create and test a simple simulation on the daVinci S console
   • Planned Date: 21st April
   • Expected Date: 7th May
   • Status: Futuristic
6. Milestone name: Extend CISST components for additional features
   • Planned Date: 28th April
   • Expected Date: 7th May
   • Status: IN PROGRESS
7. Milestone name: Mimic the ISI sandbox example to use with the extended sandbox
   • Planned Date: 5th May
   • Expected Date: 13th May
   • Status: Futuristic
8. Milestone name: Develop additional models for application related to port placement, etc.
   • Planned Date: 25th April
   • Expected Date: xxx
   • Status: REMOVED
9. Milestone name: Extend the sandbox further for these applications and test on the S console
   • Planned Date: 10th May
   • Expected Date: xxx
   • Status: REMOVED

• All of the following documents can be found in this zip file
• Project Plan (due Thursday, February 14th)
  • Project plan presentation
  • Project plan proposal
• Project Background Reading
  • See Bibliography below for links.
• Project Checkpoint
  • Project Checkpoint presentation (pdf)
• Project Mini-Checkpoint
  • Project Mini Checkpoint presentation (pdf)
• Paper Seminar Presentations
  • Seminar Critical Review (pdf)
  • Seminar Presentation (pdf)
  • Papers reviewed:
    • Robotic Surgery Training with Commercially Available Simulation Systems in 2011: A Current Review and Practice Pattern Survey from the Society of Urologic Robotic Surgeons
    • Current status of validation for robotic surgery simulators – a systematic review
• Project Teaser Slide
  • Teaser PDF file
  • Teaser PPT file
• Project Final Presentation
  • PDF of Poster
• Project Final Report
  • Final Report

1. S. DiMaio and C. Hasser, The da Vinci Research Interface, MICCAI Workshop on Systems and Arch. for Computer Assisted Interventions, Sep. 2008
2. A. Deguet and R. Kumar and R. Taylor and P. Kazanzides, The cisst libraries for computer assisted intervention systems, MICCAI Workshop on Systems and Arch. for Computer Assisted Interventions, Sep. 2008
3. V. Balazs and D. Simon and D. Anton and K. Peter and K. Rajesh and H. Christopher and T. Russell, The Surgical Assistant Workstation, MICCAI Workshop on Systems and Arch. for Computer Assisted Interventions, Sep. 2008
4. Intuitive Surgical Inc., ISI API User Guide
5. Intuitive Surgical Inc., daVinci S System User Manual
6. Intuitive Surgical Inc., daVinci Skills Simulator Manual

• Source Code for the project
  • Under version control: SVN repository for source code
• H3DAPI
  • Download page
  • Latest source
  • Wiki page
• H3DPhysics Toolkit - H3D implementation on common physics engines
  • Download page
  • Latest source
  • Wiki page
• CISST: libraries for computer assisted interventions and SAW: Surgical Assistant Workstation, platform to enhance a surgeon's capabilities
  • Home Page
  • Latest Source