=======DaVinci-Assisted Continuum Robot Navigation and Manipulation======= **Last updated: 05/10/2023** ======Summary====== * **Students:** * Heyun Wang, ME Class of '23 * Jaspor Jiang, Robotics Class of '23 * Chenhan Zhang, ME Class of '23 * **Mentors:** Dr. Mohammad Salehizadeh, Anton Deguet, and Dr. Russell Taylor {{courses:456:2023:projects:456-2023-15:davinci.png?300x250}}{{ courses:456:2023:projects:456-2023-15:ion_1.png?300x250}} \\ **Figure 1**: Commercial surgical robots: (a) 6-DOF DaVinci surgical robots[1].(b) Ion Inc.'s Endoluminal System which has a continuum robot end[2] The DaVinci robot arm is a state-of-art surgical robot that offers 6-DOF translational and rotational joint control. Tendon-driven continuum robots, on the other hand, use flexible, curving structures to navigate tight spaces and perform delicate tasks (shown in Fig. 1). They have several advantages over traditional rigid robots, including improved accuracy, flexibility, and reachability. Our proposed robot system builds on these technologies by combining the accuracy, flexibility and reachability of the dVRK robot arm with the unique capabilities of a tendon-driven continuum robot end. By doing so, we aim to build a new design of dVRK and a corresponding navigation system that overcomes the limitations of traditional rigid robots and offers improved reachability in surgical robotics systems. ======Background, Goal, and Significance====== **__Background__**\\ {{ courses:456:2023:projects:456-2023-15:IonRobot.gif?400 }} **Figure 2**: Ion Inc.’s Endoluminal System. As shown in Fig. 2, the inspiration for this project was drawn from the[[https://www.youtube.com/watch?v=0ZaobUiJhCQ&feature=youtu.be|ION surgical robot]], which has demonstrated the potential of flexible and accurate robotic catheter ends for minimally invasive solutions in lung biopsy[3]. As for the development of dVRK system, We aim to leverage the available resources of the DaVinci platform and dVRK resources at Johns Hopkins University and will receive technical training from Anton Deguet in the Laboratory for Computational Sensing and Robotics (LCSR) to develop our navigation and manipulation algorithm. Although a good dynamic model of the tendon-driven catheter exists, the limited practical application of flexible catheters on the dVRK system has created a compelling opportunity for us to fully develop our project and expand the scope of flexible catheter applications in surgical robotics. **__Goal__**\\ The primary goal of this project is to develop a continuum robot navigation and manipulation system that combines the advantages of accuracy and reachability of the dVRK robot arm and the unique flexibility of a tendon-driven continuum robot end. To achieve this goal, we will first generate a new design of dVRK with a flexible endoscope, and then build a corresponding system of navigation and remote actuation, where we will re-identify the workflow of the continuum dVRK starting from //IO// and //PID//. Throughout the whole design and development procedures, we will especially focus on adapting various surgical catheters to our continuum dVRK. The teleoperation between //ARM// and //MTM// is our highest expectation in the end. **__Significance__**\\ As a comprehensive project with novel design, where we make our own continuum dVRK and build the corresponding navigation system from the bottom. The project could lead to a further enhancement of dVRK platform The significance of our proposed system also lies in its potential impact on the field of surgical robotics. By offering better reachability and dexterity in delicate surgical procedures, our proposed DaVinci-assisted continuum robot navigation and manipulation system has the potential to improve surgical robotics and patient outcomes. ======Updated Milestones and status====== {{ courses:456:2023:projects:456-2023-15:milestone.png?700 }} ======Technical Approach====== **__Work Flow__**\\ In this section, the technical approach for developing a new continuum robot end effector for surgical applications will be presented. The proposed workflow consists of **Mechanical Design,** **Experiments and Parameters measurements,** **Code development,** and **Potential Applications**, as shown in Fig 3.\\ {{ courses:456:2023:projects:456-2023-15:workflow.png?580 }}\\ **Figure 3**: Updated Workflow. {{ courses:456:2023:projects:456-2023-15:system.png?380 }}\\ **Figure 4**: Updated System design. **__Mechanical Design__**\\ In this section, the focus is on designing a new continuum robot end effector that incorporates the desirable characteristics of the available endoscope catheter and Acunav catheter. The team will compare the available actuators, mechanisms, and materials, and identify suitable components for the new end effector. Once the design is complete, the Acunav catheter will be assembled into the dvrk baseplate based on the endoscope mechanism. Basic actuation on dvrk will then be achieved to ensure that the hardware design is functional. As shown below, We first design an adaptor and two retainers to centralize and fix the AcuNav catheter to the tube. Then we will use a 3D printer to print them out and assemble them together a few weeks later. The second figure shows the mechanism of how we will assemble the tendon of AcuNav to the dVRK baseplate. {{ courses:456:2023:projects:456-2023-15:adaptor.png?480 }}\\ **Figure 4**: AcuNav-dVRK adaptor. {{ courses:456:2023:projects:456-2023-15:pulleys.png?680 }}\\ **Figure 5**: Motion control principle of the Acunav catheter. Here is the break view of our final integrated AcuNav-dVRK continuum robot. {{ courses:456:2023:projects:456-2023-15:acunav_latest.png?150 }}\\ **Figure 6**: AcuNav-dVRK continuum robot assembly. **__Experiments and Parameters measurements__** \\ After that, we took many experiments. We first took the joint limitation experiment to avoid interference between our robot and the environment for safety considerations. Then we also did many measurements for rough kinematics values of the endoscope catheter by caliper and encoders. For the next step, we will measure those values of the AcuNav-dVRK robot via the EM tracking system for more precise manipulation and navigation. Considering the non-disclosure police, we will only show the model of our joint mapping functions and D-H parameters here. {{ courses:456:2023:projects:456-2023-15:jointmapping.png?680 }}\\ **Figure 7**: Values measured from experiments for these equations **__Code development__** \\ The third phase mainly focused on developing forward and inverse kinematics code via Python. At the beginning of this phase, the team learned how to actuate the robot arm and catheter using simple Python commands and detect the actuation via the GUI of dVRK. In order to manipulate and navigate the continuum robot catheter on the patient side manipulator, the joint mapping function, forward, and inverse kinematics model was developed. The team calculate the D-H parameters based on the previous experiments and then accomplished all of our models in codes by Python. {{ courses:456:2023:projects:456-2023-15:arc.png?210 }}\\ **Figure 8**: Overall architecture of dVRK[4] **__Application__** \\ In this section, the focus is on identifying a suitable surgical application for the new continuum robot end effector. The team will evaluate different surgical procedures and select one that requires the use of a continuum robot catheter. They will then test the new end effector and the algorithms on a phantom or a simulated environment to ensure that the design is functional and efficient. The team will also assess the safety and efficacy of the new device and make any necessary modifications before proceeding to clinical trials. The goal is to develop a new surgical tool that will improve patient outcomes and advance the field of minimally invasive surgery. ===== Results ===== The figure below is the designed and desired trajectory recorded by our EM tracking system for accuracy comparison. {{ courses:456:2023:projects:456-2023-15:track.png?280 }}\\ **Figure 9**: Track comparison. For more detailed results, please see page 12, section 3 "Experimental Evaluation and Validation" in our report. {{ courses:456:2023:projects:456-2023-15:demo_traj.mp4?500 }}\\ demo of trajectory video. ======Deliverables====== * **Minimum:** (4 weeks) - Mechanical design enhancement and design innovation (complete). - Verify the basic actuation function of dVRK with continuum end-effector through ROS and GUI (complete). * **Expected:** (4 weeks) - Obtain basic experimental results needed for identifying catheter parameters using DaVinci platform actuation (complete). - Code development and forward kinematics actuation based on catheter parameters (complete). * **Maximum:** (4 weeks) - Prototyping & testing of Acunav Catheter​ (complete). - Build EM tracking system on ROS to accurately measure the position and gesture of dVRK-AcuNav (complete). - Find a fit surgical application to our DaVinci-assisted continuum robot navigation and manipulation technology (complete) and (Optional) apply it in a phantom experiment. - (Optional) Teleoperation of the catheter using DaVinci dVRK(Optional) Teleoperation of the catheter using dVRK-AcuNav ======Dependencies====== describe dependencies and effect on milestones and deliverables if not met ^Dependency​^Need​^Contingency plan​^Planned​^Hard​^Status​^ |End-effectors for testing​|DaVinci dVRK, endoscope and Acunav catheter​| dVRK in Robotrium|​2/27|2/29|​Acquired| | Access to hardware Models​| CAD Model of Acunav catheter and DaVinci baseplate​|Manually measurement and model ​|2/27|​2/29|Detailed CAD files not found, manually measurement done​| |Software installation: ROS, Matlab, CISST library​ |Access and license​ | Use Lab Computer with pre-installed software​ |2/24​ | 2/27​ |Acquired​ ​| |Access to dVRK|​Access and training from Anton Deguet​ |N/A|​ 2/25​ |3/02|​ Acquired​| |Identification experiments​ |Two EM tracker sensors​ & Optical marker​ & Test weights​|from LCSR repository​ |4/07​| 4/17​ |Acquired| |Phantom​ | Testing​| Go without phantom testing​ |4/22​| 4/27​ |Cancelled​| ======Management====== **Weekly meetings​:** * Student team meeting: brainstorming, three times a week​ * Lab meeting: dVRK training​ * Mentor meeting: progress report in person, 3:30 PM-5:30 PM each Friday​ **Platforms:​** * Zoom, Email: communication​ * Github: codes​ * Microsoft Teams: communication, documentation​, link sharing **Updated Timeline**: {{ :courses:456:2023:projects:456-2023-15:timeline.png?600 |}} ======Acknowledgement====== We would like to thank Professor Emad Boctor for generously providing us with the Acunav ultrasound catheter and Professor Iulian Iordachita for the precious NDI EM tracking systems. We would like to extend special thanks to Anton Deguet, an expert in dVRK from Johns Hopkins University. Anton has maintained the dvrk repository and provided invaluable assistance to our team during the development of this project. ======Reports and presentations====== * Project Plan * {{:courses:456:2023:projects:456-2023-15:presentation_0223_final.pdf| Project plan presentation (pdf)}} * {{:courses:456:2023:projects:456-2023-15:presentation_0223_final.pptx| Project plan presentation (pptx)}} * {{:courses:456:2023:projects:456-2023-15:cisii_proposal_group15.pdf|Project plan proposal}} * Project Background Reading * {{:courses:456:2023:projects:456-2023-15:background_group15.pdf|Background Reading}} * See Bibliography below for links. * Project Checkpoint * {{:courses:456:2023:projects:456-2023-15:checkpoint_0406.pdf| Project checkpoint presentation (pdf)}} * Paper Seminar Presentations * {{:courses:456:2023:projects:456-2023-15:teaser_group15_apr27.pdf| Teaser presentation (pdf)}} * Project Final Presentation * {{:courses:456:2023:projects:456-2023-15:poster_group15.pdf|PDF of Poster}} * Project Final Report * {{:courses:456:2023:projects:456-2023-15:cisii_finalreport_group15.pdf|Final Report}} * {{:courses:456:2023:projects:456-2023-15:system_design.pdf| System Design (pdf)}} * {{:courses:456:2023:projects:456-2023-15:requirement.pdf | Requirements & Functional Specifications (pdf)}} * Project Mentor Report * {{:courses:456:2023:projects:456-2023-15:cis_ii_project_mentors_report.pdf|Mentor Report}} ======Project Bibliography======= * C. Song, X. Ma, X. Xia, P. W. Y. Chiu, C. C. N. Chong, and Z. Li, “A robotic flexible endoscope with shared autonomy: a study of mockup cholecystectomy,” Surg Endosc, vol. 34, no. 6, pp. 2730–2741, Jun. 2020, https://doi.org/10.1007/s00464-019-07241-8. * M. Moradi Dalvand, S. Nahavandi and R. D. Howe, "General Forward Kinematics for Tendon-Driven Continuum Robots," in IEEE Access, vol. 10, pp. 60330-60340, 2022, https://doi.org/10.1109/ACCESS.2022.3180047.​ * Degirmenci, A., Loschak, P. M., Tschabrunn, C. M., Anter, E., & Howe, R. D. (2016, May). Compensation for unconstrained catheter shaft motion in cardiac catheters. In 2016 IEEE International Conference on Robotics and Automation (ICRA) (pp. 4436-4442). IEEE. * Jessica Burgner-Kahrs, D. Caleb Rucker, and Howie Choset. 2015. Continuum Robots for Medical Applications: A Survey. Trans. Rob. 31, 6 (Dec. 2015), 1261–1280. https://doi.org/10.1109/TRO.2015.2489500 * T. Kato, I. Okumura, S. -E. Song, A. J. Golby and N. Hata, ”Tendon-Driven Continuum Robot for Endoscopic Surgery: Preclinical Development and Validation of a Tension Propagation Model,” in IEEE/ASME Transactions on Mechatronics, vol. 20, no. 5, pp. 2252-2263, Oct. 2015, https://doi.org/10.1109/TMECH.2014.2372635. * P. Kazanzides, Z. Chen, A. Deguet, G. S. Fischer, R. H. Taylor and S. P. DiMaio, "An open-source research kit for the da Vinci® Surgical System," 2014 IEEE International Conference on Robotics and Automation (ICRA), Hong Kong, China, 2014, pp. 6434-6439, https://doi.org/10.1109/ICRA.2014.6907809. * Kesner SB, Howe RD. Position Control of Motion Compensation Cardiac Catheters. IEEE Trans Robot. 2011 Jul 21;PP(99):1-11. https://doi.org/10.1109/TRO.2011.2160467. PMID: 21874124; PMCID: PMC3160644. * An Open-Source Research Kit for the da Vinci Surgical System, Kazanzides P, Chen Z, Deguet A, Fischer GS, Taylor RH, DiMaio SP, ICRA 2014, May 2014. * Cha H-J, Yi B-J, Won JY. An assembly-type master–slave catheter and guidewire driving system for vascular intervention. Proceedings of the Institution of Mechanical Engineers, Part H: Journal of Engineering in Medicine. 2017;231(1):69-79. https://doi.org/10.1177/0954411916679328 ======Other Resources and Project Files====== Here give list of other project files (e.g., source code) associated with the project. ===== Packages, codes and documentation ===== * **Github repository**: [[https://github.com/SP23-CISII-dvrk]] for available python packages, readme, documentation * available copy: {{:courses:456:2023:projects:456-2023-15:dvrk-continuum.zip|code and documentation (zip)}} * {{:courses:456:2023:projects:456-2023-15:dvrk_user_guideline.pdf| dVRK ROS and GUI Documentation (pdf)}} ===== NDI Aurora EM tracking ===== * {{:courses:456:2023:projects:456-2023-15:ddug-090-024-05-_aurora_v3.1_user_guide_rev._5.pdf| Aurora User Guideline(pdf)}} * {{:courses:456:2023:projects:456-2023-15:combined_api_sample_c_v1.9.7.zip| C++ API Sample (zip)}} ===== CAD ===== {{:courses:456:2023:projects:456-2023-15:davinci_cad.zip| related CAD file for mechanical design(zip)}}