======Intra-Ocular Micro-Manipulation====== **Last updated: Feb 18** ======Summary====== Our goal is to control a 5 DoF Steady Hand Eye Robot ("SHER") and a 2 DoF I2RIS distal-end robot together as a single entity. Whereas currently the system is controlled with the surgeon manually adjusting the I2RIS and moving the SHER at the attachment point, we aim to create an control algorithm that creates an optimized trajectory and is controlled from a Phantom Omni device. FBG force sensors on the end will feedback force on the tool tip, so that constraints on force can be enforced to minimize patient trauma. * **Students:** Yishun Zhou, Kaiyu Shi * **Mentor(s):** Dr. Gang Li, Prof. Iordachita {{:courses:456:2021:projects:456-2021-07:screenshot_2021-02-18_160609.png?400|}} ======Background, Specific Aims, and Significance====== {{:courses:456:2021:projects:456-2021-07:vitreoretinal_surgery.png?400|}} Vitreoretinal surgery requires advanced surgical skills at or over the limit of surgeons’ physiological capabilities. The surgery is performed in a confined intraocular space with restricted free motion of surgical tools. The forces exerted between the ophthalmic tools and eye tissue are often well below human sensory thresholds. For instance, the epiretinal membrane (ERM) peeling surgery, where a micron-scale membrane on the retinal surface is removed, require the forces exerted by the surgeon to be less than 7.5 mN. And a force above the 7.5 mN could cause irreversible damage to the retina. This project aims to move towards solving the clinical challenges mentioned above by providing the surgeons a cooperatively controlled robotic system with 2 DOF snake-like manipulator and a 5 DOF Steady Hand Eye Robot that has the capabilities of 1) tremor-free tool manipulation, 2) increased dexterity to ensure safe access to target from suitable directions, and 3) force sensing at the tool tip and sclerotomy. ======Deliverables====== * **Minimum:** (Planned for 3/29, Expected by 4/9) - A report that includes calculation of forward kinematics, inverse kinematics, and jacobian of the combined system (Planned for 3/9, Expected by 4/9, the current one is missing optimization of end effector rotation) {{ :courses:456:2021:projects:456-2021-07:kinematics.pdf |}} - A report on the force distribution analysis (3/15) {{ :courses:456:2021:projects:456-2021-07:force_distribution_model.pdf |}} - A schematic of the control algorithm design (Planned for 3/29, Expected by 4/9) * **Expected:** (Expected by 5/5) - A functioning gazebo simulation in which the end-effector of the simulated eye robot follow several optimized trajectories (4/5) - A report that summarizes the control algorithm, and an evaluation of the simulated system (5/5) * **Maximum:** (Expected by 5/5) - Implemented control system on real hardware (5/5) - Documentation of implementation (5/5) ======Technical Approach====== ===== Simulation===== Simulation will be utilized in the development of the control algorithm, given the benefits of ease of access and risk mitigation. The robot system will be recreated in Gazebo, which allows for control via ROS nodes just like the real system. To create the meshes for a Gazebo simulation, the existing CAD files, in both Creo and SolidWorks formats, are converted to a neutral STEP format then into the STL format. To construct the segments of the distal (snake) end in the simulation, an invisible link will be used to connect the two segments of each real joint, with two virtual joints at the center of invisible circles that are concentric to the segment curves. The amount of rotation will be constrained such that both sides are the same; so the surfaces are effectively in non-slip contact. {{:courses:456:2021:projects:456-2021-07:picture3.png?350 |}} {{:courses:456:2021:projects:456-2021-07:picture4.png?350|}} ===== Kinematics and Force Distribution Analysis===== Prior work focused on the control of SHER and improvement of snake robot, which were validated by experiments separately. Although they were assembled mechanically, the controls of them are kept separate. The force distribution model of the surgical tool was also based on the assumption that the FBG sensors are embedded on a rigid surgical tool with no bending. However, the tool tip force and sclera contact force should take into consideration the bending angle of the snake robot. And an integrated control of SHER and I2RIS could further simplify the user interface by combining the control of SHER and I2RIS, which also allows the surgery to be performed by the surgeon on a haptic device such as Phantom Omni. Both the kinematics model and force distribution model of the system are needed to control the integrated system of SHER and I2RIS. In this work, forward kinematics, inverse kinematics and jacobians of the integrated system will be calculated from existing kinematic model of SHER, geometric model of I2RIS and CAD model that has measurements of the robot. FEA on I2RIS, with consideration of its pose, will be conducted to map the FBG sensor readings to tool tip force and sclera contact force. Calibration of the kinematics model and force distribution model will be performed to eliminate errors between real robot and CAD model. ===== Control Algorithm ===== {{:courses:456:2021:projects:456-2021-07:picture5.png?400 |}} {{:courses:456:2021:projects:456-2021-07:proposed_control.png?400 |}} As mentioned above, SHER and I2RIS are separately controlled. The surgeon needs to provide two different inputs (force and joystick/tactile switch) to the robots, which might complicate the user’s experience of the integrated eye robot, especially when the tool is inserted into a restrictive intraocular space with limited, and risky, free movement of the tool. In this work, we will design a constrained control algorithm that integrates the control of SHER and I2RIS as shown in Figure 7. The hand-over-hand control of SHER will be employed until the surgical tool is inserted into the eye, at which point the integrated control algorithm takes over. The input provided by the surgeon along with the force information from FBG sensors control the pose, velocity and forces of the integrated eye robot. This control algorithm is key in keeping the tool tip force and sclera contact force within safe limits, which prevents irreversible damage during the surgery. ===== Software ===== CAD modeling will be done in SolidWorks. Kinematics modeling will be done in MATLAB. Simulation will be done with Gazebo in conduction with nodes running on the ROS (melodic) framework, to maintain compatibility with existing SHER ROS code. ======Dependencies====== {{:courses:456:2021:projects:456-2021-07:dependencies.png?400 |}} ======Milestones and Status ====== - Milestone name: Setup Robot Model in Gazebo * Planned Date: 3/4 * Expected Date: 3/4 * Status: 100% - Milestone name: Kinematics Model * Planned Date: 3/8 * Expected Date: 4/9 * Status: 95% - Milestone name: Force Distribution Analysis * Planned Date: 3/15 * Expected Date: 3/15 * Status: 100% - Milestone name: Control Algorithm Psuedocode * Planned Date: 3/20 * Expected Date: 4/10 * Status: 50% - Milestone name: Control Algorithm Testing and Validation in Simulation * Planned Date: 4/5 * Expected Date: 4/20 * Status: 0% - Milestone name: Control Algorithm Testing and Validation on Real Robot * Planned Date: 5/5 * Expected Date: 5/5 * Status: 0% {{ :courses:456:2021:projects:456-2021-07:project_poster.pdf |}} ======Reports and presentations====== * Project Plan * {{:courses:456:2021:projects:456-2021-07:project_plan.pdf| Project plan presentation}} * {{ :courses:456:2021:projects:456-2021-07:project_proposal_07_intraocular_manipulation.pdf |Project plan proposal}} * Project Background Reading * See Bibliography below for links. * Project Checkpoint * {{ :courses:456:2021:projects:456-2021-07:project_checkpoint.pdf | Project checkpoint presentation}} * Paper Seminar Presentations * {{ :courses:456:2021:projects:456-2021-07:a_stiffness-adjustable_hyperredundant_manipulator.pdf |Yishun Zhou Seminar Paper}} * {{ :courses:456:2021:projects:456-2021-07:seminar_presentation.pdf |Yishun Zhou Seminar Presentation}} * {{ :courses:456:2021:projects:456-2021-07:paper_critique.pdf |Yishun Zhou Paper Critique}} * {{ :courses:456:2021:projects:456-2021-07:kaiyu_shi_paper.pdf |Kaiyu Shi Seminar Paper}} * {{ :courses:456:2021:projects:456-2021-07:paper_presentation_kaiyu.pdf |Kaiyu Shi Seminar Presentation}} * {{ :courses:456:2021:projects:456-2021-07:kaiyu_paper_report.pdf |Kaiyu Shi Paper Critique}} * Project Final Teaser Slides * {{ :courses:456:2021:projects:456-2021-07:group_7_teaser_slide.pdf |}} * Project Final Presentation * {{ :courses:456:2021:projects:456-2021-07:project_poster.pdf |Final Project Presentation}} * Project Final Report * {{ :courses:456:2021:projects:456-2021-07:final_report.pdf | Project Final Report}} * links to any appendices or other material ======Project Bibliography======= * Jinno, Makoto, and Iulian Iordachita. “Improved Integrated Robotic Intraocular Snake*.” 2020 International Symposium on Medical Robotics (ISMR), 2020, doi:10.1109/ismr48331.2020.9312927. * P. Gupta, P. Jensen, and E. de Juan, “Surgical forces and tactile perception during retinal microsurgery,” in International Conference on Medical Image Computing and Computer Assisted Intervention, vol. 1679, 1999, pp. 1218–1225 * Makoto Jinno, Gang Li, Niravkumar Patel, Iulian Iordachita, “An Integrated High-dexterity Cooperative Robotic Assistant for Intraocular Micromanipulation”, 2021., Kokushikan University * He, Xingchi. Force Sensing Augmented Robotic Assistance for Retinal Microsurgery. 2015. Johns Hopkins U, PhD dissertation. *3D Systems. “Touch.” 3D Systems, 4 June 2020, www.3dsystems.com/haptics-devices/touch. * Üneri, Ali, Marcin A. Balicki, James Handa, Peter Gehlbach, Russell H. Taylor, and Iulian Iordachita. "New steady-hand eye robot with micro-force sensing for vitreoretinal surgery." In 2010 3rd IEEE RAS & EMBS International Conference on Biomedical Robotics and Biomechatronics, pp. 814-819. IEEE, 2010. ======Other Resources and Project Files====== Here give list of other project files (e.g., source code) associated with the project. If these are online give a link to an appropriate external repository or to uploaded media files under this name space (2021-07).