Last updated: 2023/4/11

The project aims to implement the simulation of the combined teleoperation system in Asynchronous multi-body framework (AMBF). Simulation is one of the common means of testing robots, it enables researchers to quickly develop, validate and test control algorithms without worrying about damaging the robot. To achieve the goal, the model of the snake robot should be built in Blender according to the CAD drawing and then connected to eye robot in the simulation. Meanwhile, an AMBF plugin will be developed to manipulate the snake robot(Fig.1 C) attached to SHER(Fig.1 B) and interact with an OCT Scan of the eye.In addition, we also aim to develop a controller using haptic device(Fig.1 D) in AMBF and further test on the real robot. Figure 1. Envisioned high dexterity intraocular manipulator:(B) Steady Hand Eye Robot; (C) Integrated robotic intraocular snake robot; (D) Phantom Omni;

• Students: Xiao Wei xwei28@jhu.edu
• Mentor(s): Mojtaba Esfandiari, Adnan Munawar, Dr. Iulian Iordachita

Background
Vitreoretinal surgery is a highly delicate and difficult intraocular surgery that requires the operation on the retina deep within the eye. Such surgeries are very demanding, requiring exceptional precision to operate on micron scale targets presented by the retina while also maneuvering in a tightly constrained and fragile workspace.
The motivation to implement this teleoperation setup is multifold. First and foremost, teleoperating a device like the eye robot help the user/surgeon perform stable and meticulous manipulation, which can understandably improve surgical performance. In addition, replacing the traditional stiff end effector with a snake robot can bring benefits, such as expanding the working space, enhancing flexibility, and reducing the risk of causing injury to the eye to some extent.

Specific Aims
Therefore, the main goal of the project is to implement the simulation of the combined teleoperation system in Asynchronous multi-body framework (AMBF). The project can be divided into following tasks:

1. Learn about AMBF and ADF
2. Learn how to use AMBF Blender addon and use it to build snake robot config models according to CAD drawing
3. Develop an AMBF plugin to manipulate the snake robot attached to SHER and interact with an OCT Scan of the eye.
4. Implement a control algorithm on the AMBF model using haptic device.
5. Apply the controller on the real robot.

Importance
Simulation allows researchers to quickly develop, validate and test control algorithms without fear of damaging the robot. In this project, a robot integration system was built by simulating a snake robot to enable drilling of the eyeball scan. The functionality and performance of the snake robot can be tested in AMBF. Also, the simulation will be used to develop and test control strategies and allow for visualization of feasibility.

• Minimum: (Expected by Apr.3rd)
  1. Develop a simulation model for the Eye Robot and the Eye Snake systems in AMBF
     1. Learn AMBF + ADF + Blender addon
     2. Learn about how to use various plugin
     3. Get familiar with volume and 3D scans

• Expected: (Expected by Apr.17th)
  1. Develop and implement a control algorithm on the AMBF model using a haptic device

• Maximum: (Expected by Apr.23rd)
  1. Implement the controller on the real robots

AMBF Simulation
AMBF offers a real-time dynamic simulation of robots, free bodies, and multi-link puzzles coupled with real-time haptic interaction via several haptic devices. Compared with Gazebo, AMBF has the advantage of being able to interact with and remove parts of a volumetric model which is of vital importance in medical manipulation like drilling. In the simulation work, the following portions will be included:
1. Blender addon
The blender plugin is to ease the creation of multi-body config files that are used in AMBF. With the aid of the addon, we can model the snake robot from a given CAD drawing and configure the joints that will be used in the simulation.
2. Volumetric Drilling Plugin
The plugin is a basic one for volumetric drilling for skull base surgery with a handheld drill as shown in Fig. 2, with which we can actively modify anatomy with a virtual drill in the simulation. Although the application is not the same as this project, learning how to use this plugin can better help understand the functionality as well as the usage of the plugin. 3. Plugin related to UR5 and eye robot
There is other plugin built based on the volumetric drilling plugin to cater to a different application, i.e. a robot with a snake attached to it for spine/ortho surgery. Inspired by this plugin, we can build the combined system of eye robot and snake robot for vitreoretinal surgery.

Control Algorithm
In order to design control algorithms for snake robots, we need to have knowledge about robot kinematics. Given that modeling the kinematics of a continuum robot is not as straightforward as a traditional manipulator, our goal is to develop an experimental forward kinematics and calibrate the motion of a new snake robot (I2RIS) to produce a mapping between the execution space of the snake robot and its configuration space, and finally to its task space. This will allow for accurate forward kinematic mapping, which is essential for modeling any form of inverse kinematics.

1. Milestone name: Learn AMBF + Blender
   • Planned Date: 2/20
   • Expected Date: 2/27
   • Status: Done
2. Milestone name: Learn about different AMBF plugins
   • Planned Date: 2/27
   • Expected Date: 3/13
   • Status: Done
3. Milestone name: Incorporate eye robot and snake robot in AMBF
   • Planned Date: 3/13
   • Expected Date: 3/27
   • Status: Done
4. Milestone name: Develop a controller using haptic device
   • Planned Date: 3/27
   • Expected Date: 4/17
   • Status: Done
5. Milestone name: Implement controller on real robot
   • Planned Date: 4/11
   • Expected Date: 4/23
   • Status: /

• Project Plan
  • Project plan presentation
  • Project plan proposal
• Project Background Reading
  • See Bibliography below for links.
  • Project Background Reading
• Project Checkpoint
  • Project checkpoint presentation
• Project Final Presentation
  • PDF of Poster
• Project Final Report
  • Final Report
  • Teleoperation setupinstructions

1. Shi, K., Zhou, Y., Ebrahimi, A., Li, G., & Iordachita, I. (2022, April). Optimization-based Concurrent Control of a High Dexterity Robot for Vitreoretinal Surgery. In 2022 International Symposium on Medical Robotics (ISMR) (pp. 1-7). IEEE.
2. Munawar, A., Wang, Y., Gondokaryono, R., & Fischer, G. S. (2019, November). A real-time dynamic simulator and an associated front-end representation format for simulating complex robots and environments. In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (pp. 1875-1882). IEEE.
3. Munawar, A., Li, Z., Kunjam, P., Nagururu, N., Ding, A. S., Kazanzides, P., … & Unberath, M. (2022). Virtual reality for synergistic surgical training and data generation. Computer Methods in Biomechanics and Biomedical Engineering: Imaging & Visualization, 10(4), 366-374.
4. Roizenblatt, M., Edwards, T. L., & Gehlbach, P. L. (2018). Robot-assisted vitreoretinal surgery: current perspectives. Robotic Surgery: Research and Reviews, 1-11.
5. Fleming, I., Balicki, M., Koo, J., Iordachita, I., Mitchell, B., Handa, J., … & Taylor, R. (2008). Cooperative robot assistant for retinal microsurgery. In Medical Image Computing and Computer-Assisted Intervention–MICCAI 2008: 11th International Conference, New York, NY, USA, September 6-10, 2008, Proceedings, Part II 11 (pp. 543-550). Springer Berlin Heidelberg.
6. “The peripheral aspects of vitreoretinal surgery,” Centre for Sight, 07-Mar-2020. [Online]. Available: https://www.centreforsight.net/blog/the-peripheral-aspects-of-vitreoretinal-surgery/. [Accessed: 01-Mar-2022].

See GitHub for more details: https://github.com/ohwx/SHER_I2RIS_simulation