Last updated: 2/20/2022

Our goal is to develop an OpenAI Gym compatible interface for the Surgical Robotics Challenge (SRC) environment with efficient, accurate reinforcement learning (RL) algorithms. Currently, no robust framework exists for RL tasks in surgical robotics environments. We propose an OpenAI Gym environment based on a autonomous robotic suturing environment with benchmark algorithms to pave the way for future surgical automation.

• Students:
  1. Walee Attia (CS & BME '23)
  2. Jocelyn Hsu (CS & BME '23)
  3. Jihoon Kim (CS & Cog. Sci '23)
• Mentor(s):
  1. Anqi Liu (CS Assistant Prof.)
  2. Adnan Munawar (Assistant Research Scientist)
  3. Manish Sahu (Postdoc researcher)
  4. Peter Kazanzides (CS Research Prof.)

Current RL techniques have seen significant progress in the robotics domain; mostly due to open-source RL frameworks and an emerging corpus of environments to train autonomous robotic simulations, such as OpenAI Gym. However, there exist a lack of platforms which offer environments conducive to medical robotics.

The Surgical Robotics Challenge (SRC) is a simulation platform to develop algorithms to address various questions in surgical robotics automation with:

• Two 7-DOF instrument arms based on the da Vinci Surgical System large needle driver
• Controllable camera based on the da Vinci Endoscopic Camera Manipulator
• Suturing phantom
• Needle with a suture

Specifically, the 2021-2022 SRC consists of three challenges:

1. Challenge 1: Find the needle
2. Challenge 2: Grasp needle and drive through tissue
3. Challenge 3: Suture the phantom

Our aim is to make the SRC environment compatible with the OpenAI Gym API for RL research and provide benchmark RL algorithms for the 2021-2022 SRC challenge tasks. Generally, we seek to create an OpenAI Gym environment with different rewards for each task (e.g. grasping a needle, inserting the needle, and guiding the needle to the target exit hole) and use state-of-the-art (SOTA) RL methods to learn the policy as a benchmark for future surgical automation research. Having an open-source environment on OpenAI Gym will produce wide applicability and drive more innovation and attention in autonomous surgical robotics systems.

• Minimum: Open-source OpenAI Gym environment to train RL algorithms (Expected by 4/12/2023)
  1. Transfer of robotics surgical challenge to OpenAI Gym environment (sandbox env)
  2. Core functionality for OpenAI Gym: `make()`, `reset()`, `step()`, `render()`
  3. Environments built to accomplish SRC Challenge #2: grasp needle and drive through tissue
  4. Documentation of our work through entire development pipeline
  5. Compatibility with dVRK tools in the LCSR lab

• Expected: Benchmarked performances of the SOTA RL algorithms (Expected by 4/28/2023)
  1. Literature review of SOTA RL Algorithms
  2. RL algorithm training for automated suturing task (SRC #2)
  3. Performance evaluation (accuracy and efficiency) of RL algorithms & 2021-2022 SRC winners

• Maximum: Submission as NeurIPS (Datasets and Benchmark track), additional SRC tasks (Expected beginning of June)
  1. Paper for NeurIPS conference (database and benchmarking track): 6/1/23 for abstract, 6/7 for full paper

Our goal is to develop an OpenAI gym environment to automate the suturing process with reinforcement learning. Developing the environment and RL algorithm in accordance with guidelines of Surgical Robotics Challenge #2 (grasping and driving the needle), we will ensure compatibility with the AMBF-RL toolkit environment. Ultimately, we hope to submit a paper to the NeurIPS conference in June 2023.

To address the suturing task, we will develop an environment with three sub-tasks: grasp, insert, and target. Given the robotic arm location and needle location, the `grasp` task will navigate the robotic arm towards the needle at an ideal angle and pick up the needle. The `insert` task will handle the needle positioning and puncturing through the starting point of the tissue, and the `target` task will guide the needle through the tissue and pull the needle out of the tissue.

Our software design requirements and instructions to run our code can be found here.

For a list of technical issues we've encountered and their resolutions, please refer to the troubleshooting document.

1. Milestone name: Set up OpenAI Gym, ROS, AMBF
   • Planned Date: 2/17/23
   • Expected Date: 2/23/2023
   • Status: Completed
2. Milestone name: Literature review of SRC, AMBF
   • Planned Date: 2/20/23
   • Expected Date: 2/23/2023
   • Status: Completed
3. Milestone name: Completion of RL Online Course
   • Planned Date: 3/3/23
   • Expected Date: 3/6/2023
   • Status: Completed
4. Milestone name: Transfer SRCEnv to OpenAI Gym - make(), render()
   • Planned Date: 3/11/23
   • Expected Date: 3/15/2023
   • Status: Completed
5. Milestone name: Transfer SRCEnv to OpenAI Gym - step(), reset()
   • Planned Date: 3/11/23
   • Expected Date: 3/15/2023
   • Status: Completed
6. Milestone name: Develop reward functions for needle grasp, insert, target (SRC #2)
   • Planned Date: 3/28/2023
   • Expected Date: 3/31/2023
   • Status: Completed
7. Milestone name: Test and modify SRC #2 reward functions exhaustively
   • Planned Date: 4/19/2023
   • Expected Date: 4/22/2023
   • Status: In-progress 50%
8. Milestone name: Train RL algorithm for SRC #2
   • Planned Date: 4/26/2023
   • Expected Date: 4/29/2023
   • Status: In-progress 50%
9. Milestone name: Evaluate RL performance w/ SRC 2022 winners
   • Planned Date: 5/1/2023
   • Expected Date: 5/5/2023
   • Status: Not started
10. Milestone name: Write abstract for NeurIPS submission
    • Planned Date: 5/12/2023
    • Expected Date: 6/1/2023
    • Status: Not started
11. Milestone name: Write paper for NeurIPS submission
    • Planned Date: 5/12/2023
    • Expected Date: 6/7/2023
    • Status: Not started

• Project Plan
  • Project plan presentation
  • Project plan proposal
• Project Background Reading
  • Background reading presentation
  • Background reading report
  • Paper #1: Hindsight experience replay
  • Paper #2: AMBF-RL: A real-time simulation based reinforcement learning toolkit for medical robotics
  • Paper #3: Open-sourced reinforcement learning environments for surgical robotics
• Project Checkpoint
  • Project checkpoint presentation
• Paper Seminar Presentations
  • Project teaser presentation
• Project Final Presentation
  • PDF of Poster
• Project Final Report
  • Final Report
  • Mentor feedback report
• Documentation
  • Software design specifications
  • Troubleshooting document
• GitHub
  • GitHub Repo

[1] Introduction to reinforcement learning with David Silver. DeepMind. (n.d.). Retrieved February 19, 2023, from https://www.deepmind.com/learning-resources/introduction-to-reinforcement-learning-with-david-silver
[2] Richter, F., Orosco, R. K., & Yip, M. C. (2019). Open-sourced reinforcement learning environments for surgical robotics. arXiv preprint arXiv:1903.02090.
[3] Medical Open Network for Artificial Intelligence. MONAI. (n.d.). Retrieved February 19, 2023, from https://monai.io/index.html
[4] 2021-2022 AccelNet Surgical Robotics Challenge (online). Collaborative Robotics Toolkit (CRTK). Retrieved February 19, 2023, from https://collaborative-robotics.github.io/surgical-robotics-challenge/challenge-2021.html
[5] Gymnasium documentation. Basic Usage. (n.d.). Retrieved February 19, 2023, from https://gymnasium.farama.org/content/basic_usage/
[6] NeurIPS 2023. Neural Information Processing Systems Foundation. (n.d.). Retrieved February 19, 2023, from https://nips.cc/Conferences/2023

Here give list of other project files (e.g., source code) associated with the project. If these are online give a link to an appropriate external repository or to uploaded media files under this name space (2023-20).