| Both sides previous revisionPrevious revisionNext revision | Previous revision |
| courses:456:2022:projects:456-2022-20:project-20 [2022/05/11 19:33] – [Summary] hwei15 | courses:456:2022:projects:456-2022-20:project-20 [2022/05/16 20:16] (current) – [Reports and presentations] hwei15 |
|---|
| ======Mixed Reality Surgical Team Training====== | ======Mixed Reality Surgical Team Training====== |
| **Last updated: Mar 28th** | |
| | Team member: Haochen Wei |
| | |
| | Mentor: Peter Kazanzides, Dayeon Kim |
| | |
| | **Last updated: May 11st** |
| |
| |
| ======Summary====== | ======Summary====== |
| Laparoscopic surgery has many advantages over traditional open surgery. During laparoscopic surgery, an endoscope was inserted through a keyhole on the patient’s abdominal wall. The real-time image captured by the endoscope is displayed on a monitor, and based on the visual guidance, the surgeon manipulates laparoscopic instruments to perform the procedure. | We propose a solution to provide an endoscopic surgery training simulator for training rookie surgeons and solve the problem of limited training sources. |
| | |
| Due to the difficulty, it required lots of training from a trainee to a qualified surgeon. Recently several Augmented Reality based surgical simulator has been developed, but most of them only considered several specific types of procedure and ignore the robot-assist surgery system integration. | The simulator will locate the endoscopic tools via stereo camera and render the model of tools on the screen. The tools are overlaid on the virtual simulation environment to create a training setup. The whole system behaves just like "Augment Reality", but the output is on a 2D screen just like the regular endoscopic surgery. |
| |
| We use the Augments reality technic to create a simulation environment to train the surgeon. The simulator will locate the position and pose of the tools and will overlay tools on the virtual anatomy structure. Furthermore, Da Vinci also required some assistants to hold tools near the bed, and their technic is like regular endoscopic surgery, thus the system can be combined with Da Vinci System to use the existed anatomy simulator for bedside assistant training. And can be integrated with the Da Vinci System build-in simulator to train the whole team together. | The simulator can locate the tools at millimeter level accuracy and can maintain the performance of at least 30FPS constantly within a normal computer. |
| |
| * **Students:** Haochen Wei | * **Students:** Haochen Wei |
| |
| ======Background, Specific Aims, and Significance====== | ======Background, Specific Aims, and Significance====== |
| Endoscopic surgery is important in modern medicine, but it has one major drawback: difficulty to train due to the limitation of patient resources, and safety issues. | Laparoscopic surgery has many advantages over traditional open surgery. During laparoscopic surgery, an endoscope was inserted through a keyhole on the patient’s abdominal wall. The real-time image captured by the endoscope is displayed on a monitor, and based on the visual guidance, the surgeon manipulates laparoscopic instruments to perform the procedure. |
| |
| The current solution is the animal model (Pig). It is still expensive and unethical, so we want to solve the issue by implementing mixed reality surgical training. | Due to the difficulty, it required lots of training from a trainee to a qualified surgeon. Recently several Augmented Reality based surgical simulator has been developed, but most of them only considered several specific types of procedure and ignore the robot-assist surgery system integration. |
| |
| Mixed reality is not a new technology and has been explored for a long time. There are several dedicated mixed reality training tools that are used to train a specific surgery, but we want to create a more general mixed reality solution mainly focused on rendering the point cloud of tools and overlay on the human organ 3d model. This simulator may include the three following features: | We use the Augments reality technic to create a simulation environment to train the surgeon. The simulator will locate the position and pose of the tools and will overlay tools on the virtual anatomy structure. Furthermore, Da Vinci also required some assistants to hold tools near the bed, and their technic is like regular endoscopic surgery, thus the system can be combined with Da Vinci System to use the existed anatomy simulator for bedside assistant training. And can be integrated with the Da Vinci System build-in simulator to train the whole team together. |
| | |
| 1.No limitation on tool numbers. 2.No limitation on tool shapes.3.No limitation on what kinds of surgery. | |
| |
| This is a relatively huge project so the aim of this semester only focused on high-quality point cloud gathering and rendering. In the future, there are many aspects that can be dug such as a reconstructed 3d mash model instead of the rendering point cloud, interaction with the virtual organ, and so on. | |
| |
| |
| * Status: Done at 4/29 | * Status: Done at 4/29 |
| {{:courses:456:2022:projects:456-2022-20:gc.png?600|}} | {{:courses:456:2022:projects:456-2022-20:gc.png?600|}} |
| ======Reports and presentations====== | {{ :courses:456:2022:projects:456-2022-20:cis_2_project_poster_group20.pdf |}}======Reports and presentations====== |
| |
| * Project Plan | * Project Plan |
| * {{ :courses:456:2022:projects:456-2022-20:seminar_mixedreality_ppt.pdf |}} | * {{ :courses:456:2022:projects:456-2022-20:seminar_mixedreality_ppt.pdf |}} |
| * Project Final Presentation | * Project Final Presentation |
| * {{:courses:456:2022:projects:456-2022-01:final_poster_pdf.pdf|PDF of Poster}} | * {{ :courses:456:2022:projects:456-2022-20:cis_2_project_poster_group20.pdf |}} |
| * Project Final Report | * Project Final Report |
| * {{ :courses:456:2022:projects:456-2022-20:cis2_team20_finalreport.pdf |}} | * {{ :courses:456:2022:projects:456-2022-20:cis2_team20_finalreport.pdf |}} |
