Surgical phase recognition plays a crucial role in the era of digitized surgery. Deep learning solutions have seen great success in endoscopic surgeries. Currently, no prior work has investigated its application in skull-base surgery (Cortical Mastoidectomy). In this project, we will benchmark existing DL solutions and create an innovative DL segmentation algorithm in skull-based surgery.

• Students: Xucheng Ma, Xiaorui Zhang, Wenkai Luo
• Mentor(s): Max Li, Danielle Trakimas, Dr.Francis Creighton, Prof.Mathias Unberath, Prof.Russell Taylor

Purely vision-based recognition has been proven to be successful in endoscopies surgeries. Spatial and temporal features are proven to be crucial and efficient in tackling the surgical phase segmentation task. Many DL networks were proposed to extract those features and achieve automatic phase segmentation effectively. An automatic surgical phase recognition has numerous potential medical applications, such as automatic indexing of surgical video databases and real-time operating room scheduling optimization. It’s also a Foundation of an intelligent context-aware system, which facilitates surgery monitoring, surgical protocol extraction, and decision support.

• Minimum:Benchmarking Existing DL Solutions​ (Expected by April 5th)
  1. New dataset from cortical mastoidectomy videos
  2. Benchmark three existing methods that are designed for a similar task (Cholecystectomy).​
  3. A well-documented benchmarking code base that can be easily reused to test out any model for our dataset.​

• Expected: Design a DL Solution for Skull-base Surgery​ (Expected by End of Spring Semester 22)
  1. Experiment results analysis of the weakness of benchmarked models.​
  2. Redesign a new model that achieved 85% accuracy, 80% recall, and precision.​

• Maximum: (Expected by End Of Summer 22)
  1. Conference paper​

Since the surgical phase segmentation is a sequential problem rather than a per-frame classification, the proposed deep learning neural network needs to extract spatiotemporal features.

• Spatial feature extraction:
  • One of the essential parts of the architecture is the spatial feature extractor, which extracts the feature from the frames and converts them into an abstract representation format. Based on the idea of transfer learning, the pre-trained convolutional neural network(CNN) models have been proved to be stat-of-the-art on many computer vision tasks. Furthermore, the pre-trained model can extract more specific features with the fine-tuning process using the mastoidectomy dataset.
• Spatiotemporal model:
  • The most crucial part of the architecture is the spatiotemporal model, which is to capture the temporal patterns from the extracted per-frame spatial features from video. The temporal patterns are essential to make a correct prediction since the patterns can provide clues of surgery environment change and the instrument motion. Recurrent neural networks such as LSTM, temporal convolution neural network (TCN), and transformer are promising architectures for capturing the temporal pattern. However, some research argues that LSTM can only capture short-term information while long-term context might be beneficial for accurate segmentation. All of them will be implemented on the mastoidectomy dataset in our project.
• High-level Classifier:
  • The fully connected neural network(FCNN) will then be trained with the ground truths to make the phase decision based on the extracted spatiotemporal feature since it can approximate any arbitrary function. One crucial concern is the over-fitting issue, which can be compensated by different training skills such as drop-out and batch-normalization.

1. Milestone name: Proposal and Plan
   • Planned Date: 02/10
   • Expected Date: 02/10
   • Status: 100%
2. Milestone name: Sample Dataset
   • Planned Date: 02/20
   • Expected Date: 02/20
   • Status: 100%
3. Milestone name: Fully Annotated Dataset
   • Planned Date: 03/17
   • Expected Date: 04/17
   • Status: 12/15 80%
4. Milestone name: Minimum Deliverables
   • Planned Date: 03/27
   • Expected Date: 03/27
   • Status: 100%
5. Milestone name: Initial Network Design
   • Planned Date: 04/22
   • Expected Date: 04/22
   • Status: 30%
6. Milestone name: Expected Deliverables
   • Planned Date: 04/29
   • Expected Date: 04/29
   • Status: 60%
7. Milestone name: Final Presentation
   • Planned Date: 05/02
   • Expected Date: 05/02
   • Status: 0%

• Project Plan
  • Project plan presentation
  • Project plan proposal
• Project Background Reading
  • See Bibliography below for links.
• Project Checkpoint
  • Project checkpoint presentation
• Paper Seminar Presentations
  • Seminar presentation
  • Critical Review
  • Papers:
    • SV-RCNet: Workflow Recognition From Surgical Videos Using Recurrent Convolutional Network
    • TeCNO: Surgical Phase Recognition with Multi-Stage Temporal Convolutional Networks
    • Trans-SVNet: Accurate Phase Recognition from Surgical Videos via Hybrid Embedding Aggregation Transformer
• Project Final Presentation
  • Final poster
• Project Final Report Final Report
• links to any appendices or other material

1. Andru Putra Twinanda, Sherif Shehata, Didier Mutter, Jacques Marescaux, Michel de Mathelin, and Nicolas Padoy. Endonet: A deep architecture for recognition tasks on laparoscopic videos. CoRR, abs/1602.03012, 2016.​
2. Xiaojie Gao, Yueming Jin, Yong-Hao Long, Qi Dou, and Pheng-Ann Heng. Trans-svnet: Accurate phase recognition from surgical videos via hybrid embedding aggregation transformer.CoRR, abs/2103.09712, 2021.​
3. Carly Garrow, Karl-Friedrich Kowalewski, Linhong Li, Martin Wagner, Mona Schmidt, Sandy Engelhardt, Daniel Hashimoto, Hannes Kenngott, Sebastian Bodenstedt, Stefanie Speidel, Beat Müller, and Felix Nickel. Machine learning for surgical phase recognition a systematic review. Annals of Surgery, Publish Ahead of Print, 11 2020.​
4. Henry Lin, Izhak Shafran, David Yuh, and Gregory Hager. Towards automatic skill evaluation: Detection and segmentation of robot-assisted surgical motions. Computer aided surgery : official journal of the International Society for Computer Aided Surgery, 11:220–30, 10 2006.​
5. Tobias Blum, Hubertus Feussner, and Nassir Navab. Modeling and segmentation of surgical workflow from laparoscopic video. volume 13, pages 400–7, 09 2010.​
6. Joonmyeong Choi, Sungman Cho, Jong Chung, and Namkug Kim. Video recognition of simple mastoidectomy using convolutional neural nets: Detection and segmentation of surgical tools and anatomic regions. Computer Methods and Programs in Biomedicine, 208:106251, 06 2021.​
7. Colin Lea, Joon Hyuck Choi, Austin Reiter, and G Hager. Surgical phase recognition: from instrumented ORs to hospitals around the world.​
8. Manish Sahu, Angelika Szengel, Anirban Mukhopadhyay, and Stefan Zachow. Surgical phase recognition by learning phase transitions. Current Directions in Biomedical Engineering, 6(1):20200037, 2020.4​
9. Lea, Colin, Austin Reiter, René Vidal, and Gregory D. Hager. “Segmental spatiotemporal cnns for fine-grained action segmentation.” In European Conference on Computer Vision, pp. 36-52. Springer, Cham, 2016.​
10. Jin, Yueming, Yonghao Long, Cheng Chen, Zixu Zhao, Qi Dou, and Pheng-Ann Heng. “Temporal memory relation network for workflow recognition from surgical video.” IEEE Transactions on Medical Imaging 40, no. 7 (2021): 1911-1923.

• Gantt Chart
• GitHub Repository