Last updated: 02/26/2020

Aiming to Build a robust system to assist surgeons especially novice surgeons perform at expert-level by a simple query “Do something(task) somewhere(task context) as an expert”.

• Students: Hao Ding
• Mentor(s): Mathias Unberath

Skill is among the strongest predictors for patient outcome: The higher surgeon skill, the better the outcome

Empower novice surgeons: “Translate” beginner-level commands to expert-level proficiency

Recent deep learning algorithm is powerful but vulnerable: Deep learning algorithms dominate large amount of benchmarks but suffer from generalization issues and low interpretability.

Causality brings more robustness and interpretability: With casual relations provided and causal inference mechanisms, we might be able to make the algorithm more robust and easier to interpret.

Data: Build a suitable dataset for causal inference study in surgery area.

Understandings: Explore the difference between novice- and expert- level commands.

Algorithm: Explore methods to incorporate causal inference mechanisms into deep learning algorithms of kinematic prediction for more robustness and interpretability.

• Minimum: Task-aligned JIGSAW dataset for causal analysis (Dataset and code are available in the project files part)
  1. Paired videos.
  2. Manual annotations.
  3. etc

• Expected: Statistical analysis for some properties of the trajectory (The Report is available in the project files part)
  1. Written analysis of understandings into novice- and expert-level robot command
  2. etc

• Maximum: Development and evaluation of counterfactual model (Expected by May 1st)
  1. Implementation of the Kinematic Prediction Network
  2. Implementation of the Network incorporated with Causal inference mechanism
  3. Reports of the performance of our methods
  4. etc

• Overall Approach
  • data: prerequisites of the understandings and the algorithm
  • Understandings: gain understanding from data, and help build the algorithm
  • Algorithm: Train on the data.

• Technical Approach – data
  • Fundamentals: JHU-ISI Gesture and Skill Assessment Working Set （JIGSAW): The dataset was captured using the da Vinci Surgical System from eight surgeons with different levels of skill performing five repetitions of three elementary surgical tasks on a bench-top model
    • The JIGSAWS dataset consists of three components:
      1. kinematic data
      2. video data
      3. manual annotations: gesture, skill level.

• Target: Temporally align JIGSAW samples for causal analysis
• Works to be done:
  • Manually segmenting and paring videos:
    • Annotate the start and the end frame of each motion
    • Label each motion according to the purpose and the state of the motion
  • Making paired segments into the same length:
    • Algorithm: dynamic time warping
    • Distance function: l1 norm between normalized traveled distance
• Technical Approach – understanding
• Do a statistical analysis for the motions performed by surgeons at different levels
  • Measurement: time consumed, failure rate, etc.
• Analyze the cause of failure and different performance.
• Technical Approach – Kinematics prediction network
• Architecture: Basic Transformer Network
• Input:
  1. Task information: task id and its context
  2. Expected skill level: novice- or expert- level
  3. Kinematics of previous frames
• Output: Kinematics of future frames

• Technical Approach – Causal Inference
  • Treat the assistance as a counterfactual queryWhat would the robot commands have been if, contrary to fact, the surgeon were an expert?

1. Manually segment and pair videos. (Finished on March 14th)
2. Aligning videos segments (Finished on March 20th)
3. supplementary annotation (Finished on April 4th)
4. Statistical analysis for some properties of the trajectory (Finished on April 10th)
5. Kinematic Prediction Network (Finished on May 4th)
6. Incorporate Causal inference (Put into future plan)

• Project Plan
  • Project plan presentation
  • Project plan proposal
• Project Background Reading
  • See Bibliography below for links.
• Project Checkpoint
  • Project checkpoint presentation
• Paper Seminar Presentations
  • Link to paper: Learning Invariant Representation of Tasks for Robust Surgical State Estimation.
  • literature_review-hao_ding.pdf
  • Hao Ding's seminar presentation
• Project Final Presentation
  • PDF of Poster
• Project Final Report
  • Final Report
  • links to any appendices or other material

1. Murat Kocaoglu,  Christopher Snyder,  Alexandros G. Dimakis,  and Sriram Vishwanath.  Causalgan: Learning causal implicit generative models with adversarial training. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018
2. hristos Louizos, Uri Shalit, Joris M. Mooij, David A. Sontag, Richard S. Zemel, and Max Welling. Causal effect inference with deep latent-variable models. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 6446–6456, 2017
3. Nick Pawlowski, Daniel Coelho de Castro, and Ben Glocker. Deep structural causal models for tractable counterfactual inference. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
4. Yu Kong and Yun Fu. Human action recognition and prediction: A survey. CoRR, abs/1806.11230, 2018

1. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998–6008, 2017.
2. Yixin Gao, S. Swaroop Vedula, Carol E. Reiley, Narges Ahmidi, Balakrishnan Varadarajan, Henry C. Lin, Lingling Tao, Luca Zappella, Benjam ́ın B ́ejar, David D. Yuh, Chi Chiung Grace Chen, Ren ́e Vidal, Sanjeev Khudanpur and Gregory D. Hager, The JHU-ISI Gesture and Skill Assessment Working Set (JIGSAWS): A Surgical Activity Dataset for Human Motion Modeling, In Modeling and Monitoring of Computer Assisted Interventions (M2CAI) – MICCAI Workshop, 2014.
3. Narges Ahmidi, Lingling Tao, Shahin Sefati, Yixin Gao, Colin Lea, Benjamın Bejar Haro, Luca Zappella, Sanjeev Khudanpur, Rene Vidal, Fellow, IEEE, Gregory D. Hager, A Dataset and Benchmarks for Segmentation and Recognition of Gestures in Robotic Surgery,  Transaction of Biomedical Engineering, 2017. 

Here give list of other project files (e.g., source code) associated with the project. If these are online give a link to an appropriate external repository or to uploaded media files under this name space (2021-13).

Re-annotated and Segmented JIGSAW-KNOT-TYING dataset

Codes and Readme for annotating and segmenting/paring the dataset

Report for the analysis of the dataset

Code for kinematics predictor