======Project 6: Vision Guided Mosquito Dissection for the Production of Malaria Vaccine======
**Last updated: March 1, 2022 14:02**


======Summary======
For the development and implementation of computer vision algorithms for the automatic mosquito dissection robot for Sanaria Inc. using classical and deep learning based methods.

  * **Students:** John Han
  * **Mentor(s):** Balazs Vagvolgyi

======Background, Specific Aims, and Significance======
Malaria accounted for 229 million cases and 409,000 deaths worldwide as of 2019. It is an understatement to say that there is a need for vaccine development and deployment. Sanaria Inc. has found groundbreaking results and progress in a malaria vaccine, which uses mosquito salivary glands for research and synthesis. However, these glands are difficult to extract manually and often requires high skill and precision, not to mention it also being time consuming. As a result, JHU is developing an automated system that can mass-extract salivary glands from mosquitos. There has been a collaboration between JHU CIIS and Sanaria Inc. in order to produce a robotic system to complete this task. 
======Deliverables======
 
  * **Minimum:** (Expected by April 2022)
    - Relatively accurate algorithms for both tasks 
    - An empirically executed experiment report to compare classical + DL methods 

  * **Expected:** (Expected by April 2022)
    - Complete integration with ROS robot workflow 
    - A report on the Gitlab repository for documentation and codebase organization 

  * **Maximum:** (Expected by date)
    - A report on the Gitlab repository for documentation and codebase organization 
    - A set of labeled images from Sanaria Inc

======Technical Approach======
As a general overview, both goals will involve an implementation of both classical and deep learning-based methods. The two approaches will be compared, one will be chosen based on performance and accuracy.  

First, the technical approach of the first goal will be explained. The objective is to detect any clutter of any kind around the cutting station and blade. The classical image processing approach will involve a combination of object detection, by thresholding and/or segmentation. 

More specifically, if we get the contrast of the image, apply histogram equalization, and threshold the image to detect high pixel values (which would correspond to mosquitos, since mosquitoes are black), this is a simple image processing algorithm to segment the mosquitoes. Then we can count the blobs by labeling connected components.  

The deep learning approach would be to acquire many images in order to train a neural network. The NN model could be something like VGGNet or ResNet. More specific details will be explored in time of implementation. I believe that a simple image processing approach would be better suited for this task, since detecting blobs has always been accurate and achievable by classical methods. 

Second, the following explains the technical approach of the second goal. The objective is to label “good” exudates from “bad” exudates. This will be from the following criteria: visibility of salivary gland, undesirable limbs and organs, and any other issues such as random debris.

The classical image processing approach is not favorable, due to its highly volatile nature of inconsistent features. It can be attempted by applying a region of interest (ROI) and calculating the volume of the exudate with certain geometric assumptions. In addition, we can use foreground-background segmentation to differentiate if there is any undesirable features inside of the exudate.  

The deep learning-based approach will be more feasible due to the flexibility and adaptability of a neural network. This will require a huge batch of data and annotations, however. Some communication with Sanaria Inc. will be established in order for the company to assist with the labeling/annotation process.  
======Dependencies======
There are dependencies, especially regard to Sanaria Inc's timely response to provide labels for task 2: quality assurance of the exudate. In this case, a neural network training script will still be created, even if the annotations from Sanaria are not provided.
======Milestones and Status ======
  - Milestone name: Preparation: Data organization and Contact Sanaria for labels
    * Planned Date: Week 1
    * Status: Finished
  - Milestone name: Classical methods' scripts written 
    * Planned Date: Week 4
    * Status: Finished
  - Milestone name: Deep learning training scripts written
    * Planned Date: Week 6
    * Status: Finished
  - Milestone name: System Integration
    * Planned Date: Week 8
    * Status: Finished
  - Milestone name: Improvements/Optimization
    * Planned Date: Week 12
    * Status: In Progress
  - Milestone name: Final Organization and Produce results report
    * Planned Date: Week 12
    * Status: Finished
  - Milestone name: Final Presentation
    * Planned Date: Week 12
    * Status: In Progress
======Reports and presentations======

  * Project Plan
    * [[https://docs.google.com/presentation/d/1K9xyj8qoW_XkIqaE89BgkTtBs0MlwTpH/edit?usp=sharing&ouid=107581954509950915958&rtpof=true&sd=true|Project Plan Presentation]]
    * [[https://docs.google.com/document/d/1mbr0ORSPjyEaCc7v8FBcsfvA-AwcOCLu/edit?usp=sharing&ouid=107581954509950915958&rtpof=true&sd=true|Project Plan Proposal]]
  * Project Background Reading 
    * See Bibliography below for links.
  * Project Checkpoint
    * {{https://drive.google.com/file/d/18Y0eTWOjPF2lQKI3yBUZW3cdZxwWN4M3/view?usp=sharing| Project checkpoint presentation}}
  * Paper Seminar Presentations
    * [[https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=8842953|Multi-Mosquito Object Detection and 2D Pose Estimation for Automation of PfSPZ Malaria Vaccine Production]]
    * [[https://docs.google.com/presentation/d/1Wn1XVvqtjrjSZwNhuUSuYYWf2fjatQib/edit?usp=sharing&ouid=107581954509950915958&rtpof=true&sd=true|Background Reading Slides]]
    * [[https://drive.google.com/file/d/1oov7-7qRfWK0xj3axsGfHTZQdkFAhHY6/view?usp=sharing|Background Reading Report]]
  * Project Final Presentation
    * {{https://drive.google.com/file/d/1WxWdLmfxysd1t-Q-c08dCc2uWVBcbtPP/view?usp=sharing|PDF of Poster}}
  * Project Final Report
    * {{https://drive.google.com/file/d/1SfhD6F5OZfnpEnSdHZO4x_B2aRo3HjLf/view?usp=sharing|Final Report}}
    * links to any appendices or other material

======Project Bibliography=======

  * Phalen, Henry, et al. “A Mosquito Pick-and-Place System for PfSPZ-Based Malaria Vaccine Production.” IEEE Transactions on Automation Science and Engineering, vol. 18, no. 1, 2021, pp. 299–310., https://doi.org/10.1109/tase.2020.2992131.
  * Schrum, Mariah, et al. “An Efficient Production Process for Extracting Salivary Glands from Mosquitoes” 
  * Wu, Hongtao, et al. “Multi-Mosquito Object Detection and 2D Pose Estimation for Automation of Pfspz Malaria Vaccine Production.” 2019 IEEE 15th International Conference on Automation Science and Engineering (CASE), 2019, https://doi.org/10.1109/coase.2019.8842953. 
======Other Resources and Project Files======
  * Personal Git Repo: https://github.com/juseonghan/CIS_Sanaria
  * LCSR Sanaria CV DL Git Repo: https://git.lcsr.jhu.edu/mosquito-vision/sanaria_cv_dl