Last updated: 05/14/2023, 7 p.m.

Sanaria Inc. has developed a viable vaccine for malaria that involves using the salivary glands dissected from mosquitoes. Automating current manual dissection processes could help Sanaria reach global-scale production-level targets. An efficient automated mosquito salivary gland extraction system requires robust, high-performance computer vision methods for robot control and quality control. We aim to use model-based (IP) and machine-learning-based (DL) computer vision methods to facilitate robotic mosquito dissection:

1. Model based approach: Conventional image processing methods
2. Machine learning based approach: Using Deep Learning

• Students:
  • Yutai Wang, Robotics MSE
• Mentor(s):
  • Balazs Vagvolgyi, Associate Research Scientist

Malaria is a serious and sometimes fatal disease caused by a parasite that commonly infects a certain type of mosquito that feeds on humans. People who get malaria are typically very sick with high fevers, shaking chills, and flu-like illnesses. About 2,000 cases of malaria are diagnosed in the United States each year. There are over 200 million cases of malaria every year globally which results in more than 400,000 deaths. The disease is caused by a parasite that incubates inside the salivary glands of mosquitoes. Extracting these sporozoites from mosquito salivary glands enables the manufacturing of one promising malaria vaccine.

However, the current extraction process is fully manual and requires highly trained technicians to perform delicate manual operations under a microscope. The process is time-consuming and expensive. An automated dissection process is being developed at LCSR that uses a robotic microsurgical instrument to manipulate mosquitoes. The autonomy of the robotic system hinges on sophisticated computer vision methods to detect mosquitoes and their body parts and to provide quality control during the process.

This project aims to create vision algorithms for the robot mosquito dissection system, which is an important part of continuing development. Specific aims are to develop new DL-based CV methods and integrate existing CV methods for the mosquito dissection system, which include:

1. Mosquito Orientation Detection
2. Exudate Quality Evaluation
3. Prediction of Dissection Success
4. Exudate Volume Estimation

This project has great importance as it seeks to progress towards building a proficient robotic architecture for dissecting mosquitoes. These visual algorithms will amplify the efficacy of the system and observe its performance of the system. The computer vision methods are essential for reliable and efficient operation of the system. And, these algorithms can also minimize operational expenses during implementation. This project will have much broader implications, propelling Sanaria nearer to mass-producing a malaria inoculation.

• Minimum: (Expected 04/14/2023)
  1. Completed mosquito orientation detection code in Gitlab repository/ Documentation in Gitlab Wiki and Readme files.
  2. Completed exudate quality evaluation code in Gitlab repository/ Documentation in Gitlab Wiki and Readme files.

• Expected: (Expected 05/09/2023)
  1. Completed prediction of dissection success code in Gitlab repository/ Documentation in Gitlab Wiki and Readme files.

• Maximum: (Expected 05/09/2023)
  1. Completed exudate volume estimation code in Gitlab repository/ Documentation in Gitlab Wiki and Readme files.

The key activities can be found below

Codes and label files in Gitlab repository for these tasks: (Access to the project Gitlab is required. Ask Prof.Russell H. Taylor or Balazs Vagvolgyi to grant permission.)

Classification Deep Learning Architectures: https://git.lcsr.jhu.edu/mosquito-vision/sanaria_cv_dl/-/tree/main/lib/sanaria_classification_dl

Mosquito Orientation Classification: https://git.lcsr.jhu.edu/mosquito-vision/sanaria_cv_dl/-/tree/main/tasks/mosquito_orientation_2

Exudate Image Quality Classification: https://git.lcsr.jhu.edu/mosquito-vision/sanaria_cv_dl/-/tree/main/tasks/exudate_image_quality_classification

Exudate Quality Classification: https://git.lcsr.jhu.edu/mosquito-vision/sanaria_cv_dl/-/tree/main/tasks/exudate_quality_classification

Prediction of Dissection Success: https://git.lcsr.jhu.edu/mosquito-vision/sanaria_cv_dl/-/tree/main/tasks/success_prediction_2

GitLab Wiki and documentation for these tasks: (Access to the project Gitlab is required. Ask Prof.Russell H. Taylor or Balazs Vagvolgyi to grant permission.)

Classification Deep Learning Architectures: https://git.lcsr.jhu.edu/mosquito-vision/sanaria_cv_dl/-/wikis/Classification-Deep-Learning-Architectures

Mosquito Orientation Classification: https://git.lcsr.jhu.edu/mosquito-vision/sanaria_cv_dl/-/wikis/Orientation-Classification

Exudate Image Quality Classification: https://git.lcsr.jhu.edu/mosquito-vision/sanaria_cv_dl/-/wikis/Exudate-Image-Quality-Classification

Exudate Quality Classification: https://git.lcsr.jhu.edu/mosquito-vision/sanaria_cv_dl/-/wikis/Exudate-Quality-Classification

Prediction of Dissection Success: https://git.lcsr.jhu.edu/mosquito-vision/sanaria_cv_dl/-/wikis/Prediction-of-Dissection-Success

Here is a flow chart of the technical approach:

Images and Annotations

A large amount of image data is needed to train the classification model. The automated robotic system has several cameras with different angles in different positions. Many images have been stored in the database since the system was built. However, not every image has the correct label. Need to correct the label for each image in this database. Extract images and their corresponding annotation information from the database. Perform data type conversion and make training sets.

Deep Neural Network

Enough data has now been collected for this project. So, it is the best choice to use deep neural network to complete the above classification task. For classification tasks, deep neural network has many advantages. Deep neural networks offer a powerful and flexible approach to classification tasks, with many advantages over traditional machine learning algorithms. Deep neural networks can learn to automatically extract relevant features from raw data, without the need for hand-crafted feature engineering. This can save a lot of time and effort, as it eliminates the need for domain-specific knowledge or expert input in the feature extraction process. Besides, it can be scaled up to handle very large datasets with many input features and output classes. This makes the deep neural networks suitable for a wide range of classification tasks. Moreover, pre-trained deep neural networks can be used as a starting point for new classification tasks, by fine-tuning the network on a smaller dataset or a new set of classes. This can significantly reduce the amount of data and training time required for new tasks and can lead to better performance than training a new model from scratch. We mainly use three deep neural networks: ResNet, VGG, and DenseNet. Those three networks could be imported by Pytorch. PyTorch is used for this project because it provides a Python-based interface for building and training machine learning models, which can make the training process much easier.

Data Augmentation

Data augmentation refers to the process of artificially increasing the size of a training dataset by applying various transformations or modifications to the existing images. These transformations create new training samples that are variations of the original images while preserving their label or class. Data augmentation is necessary for the small number of datasets like ones I use in this project. The image data in the database are not updated for a long time. I only have a limited amount of image data available. By augmenting the existing dataset, the effective size of the training data is increased, which provides more diverse examples for the model to learn from. This helps to prevent overfitting and improves the model's ability to generalize well to unseen data. It can also introduce variations in the training samples, making the model more robust to changes in the input data. It allows the model to learn features that are invariant to transformations like rotation, scaling, or translation, which can occur in real-world scenarios. There are many kinds of data augmentation methods, such as geometric and color augmentation, such as reflecting the image, cropping, and translating the image, changing the color palette of the image, color processing, and geometrical transformations. In this project, I use image blurry to create more bad quality images, and randomly select ROI in each image to create more bad view images.

The project is mainly virtual so there are no physical dependencies. The dependencies are listed in the figure.

1. Milestone name: Mosquito Orientation Detection
   • Planned Date: March 12th, 2023
   • Expected Date: March 16th, 2023
   • Status: Completed
2. Milestone name: Exudate Quality Evaluation
   • Planned Date: April 1st, 2023
   • Expected Date: April 14th, 2023
   • Status: Completed
3. Milestone name: Prediction of Dissection Success
   • Planned Date: April 30th, 2023
   • Expected Date: May 9th, 2023
   • Status: Completed
4. Milestone name: Exudate Volume Estimation
   • Planned Date: May 9th, 2023
   • Expected Date: May 14th, 2023
   • Status: Future Work

• Credits

1. Yutai Wang (Team Leader/ Team Member): Responsible for all tasks. Developed classification neural networks. Collected image data and collaborated with labeling. Designed the experiment platform and conducted experiments for the extra work. Wrote documentation and uploaded corresponding codes to project’s Gitlab for each task.
2. Balazs Vagvolgyi (Mentor): Provide mentorship. Have regular weekly meeting with me to discuss results and determine what should be the next step.

• Accomplished VS Planned

• Future Plan

1. Based on the Prediction of Dissection Success results, investigate methods to locate specific regions on mosquito images that contribute strongest to variability in exudate quality. Record relevant codes and documentation in Gitlab.
2. Develop exudate volume estimation based on deep learning methods. Complete exudate volume estimation codes in Gitlab repository. Write documentation in Gitlab Wiki and Readme files.

• Project Plan
  • project_plan_presentation.pdf
  • project_plan_proposal.pdf
• Project Background Reading
  • See Bibliography below for links.
  • background_reading_presentation.pdf
  • Paper for this presentation: Yang Y, Zhang L, Du M, Bo J, Liu H, Ren L, Li X, Deen MJ. A comparative analysis of eleven neural networks architectures for small datasets of lung images of COVID-19 patients toward improved clinical decisions. Comput Biol Med. 2021 Dec;139:104887. doi: 10.1016/j.compbiomed.2021.104887. Epub 2021 Sep 24. PMID: 34688974; PMCID: PMC8461289.
• Project Checkpoint
  • checkpoint_presentation.pdf
• Project Final Presentation
  • team04_yutaiwang_cisii_poster.pdf
• Project Final Report
  • project4_final_report_revised.pdf
  • cis_ii_2023_project_mentors_report_team4.pdf

1. Image classification. Papers With Code. (n.d.). Retrieved February 20, 2023, from https://paperswithcode.com/task/image-classification
2. Deep Neural Networks. Tutorials Point. (n.d.). Retrieved February 20, 2023, from https://www.tutorialspoint.com/python_deep_learning/python_deep_learning_deep_neural_networks.htm
3. • Learning pytorch with examples. Learning PyTorch with Examples - PyTorch Tutorials 1.13.1+cu117 documentation. (n.d.). Retrieved February 23, 2023, from https://pytorch.org/tutorials/beginner/pytorch_with_examples.html

1. Centers for Disease Control and Prevention. (2022, March 22). CDC - Malaria - about malaria - faqs. Centers for Disease Control and Prevention. Retrieved February 5, 2023, from https://www.cdc.gov/malaria/about/faqs.html
2. Schrum, M., Canezin, A., Chakravarty, S., Laskowski, M., Comert, S., Sevimli, Y., … & Taylor, R. H. (2019). An efficient production process for extracting salivary glands from mosquitoes. arXiv preprint arXiv:1903.02532
3. W. Li et al., “Automated Mosquito Salivary Gland Extractor for PfSPZ-based Malaria Vaccine Production,” 2021 IEEE International Conference on Robotics and Automation (ICRA), Xi'an, China, 2021, pp. 866-872, doi: 10.1109/ICRA48506.2021.9560959
4. M. Xu et al., “Mosquito Staging Apparatus for Producing PfSPZ Malaria Vaccines,” 2019 IEEE 15th International Conference on Automation Science and Engineering (CASE), Vancouver, BC, Canada, 2019, pp. 443-449, doi: 10.1109/COASE.2019.8843147
5. He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 770-778)
6. Simonyan, K., & Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556
7. Huang, G., Liu, Z., Van Der Maaten, L., & Weinberger, K. Q. (2017). Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 4700-4708)