Table of Contents

Mosquito Vision: Guidance for Automated Robotic Mosquito Salivary Gland Extraction

Last updated: May 9, 2020

Summary

The goal of this project is to create a ROS-integrated computer vision system for mosquito detection and keypoint identification to guide an automated mosquito dissection robotic system for live malaria vaccine production. The target keypoints on the mosquito to be identified include the proboscis, the head, and the neck. This project is in conjunction with Sanaria Inc. (Rockville, MD).

Background, Specific Aims, and Significance

Malaria is a mosquito-borne disease that affects humans and is caused by a single-celled organism of the Plasmodium group. In 2017, there were over 200 million clinical cases of malaria, causing over 435,000 deaths, and over $12 billion USD loss in Africa alone. Despite the impact that malaria has and the clear need, there currently exists no effective malaria vaccine in the market. However, Sanaria, a biotechnology company in Rockville MD, has recently been successful in developing a live malaria vaccine that has shown to be up to 100% effect in clinical cases. These vaccines are made from attenuated Plasmodium falciparum sporozoites (PfSPZ), the very bacteria that causes malaria. Because of the live nature of these vaccines, they must be cultivated within live mosquitoes, and hence must also be extracted from mosquito salivary glands before being able to be used as a vaccine.

Currently, the workflow to create this vaccine is slow, requiring manual extraction of the attenuated PfSPZ from salivary glands using syringes. Hence, an autonomous robotic system is currently being developed in order to automate the process. The full workflow for automated PfSPZ extraction is outlined in Figure 2. Our project will focus on leveraging computer vision techniques to allow the robot to autonomously detect keypoints on the mosquito. This will allow the robot to optimally position the mosquito for decapitation and salivary gland extraction while minimizing human input.

Project goals:

  1. Develop computer vision methods for
    • mosquito localization on a rotating platform
    • mosquito body part localization: head, neck, proboscis
    • mosquito orientation recognition: left, right, other
  2. Integrate implemented methods with robotic test bed
  3. Write documentation for algorithms and system integration
  4. Validate methods using robot system and mosquitos

All of these components are required to provide robust vision guidance to support a fully autonomous system, ultimately streamlining the production of an impactful vaccine.

Deliverables

Technical Approach

Model-Based Approach

The main goals of the model-based approaches are to (1) detect and locate mosquitoes, (2) locate the center of the neck, and (3) locate the center of the proboscis. This is shown below in figure 3, as algorithms part of a proposed Sanaria CV library that communicates with the robot via the ROS client.

The goals of each of these steps is to find where the mosquitoes are on the rotating wheel, determine how far the robot must drag the mosquito to align it with the blade cutter, and determine where to grab the mosquito respectively. Each of these steps will be performed using traditional computer vision methods as follows:

  1. Image processing to isolate correct mosquito, error checking
    • Thresholding
    • Binary image processing (erosion, dilation)
    • Connected components labeling
    • Filtering, creating bounding boxes
  2. Template matching, model-based optimization for head/neck localization
  3. Image processing to identify proboscis
    • Model-based optimization to find endpoints of proboscis

Ultimately, these algorithms will be encapsulated in a single library with a function each to find a mosquito, locate its neck, and locate the endpoints of its proboscis. Furthermore, since the entire robot system and its individual components will be integrated in ROS, we will create a ROS service which can be called by the robot driver client, ultimately using these algorithms and seamlessly transmitting the outputs to later stages in the workflow.

Deep Learning Approach

The main goals of the deep learning approaches for this project are to (1) classify the orientation of the mosquito, (2) detect and locate the mosquitoes, and (3) locate keypoints of the mosquito, which include the proboscis tip, the proboscis end, the head, the neck, the thorax, and the abdomen. The process of creating deep learning neural network models and using them in practice is outlined below in figure 4.

Deep learning can occur via either training from scratch, or transfer learning. Training from scratch (shown in figure 4 as the right light-blue shaded box) occurs when a network model is initialized from random weights, and training images are provided to the network, where a series of forward propagation and backpropagation are performed so that the network is able to learn features of the image that it uses for classification. Though this specializes the model for our purposes, training is slow, and because of our limited dataset, accuracy is low. Validation images are used to evaluate the training progress of the models.

Therefore, transfer learning was used for this project, which involves taking a pretrained network (shown in figure 4 as the left light-blue shaded box) and fine-tuning it. By pretraining the network on other images, the neural network already has a representation of features of images that it can use for classification, speeding up the training process. The fine tuning process outlined in the image by using our training and validation dataset allows us to modify the model weights such that it can be used for our purposes.

Once the model has been trained, inference can occur, as shown in the light-red shaded box in the bottom of figure 4. The model is loaded, and the relevant image is imported and processed. Inference occurs when the image is fed into the trained neural network, where it will output the classification or other predictions.

Orientation Classification The orientation classification model aims to classify the mosquito as either missing, lying on its back or stomach, lying on its left side, or lying on its right side, as seen in figure 5. Deep learning is the preferred modality over image processing, since orientation is conveyed by the textures of the mosquito rather than distinct features that one can easily identify. Hence the use of deep learning to automatically learn these implicit features would have more success. Transfer learning of neural network models was performed via PyTorch. Pretrained models such as ResNet18, ResNet152, VGG16, and DenseNet121 were loaded from PyTorch, and trained on our orientation image dataset. Conditions such as data augmentation, learning rate, epoch number, and optimizers were varied to determine the best combination that would yield the highest accuracy.

  1. Target accuracy: 95%
  2. Target running time: < 1000 ms

Besides using deep learning for orientation determination, an exploratory side of this project is to use deep learning for automatic detection of keypoints of the mosquito, such as the proboscis tip and base, neck, midpoint between thorax and abdomen, and the end of the abdomen. It is the hope that if deep learning is successful, then it will be able to support (or even replace) the image processing algorithms, as sufficient training will make the detection robust to lighting conditions and other factors that may stump the image processing algorithm, providing more confident estimations of the location of the neck and other keypoints of the mosquito.

Mosquito Detection Transfer learning was also performed for our mosquito detection task, which involves locating the mosquito in the image. The Detectron2 library framework, made by Facebook AI and based on Mask R-CNN, was used for training, as it hosts a large number of pretrained models, and supports object detection, as required for this project. To use this model, Detectron2 models were loaded from the library, and training and validation images from the old system setup were used to train the model via the framework. Once training was completed, testing images were then provided to the model, where evaluation was performed.

  1. Target mAP@IoU=0.5: 0.96
  2. Target mAP@IoU=0.75: 0.84
  3. Target running time: < 400 ms

Mosquito Pose Estimation For mosquito pose estimation/keypoint detection, the Detectron2 library was also used, as it supported human pose estimation, which was re-purposed to be applicable for mosquito keypoints. Again, Detectron2 models were loaded from the library, and training and validation images from the old system setup were used to train the model via the framework. Once training was completed, testing images were then provided to the model, where evaluation was performed.

  1. Target average RMSE for all body parts: 4 pixels
  2. Target running time: < 400 ms

The proposed workflow for training both mosquito detection and mosquito pose estimation is as follows:

  1. Detectron2 pretrained model for either mosquito detection or pose estimation is loaded
  2. Training and validation images are loaded in
  3. Annotations for the training and validation images are parsed
  4. Images and annotations are provided to the Detectron2 framework, and training occurs
  5. Testing images are loaded and annotations parsed
  6. Evaluation of the models is performed on testing images
  7. Tweaking and variations of the model and training parameters are done to further refine the model

Dependencies

Items in red are no longer applicable due to the changes due to COVID-19

Milestones and Status

Items in red are no longer possible due to COVID-19 updates.

  1. Milestone name: Deep learning environment setup
    • Responsibility: Alan
    • Planned Date: 2020/02/14
    • Expected Date: 2020/02/14
    • Status: Complete
  2. Milestone name: ~95% success rate for orientation detection
    • Responsibility: Alan
    • Planned Date: 2020/02/28
    • Expected Date: 2020/02/26
    • Status: Complete
  3. Milestone name: Documentation for orientation detection workflow and algorithms
    • Responsibility: Alan
    • Planned Date: 2020/02/28
    • Expected Date: 2020/03/16
    • Status: Complete
  4. Milestone name: ROS integration of head & neck localization algorithm
    • Responsibility: Akash & Parth
    • Planned Date: 2020/02/28
    • Expected Date: 2020/02/28
    • Status: Complete 2020/02/26
  5. Milestone name: ~95% success rate for head and neck localization within 30 pixels
    • Responsibility: Akash & Parth
    • Planned Date: 2020/02/28
    • Expected Date: 2020/02/28
    • Status: Complete
  6. Milestone name: ~95% success rate for mosquito finder (IOU > 70%)
    • Responsibility: Akash & Parth
    • Planned Date: 2020/03/16
    • Expected Date: 2020/03/16
    • Status: Complete
  7. Milestone name: ~95% success rate for proboscis finder (Lateral Error < 30 pixels)
    • Responsibility: Akash & Parth
    • Planned Date: 2020/04/01
    • Expected Date: 2020/04/01
    • Status: Complete
  8. Milestone name: ~95% success rate for bounding box detection deep learning algorithm (IOU > 0.5)
    • Responsibility: Alan
    • Planned Date: 2020/04/05
    • Expected Date: 2020/04/05
    • Status: Complete
  9. Milestone name: Deep learning pose estimation average error within 5 pixels for validation set
    • Responsibility: Alan
    • Planned Date: 2020/04/30
    • Expected Date: 2020/04/30
    • Status: Complete
  10. Milestone name: Full system integration of head, neck, and proboscis detection
    • Responsibility: Akash & Parth
    • Planned Date: 2020/04/29
    • Expected Date: 2020/04/29
    • Status: Complete
  11. Milestone name: Creation of library for training and usage of bounding box and keypoint detection
    • Responsibility: Alan
    • Planned Date: 2020/05/05
    • Expected Date: 2020/05/05
    • Status: Complete

Reports and presentations

Project Bibliography

  1. H. Phalen, P. Vagdargi, M. Pozin, S. Chakravarty, G. S. Chirikjian, I. Iordachita, and R. H. Taylor, “Mosquito Pick-and-Place: Automating a Key Step in PfSPZ-based Malaria Vaccine Production”, in IEEE Conference on Automation Science and Engineering (CASE), Vancouver, BC, August 22-26, 2019. pp. 12-17.
  2. M. Schrum, A. Canezin, S. Chakravarty, M. Laskowski, S. Comert, Y. Sevimli, G. S. Chirikjian, Stephen L. Hoffman, and R. H. Taylor, “An Efficient Production Process for Extracting Salivary Glands from Mosquitoes”, arXIV, 2019, http://arxiv.org/abs/1903.02532.
  3. H. Wu, J. Mu, T. Da, M. Xu, R. H. Taylor, I. Iordachita, and G. S. Chirikjian, “Multi-mosquito object detection and 2D pose estimation for automation of PfSPZ malaria vaccine production”, in IEEE 15th International Conference on Automation Science and Engineering (CASE), Vancouver, BC, August 22-26, 2019.
  4. M. Xu, S. Lyu, Y. Xu, C. Kocabalkanli, B. K. Chirikjian, J. S. Chirikjian, J. Davis, J. S. Kim, I. Iordachita, R. H. Taylor, and G. S. Chirikjian, “Mosquito Staging Apparatus for producing PfSPZ Malaria Vaccines”, in IEEE 15th International Conference on Automation Science and Engineering (CASE), Vancouver, BC, August 22-26, 2019.

Other Resources and Project Files

Here give list of other project files (e.g., source code) associated with the project. If these are online give a link to an appropriate external repository or to uploaded media files under this name space (2020-05).

Source code and Documentation for deep learning can be found here

Source code and Documentation for image processing algorithms can be found here