Last updated: 2021/02/25 15:15

The main objective of this research project is to provide computer vision guidance to an automated mosquito dissection robotic system for live malaria vaccine production.

• Students: Alan Lai
• Mentor(s): Balazs Vagvolgyi, Dr Russell Taylor

The main tasks of this research include:

1. Deep learning-based prediction of mosquito decapitation/salivary extraction success or failure based on camera images of the mosquito acquired in the robotic processing pipeline prior to decapitation
   1. Or to determine the feasibility of such
2. Integration of tasks (if successful) into the sanaria_dl_service package for ROS integration with the robotic mosquito dissection system
3. Providing insight into the potential root causes of downstream decapitation and squeezing failures originating in the earlier stages of the pipeline, based on the analysis of collected image data
4. Provide guidance for CIS II project members

Previous work has already been done using computer vision to support and guide the current robotic system for automatic mosquito processing. Image processing-based methods include mosquito detection, proboscis detection, and neck detection, while deep learning methods include mosquito orientation classification, mosquito detection, and pose estimation. Many of these methods are already currently in use, providing support to automate the system.

Though those methods have been successful in allowing the robotic system to operate autonomously, comparatively little amounts of effort have been put in with regards to vision-based validation and error prevention of key steps. That is, most existing methods are responsible for guiding the robot’s motion to process mosquitoes but only one method was designed to facilitate error checking and recovery. All other error checking and validation are currently done manually by the operator and resolved manually as well.

Two key steps in the processing pipeline are the decapitation step and the squeezing step. Though the current vision algorithm for proboscis and neck detection work well, the variety in mosquito shapes, orientation, flexibility, and morphology means that even if keypoints are identified well and the robot moves through the correct motions, the mosquito may not be placed in the correct location, causing issues in decapitation and/or squeezing steps. There is currently no way of resolving these issues if mosquito decapitation or squeezing is ineffective, and the corresponding mosquito would be best case discarded or worst case create problems in the processing pipeline. If an algorithm is capable of predicting, based on mosquito positioning and orientation after the robot has placed the mosquito on the decapitation blades, whether decapitation and/or squeezing will fail, this presents an opportunity for the robot to reposition the mosquito to a valid position such that such wastage would not occur.

The reason why deep learning is preferred as a method over image processing with regards to prediction of decapitation/squeezing success/failure is that there are many failure cases that could occur, be it mosquito being too far up, too much to the side, too much in front, which all depend on individual mosquito morphology. It would be impossible to come up with an empirical set of rules for image processing to enforce, and hence deep learning, which allows for greater flexibility and robustness with respect to mosquito morphology, is a more promising solution to this problem. The other advantage of deep learning over image processing is that image processing requires knowledge of the causes of the problems and a method of modeling those causes in order to recognize and resolve the issues. As we currently do not know what causes these issues in decapitation or squeezing, deep learning will be able to allow us to circumvent the need to explicitly model the system.

• Expected: 2021/05/09
  1. Working algorithm on mosquito decapitation success prediction
  2. Working algorithm on mosquito salivary gland extraction success prediction
  3. ROS integration test client/server code for prediction and validation algorithms
  4. Thorough documentation on all algorithms and usage

Prediction of Success/Failure of Decapitation/Extraction

Top view and side view images will be taken after the robot has placed the mosquito onto the decapitation blades, but before the blades are actuated. If possible, mosquito images on the turntable prior to robot manipulation may also be taken and incorporated into the training. After the blades are actuated, the operator will manually determine whether the decapitation was a success. Then, the mosquitoes will be further processed and salivary glands will be attempted to be extracted via squeezing. Once the mosquitoes have been squeezed, the operator will then manually determine whether the squeezing was a success. The decapitation and/or squeezing success will then be written to a file that links these results to the original top and side view images taken of the mosquitoes prior to decapitation.

These images will then serve as training images for training our neural network. PyTorch will be used for training, and transfer learning, using PyTorch’s pretrained networks, will be used. This will simply be a classification problem, predicting classes:

1. Success for both decapitation and squeezing
2. Failure for decapitation
3. Failure for squeezing

Note that more classes may be added in the future that may further delineate differences in how successful the squeezing step was.

Images will be split 80-20 for training and validation sets, and parameters such as optimizer, learning rate, batch size, and other potential data augmentation schemes will be attempted in order to both (1) determine the feasibility of prediction of mosquito processing success (2) create a predictor that attempts to predict the result of the decapitation and/or squeezing steps. After training occurs, an inference pipeline will be created, and then used for testing with a new test set. Subsequent to testing, a ROS test client and server will be created in preparation of integration with the larger system.

describe dependencies and effect on milestones and deliverables if not met

1. Milestone name: Deep Learning Environment Setup
   • Planned Date: 2021/02/28
   • Expected Date: 2021/02/28
   • Status: Complete
2. Milestone name: Repo, Wiki Setup for Prediction task
   • Planned Date: 2021/02/28
   • Expected Date: 2021/02/28
   • Status: Complete
3. Milestone name: Feasibility of Prediction Algorithm Determined
   • Planned Date: 2021/03/15
   • Expected Date: 2021/03/15
   • Status: Complete
4. Milestone name: Success rate >70% for Decapitation Prediction Algorithm
   • Planned Date: 2021/03/28
   • Expected Date: 2021/03/28
   • Status: Complete
5. Milestone name: Success rate >70% for Squeezing Prediction Algorithm
   • Planned Date: 2021/04/04
   • Expected Date: 2021/04/04
   • Status: Complete
6. Milestone name: Complete framework for Prediction Training
   • Planned Date: 2021/04/18
   • Expected Date: 2021/04/18
   • Status: Complete
7. Milestone name: ROS integration for Prediction Algorithm
   • Planned Date: 2021/05/01
   • Expected Date: 2021/05/01
   • Status: Complete
8. Milestone name: Repo, Wiki Setup for Validation task
   • Planned Date: 2021/04/04
   • Expected Date: 2021/04/04
   • Status: Task Cancelled
9. Milestone name: ROS integration for Validation Algorithm
   • Planned Date: 2021/05/02
   • Expected Date: 2021/05/02
   • Status: Task Cancelled
10. Milestone name: Documentation Complete
    • Planned Date: 2021/05/09
    • Expected Date: 2021/05/09
    • Status: In Progress

• Project Plan
  • Project plan proposal
• Project Background Reading
  • See Bibliography below for links.
• Project Checkpoint
  • mosquito_decapitation_and_squeeze_success_prediction_checkpoint_presentation.pptx
• Project Final Presentation
  • mosquito_decapitation_and_squeeze_success_prediction_final_presentation.pdf
• Project Final Report
  • robotics_mse_research_final_report.pdf

* here list references and reading material

Here give list of other project files (e.g., source code) associated with the project. If these are online give a link to an appropriate external repository or to uploaded media files under this name space (2021-19).

Source code and documentation here

• Wu, H., Mu, J., Da, T., Xu, M., Taylor, R. H., Iordachita, I., & Chirikjian, G. S. (2019). Multi-mosquito object detection and 2d pose estimation for automation of PfSPZ malaria vaccine production. In 2019 IEEE 15th International Conference on Automation Science and Engineering, CASE 2019 (pp. 411-417). [8842953] (IEEE International Conference on Automation Science and Engineering; Vol. 2019-August). IEEE Computer Society. https://doi.org/10.1109/COASE.2019.8842953
• Li, W., Zhang, Z., He, Z., Vora, P., Lai, A., Vagvolgyi, B., Leonard, S., Goodridge, A., Iordachita, I., Chakravarty, S., Sim, K. L., Hoffman, S. L., Taylor, R.H. (2021). Progress in Development of an Automated Mosquito Salivary Gland Extractor: A Step Forward to Malaria Vaccine Mass Production.

Here give list of other project files (e.g., source code) associated with the project. If these are online give a link to an appropriate external repository or to uploaded media files under this name space (2021-19).