Last updated: 2021/02/25 15:15
The main objective of this research project is to provide computer vision guidance to an automated mosquito dissection robotic system for live malaria vaccine production.
The main tasks of this research include:
Previous work has already been done using computer vision to support and guide the current robotic system for automatic mosquito processing. Image processing-based methods include mosquito detection, proboscis detection, and neck detection, while deep learning methods include mosquito orientation classification, mosquito detection, and pose estimation. Many of these methods are already currently in use, providing support to automate the system.
Though those methods have been successful in allowing the robotic system to operate autonomously, comparatively little amounts of effort have been put in with regards to vision-based validation and error prevention of key steps. That is, most existing methods are responsible for guiding the robot’s motion to process mosquitoes but only one method was designed to facilitate error checking and recovery. All other error checking and validation are currently done manually by the operator and resolved manually as well.
Two key steps in the processing pipeline are the decapitation step and the squeezing step. Though the current vision algorithm for proboscis and neck detection work well, the variety in mosquito shapes, orientation, flexibility, and morphology means that even if keypoints are identified well and the robot moves through the correct motions, the mosquito may not be placed in the correct location, causing issues in decapitation and/or squeezing steps. There is currently no way of resolving these issues if mosquito decapitation or squeezing is ineffective, and the corresponding mosquito would be best case discarded or worst case create problems in the processing pipeline. If an algorithm is capable of predicting, based on mosquito positioning and orientation after the robot has placed the mosquito on the decapitation blades, whether decapitation and/or squeezing will fail, this presents an opportunity for the robot to reposition the mosquito to a valid position such that such wastage would not occur.
The reason why deep learning is preferred as a method over image processing with regards to prediction of decapitation/squeezing success/failure is that there are many failure cases that could occur, be it mosquito being too far up, too much to the side, too much in front, which all depend on individual mosquito morphology. It would be impossible to come up with an empirical set of rules for image processing to enforce, and hence deep learning, which allows for greater flexibility and robustness with respect to mosquito morphology, is a more promising solution to this problem. The other advantage of deep learning over image processing is that image processing requires knowledge of the causes of the problems and a method of modeling those causes in order to recognize and resolve the issues. As we currently do not know what causes these issues in decapitation or squeezing, deep learning will be able to allow us to circumvent the need to explicitly model the system.
Prediction of Success/Failure of Decapitation/Extraction
Top view and side view images will be taken after the robot has placed the mosquito onto the decapitation blades, but before the blades are actuated. If possible, mosquito images on the turntable prior to robot manipulation may also be taken and incorporated into the training. After the blades are actuated, the operator will manually determine whether the decapitation was a success. Then, the mosquitoes will be further processed and salivary glands will be attempted to be extracted via squeezing. Once the mosquitoes have been squeezed, the operator will then manually determine whether the squeezing was a success. The decapitation and/or squeezing success will then be written to a file that links these results to the original top and side view images taken of the mosquitoes prior to decapitation.
These images will then serve as training images for training our neural network. PyTorch will be used for training, and transfer learning, using PyTorch’s pretrained networks, will be used. This will simply be a classification problem, predicting classes:
Note that more classes may be added in the future that may further delineate differences in how successful the squeezing step was.
Images will be split 80-20 for training and validation sets, and parameters such as optimizer, learning rate, batch size, and other potential data augmentation schemes will be attempted in order to both (1) determine the feasibility of prediction of mosquito processing success (2) create a predictor that attempts to predict the result of the decapitation and/or squeezing steps. After training occurs, an inference pipeline will be created, and then used for testing with a new test set. Subsequent to testing, a ROS test client and server will be created in preparation of integration with the larger system.
describe dependencies and effect on milestones and deliverables if not met
* here list references and reading material
Here give list of other project files (e.g., source code) associated with the project. If these are online give a link to an appropriate external repository or to uploaded media files under this name space (2021-19).
Source code and documentation here
Here give list of other project files (e.g., source code) associated with the project. If these are online give a link to an appropriate external repository or to uploaded media files under this name space (2021-19).