Last updated: May 6th, 2021

Malaria vaccine production and distribution is a critical public health challenge for many developing countries. Sanaria Inc is working with the LCSR to develop a robotic system to automate mosquito saliva gland collection, a bottleneck in vaccine production. Previous researchers at JHU have successfully extracted salivary glands by developing the proper robot hardware, control algorithm, and computer vision models. My goal will be to implement improvements to the existing control algorithm to increase parallelization, add error recovery, and support the next generation hardware and computer vision efforts.

• Students:
  • Zhuohong (Zooey) He, MSE Robotics '21
• Mentors:
  • Dr. Simon Leonard, Assistant Research Professor (Computer Science)
  • Dr. Russell Taylor, John C. Malone Professor (Computer Science)

Figure 1. Robot system produced in late 2020, new hardware and software improvements have occurred since.

The malaria epidemic affects millions of people worldwide every year. Using mosquitos as its transmission vector, malaria is highly contagious, quickly mutating, and a major killer in parts of the world. Sanaria is one company attempting to tackle this crisis by producing a malaria vaccine. However, scaling up vaccine production presents a major bottleneck at the mosquito harvesting step. Working with Sanaria, our team at JHU is developing a system to automate mosquito harvesting by constructing a complex procedure to efficiently transport, decapitate, and extract the salivary glands. The control is implemented using the Robot Operating System (ROS) which sets up the hardware as individual services which an overall controller can call.

The lab is currently simultaneously developing software and hardware. Our first-generation controller created ROS node wrappers for each of the processes and implemented a basic overall controller. The second-generation controller will increase the robustness of overall controller by adding more error checks and recovery. The new controller will also incorporate the new mechanisms developed by the hardware team. A comprehensive flowchart has already been developed. Additionally, we will perform testing both in simulation and the physical world to verify and debug the controller.

• Minimum: (Delivered Apr. 26)
  1. Fully debugged, well documented controller; Create five independent nodes to call services to execute flowchart steps.
  2. Robust errors & recovery shown in the flowchart. Define / document other errors.
  3. Controller that operates on the new hardware / setup. Focusing on using placeholders instead of Low-Level Arduino/Galil control.
  4. Visual testing using the simulation
  • Link to code and documentation: External Link Note: The hyperlink brings the user to a repository with links to each module/node. The documentation is located in the wikis of the modules/nodes.

• Expected: (Expected by Apr. 5)
  1. Create advanced figures and documentation in the README files and wiki.
  2. Testing on select hardware on the physical system.

• Maximum: (Expected by date)
  1. Future-proof the system by writing highly generalizable and inheritable code. Create abstract objects, etc.
  2. Work with CV team to implement further error handling; one example is dragging detection. Another is ensuring neck is between blades.
  3. Write Arduino/Galil code for low level control of hardware. Create working services for the physical robot.
  4. Quantitative testing on the physical system to determine success rate.

We are using an open-source framework called Robot Operating System (ROS) to implement our robot controller. ROS is a collection of tools and libraries that simplify the complex task of robot control. At the core of ROS is the node, a logical unit that can communicate with other nodes using ROS messages and execute a script to move the robot, call a neural network, output to a GUI, and more.

In order to structure the code to allow for error recovery and ease of communication, we are employing a ROS package known as actionlib. Actionlib's ActionClient and ActionServer communicate via a “ROS Action Protocol” which is built on top of basic ROS messages. The client is the caller which requests the server, or the callee, to execute a task and return the result. Thus, the client-server architecture provides the basic functions to efficiently operate the robot system.

Figure 2: ROS ActionClient and ActionServer Relationship

Our lab has already implemented a similar controller without error recovery for our last hardware iteration using the client-server architecture. My project will focus on supporting the new hardware, adding error recovery, and parallelizing processes. Our new controller will be based on a flow chart (fig. 3) which has five parallel nodes which each control independent subsystems: the robot, gland extractor, turntable cleaner, turntable, and robot cleaner.

Figure 3: Mosquito Microdissection Robot Controller Flow Chart. Click for expanded image

Our lab has two means for debugging and testing. The first option is testing on the physical robot which is located in JHU LCSR robotorium and can be accessed and operated remotely as well. The second option is an ROS RVIZ simulation which I helped develop for the last hardware iteration. It needs to be updated to the newest hardware in order to be used for our purposes.

Above is a full map detailing the action client/server architecture for 2 nodes: Mosquito Pick-Place-Decapitate (MPPD) and gland extractor. Matching blue and green objects are action client/server pairs. They communicate via a topic shown with a “cloud object”. For an interactive diagram visit: LINK

I conducted testing on 30 mosquitoes and manually determined the success rate, error recovery rate, and throughput of various components in the system. The summary data is displayed in table \ref{table:data} which shows that the number of total successes (initial success count + recovery count), the step success rate, and the cumulative success rate.

The cumulative success rate for the MPPD node is only 53% largely because of the decapitate step with a individual success rate of only 67%. In our observation, the decapitate success rate could be improved through better placing of the mosquitoes between the blades. Often, mosquito bodies were above the blade's height. When the blades actuated, these mosquitoes do not become successfully decapitated; their bodies were simply pushed up during actuation which avoids the cut. The problem could be avoided with taller blades and the implementation of the decapitate validation step. Additional hardware to keep the mosquito bodies within the slots could also mitigate the issue.

Due to a lack of mature hardware and CV, we were only able to perform quantitative testing on error recovery for the pick and place step. However, error recovery allowed us to save 100% of missed mosquitoes by the the pick step, demonstrating the potential of the CV error detection and controller recovery.

Using a handheld timer, we determined the average time for the MPPD node to process one mosquito to be roughly 10s. The MPPD node operates for longer than the gland extraction node, thus we can use the MPPD node to estimate the mosquito throughput at 360 mosquitoes per hour (mph). The gripper actuation uses a slow linear servo, which could easily be replaced to remove around 2s per mosquito, improving the rate to 450mph. Since our goal rate is 600mph, we could investigate increasing the robot move speed and cutter speed to reach our goal.

Video 1: Error Recovery Video Demonstration Video 2: Parallel Operation of the MPPD, Extraction, and Turntable Nodes

The below table summarizes the dependencies, the need justification, resolution deadline, final status, and adjustments made. Because of the hardware team was not yet ready to deliver the robot cleaner and hardware, we modified our original minimum deliverable of five nodes (MPPD, Extract, Turntable, Robot Cleaner, Turntable Cleaner) to three nodes (MPPD, Extract, Turntable). Additionally, because the CV team delivered later than expected, I simply put placeholders (i.e. simply pause 3 seconds, then return success) in place of many CV components. The simulation was also delivered late by another team member because he was writing a paper. Thus, we adjusted our timeline to perform simulation testing concurrently with the physical testing, much later than planned.

Table 1: Dependencies Table

Last Updated 5.07.2021

1. Milestone name: Implement Turntable Node
   • Planned Date: March 1st, 2021
   • Status: Completed (March 7th) DONE ✔
2. Milestone name: Implement Mosquito Pick, Place, and Dissect Node (MPPD)
   • Planned Date: March 15th, 2021
   • Status: Completed (March 24th) DONE ✔
3. Milestone name: Implement Cleaner Nodes
   • Planned Date: March 22nd, 2021
   • Status: CANCELLED
4. Milestone name: Implement Error Recovery
   • Planned Date: April 5th, 2021
   • Status: Completed (April 5th) DONE ✔
5. Milestone name: Physical and Simulation Testing
   • Planned Date: April 12th, 2021
   • Status: Completed (May 3rd) DONE ✔
6. Milestone name: Thorough documentation
   • Planned Date: April 26th, 2021
   • Expected Date: May 17th, 2021
   • Status: In Progress (May 3rd) IN PROGRESS —

Figure 4. Proposed vs Actual Timeline. The blue text represents the dates where portions of the minimium product were delivered. The sharded regions represent the originally planned schedule, and the outlined regions represent the actual development schedule.

• Project Plan
  • Project Plan Presentation
  • Project Plan Proposal
• Project Background Reading
  • Included at the end of the project plan.
• Project Checkpoint
  • Project Checkpoint Presentation (PDF)
  • Project Checkpoint Presentation (PPTX)
• Paper Seminar Presentations
  • Paper Review Presentation (PDF)
  • Paper Review Presentation (PPTX)
  • Paper Review Report
  • Reviewed Paper
• Project Final Presentation
  • Final Presentation (PPTX)
• Project Final Report
  • Final Report (PDF)

1. H. Phalen, P. Vagdargi, M. Schrum, S. Chakravarty, A. Canezin, M. Pozin, S. Coemert, I. Iordachita, S. Hoffman, G. Chirikjian, and R. Taylor, “A mosquito pick-and-place system for pfspz-based malaria vaccine production,” IEEE Transactions on Automation Science and Engineering, 2020, issn:1545-5955, doi:10.1109/TASE.2020.2992131.
2. H. Phalen, P. Vagdargi, M. Pozin, S. Chakravarty, G. S. Chirikjian, I. Iordachita, and R. H. Taylor, “Mosquito pick-and-place: Automating a key step in pfspz-based malaria vaccine production,” in 2019 IEEE 15th International Conference on Automation Science and Engineering (CASE), 2019, pp. 12–17.
3. M. Schrum, A. Canezin, S. Chakravarty, M. Laskowski, S. Comert, Y. Sevimli, G. S. Chirikjian, S. L. Hoffman, and R. H. Taylor, “An efficient production process for extracting salivary glands from mosquitoes,” 2019, arXiv:1903.02532 [q-bio.QM]. [Online]. Available: https://arxiv.org/abs/1903.02532
4. R. H. Taylor, A. Canezin, M. Schram, I. Iordachita, G. Chirikjian, M. Laskowski, S. Chakravarty, and S. Hoffman, “Mosquito Salivary Gland Extraction Device and Methods of Use,” Patent 20 170 355 951, December, 2017. [Online]. Available: https://www.freepatentsonline.com/y2017/0355951.html

I have attached links to two videos that are part of my deliverables for testing.

Video one: Parallelism Demonstration

Video two: Error Recovery Demonstration

The code is stored in a confidential GitLab repository.