Contact Us
CiiS Lab
Johns Hopkins University
112 Hackerman Hall
3400 N. Charles Street
Baltimore, MD 21218
Directions
Lab Director
Russell Taylor
127 Hackerman Hall
rht@jhu.edu
Last updated: May 6th, 2021
Malaria vaccine production and distribution is a critical public health challenge for many developing countries. Sanaria Inc is working with the LCSR to develop a robotic system to automate mosquito saliva gland collection, a bottleneck in vaccine production. Previous researchers at JHU have successfully extracted salivary glands by developing the proper robot hardware, control algorithm, and computer vision models. My goal will be to implement improvements to the existing control algorithm to increase parallelization, add error recovery, and support the next generation hardware and computer vision efforts.
Figure 1. Robot system produced in late 2020, new hardware and software improvements have occurred since.
The malaria epidemic affects millions of people worldwide every year. Using mosquitos as its transmission vector, malaria is highly contagious, quickly mutating, and a major killer in parts of the world. Sanaria is one company attempting to tackle this crisis by producing a malaria vaccine. However, scaling up vaccine production presents a major bottleneck at the mosquito harvesting step. Working with Sanaria, our team at JHU is developing a system to automate mosquito harvesting by constructing a complex procedure to efficiently transport, decapitate, and extract the salivary glands. The control is implemented using the Robot Operating System (ROS) which sets up the hardware as individual services which an overall controller can call.
The lab is currently simultaneously developing software and hardware. Our first-generation controller created ROS node wrappers for each of the processes and implemented a basic overall controller. The second-generation controller will increase the robustness of overall controller by adding more error checks and recovery. The new controller will also incorporate the new mechanisms developed by the hardware team. A comprehensive flowchart has already been developed. Additionally, we will perform testing both in simulation and the physical world to verify and debug the controller.
We are using an open-source framework called Robot Operating System (ROS) to implement our robot controller. ROS is a collection of tools and libraries that simplify the complex task of robot control. At the core of ROS is the node, a logical unit that can communicate with other nodes using ROS messages and execute a script to move the robot, call a neural network, output to a GUI, and more.
In order to structure the code to allow for error recovery and ease of communication, we are employing a ROS package known as actionlib. Actionlib's ActionClient and ActionServer communicate via a “ROS Action Protocol” which is built on top of basic ROS messages. The client is the caller which requests the server, or the callee, to execute a task and return the result. Thus, the client-server architecture provides the basic functions to efficiently operate the robot system.
Figure 2: ROS ActionClient and ActionServer Relationship
Our lab has already implemented a similar controller without error recovery for our last hardware iteration using the client-server architecture. My project will focus on supporting the new hardware, adding error recovery, and parallelizing processes. Our new controller will be based on a flow chart (fig. 3) which has five parallel nodes which each control independent subsystems: the robot, gland extractor, turntable cleaner, turntable, and robot cleaner.
Figure 3: Mosquito Microdissection Robot Controller Flow Chart. Click for expanded image
Our lab has two means for debugging and testing. The first option is testing on the physical robot which is located in JHU LCSR robotorium and can be accessed and operated remotely as well. The second option is an ROS RVIZ simulation which I helped develop for the last hardware iteration. It needs to be updated to the newest hardware in order to be used for our purposes.
Above is a full map detailing the action client/server architecture for 2 nodes: Mosquito Pick-Place-Decapitate (MPPD) and gland extractor. Matching blue and green objects are action client/server pairs. They communicate via a topic shown with a “cloud object”. For an interactive diagram visit: LINK
I conducted testing on 30 mosquitoes and manually determined the success rate, error recovery rate, and throughput of various components in the system. The summary data is displayed in table \ref{table:data} which shows that the number of total successes (initial success count + recovery count), the step success rate, and the cumulative success rate.
The cumulative success rate for the MPPD node is only 53% largely because of the decapitate step with a individual success rate of only 67%. In our observation, the decapitate success rate could be improved through better placing of the mosquitoes between the blades. Often, mosquito bodies were above the blade's height. When the blades actuated, these mosquitoes do not become successfully decapitated; their bodies were simply pushed up during actuation which avoids the cut. The problem could be avoided with taller blades and the implementation of the decapitate validation step. Additional hardware to keep the mosquito bodies within the slots could also mitigate the issue.
Due to a lack of mature hardware and CV, we were only able to perform quantitative testing on error recovery for the pick and place step. However, error recovery allowed us to save 100% of missed mosquitoes by the the pick step, demonstrating the potential of the CV error detection and controller recovery.
Using a handheld timer, we determined the average time for the MPPD node to process one mosquito to be roughly 10s. The MPPD node operates for longer than the gland extraction node, thus we can use the MPPD node to estimate the mosquito throughput at 360 mosquitoes per hour (mph). The gripper actuation uses a slow linear servo, which could easily be replaced to remove around 2s per mosquito, improving the rate to 450mph. Since our goal rate is 600mph, we could investigate increasing the robot move speed and cutter speed to reach our goal.
Video 1: Error Recovery Video Demonstration Video 2: Parallel Operation of the MPPD, Extraction, and Turntable Nodes
The below table summarizes the dependencies, the need justification, resolution deadline, final status, and adjustments made. Because of the hardware team was not yet ready to deliver the robot cleaner and hardware, we modified our original minimum deliverable of five nodes (MPPD, Extract, Turntable, Robot Cleaner, Turntable Cleaner) to three nodes (MPPD, Extract, Turntable). Additionally, because the CV team delivered later than expected, I simply put placeholders (i.e. simply pause 3 seconds, then return success) in place of many CV components. The simulation was also delivered late by another team member because he was writing a paper. Thus, we adjusted our timeline to perform simulation testing concurrently with the physical testing, much later than planned.
Table 1: Dependencies Table
Last Updated 5.07.2021
Figure 4. Proposed vs Actual Timeline. The blue text represents the dates where portions of the minimium product were delivered. The sharded regions represent the originally planned schedule, and the outlined regions represent the actual development schedule.
I have attached links to two videos that are part of my deliverables for testing.
Video one: Parallelism Demonstration
Video two: Error Recovery Demonstration
The code is stored in a confidential GitLab repository.
