Last updated: 05/01/2022 08:00PM

An automatic mosquito salivary gland extraction and collection robot platform requires high precision of robot movements. Frequent calibration is often required as the robot is being developed. However, this procedure is segmented and is both time and labor intensive. This projects aims to build an automatic calibration procedure that can fully calibrate robot components on operator call.

• Students: Miles Liu
• Mentor(s): Balazs Vagvolgyi, Anna Goodridge

Sanaria Inc. and CIIS teams have been developing a robot platform to automate salivary gland extraction for vaccine production. This robotic platform can significantly increase gland production rates. As the robot requires high precision movements, it will also require frequent manual calibration for:

1. Handeye calibration (camera to robot arm)
2. Robot tool path
3. Robot homing

While there are existing procedures for each of these calibration components, they are fully manual processes, which can be time and labor intensive.

This goal of this project is to streamline and simplify calibration routines and components, and create a well integrated single calibration package that can handle the majority of calibration tasks automatically. This project also aims to rethink how robot tool paths are defined, and implement a solution that can be maintained + interpreted easier by human developers.

• Minimum: (Expected by Mid March)
  1. Streamlined calibration components integrated with ROS Complete
  2. Prototype implementation of new “calibration tree” Complete
  3. Significant progress with development of robot homing calibration procedures Complete
  4. Documentation Complete

• Expected: (Expected by Mid April)
  1. A single calibration workflow & GUI that covers all calibration tasks Partially Complete
  2. Deployment version of the “calibration tree” Partially Complete

• Maximum: (Expected by Early May)
  1. A calibration workflow that does not require any operator input
  2. A vision-based calibration validation system
  3. Incorporate Sanaria feedback

For the robot homing component, we opted to develop a visual servoing algorithm for moving the robot to any arbitrary position in the cartesian robot workspace. The visual servoing procedure uses two cameras (one overhead and one at an arbitrary position) to capture two 2D projections of the robot’s tooltip and goal positions in images. The pixel positions (u, v) of the goal and current gripper tooltip can then be extracted from the images using an automated computer vision component or a manual procedure. The algorithm can then iteratively loop through motion steps to move the gripper tooltip to the target position, until the gripper tooltip and the target on both images converge, meaning that the tooltip is exactly on top of the goal position. At each of these motion steps, the robot performs a small motion in each of the axis (X, Y, and Z), finds the new tool position in the images, and determines the gradient of the tooltip’s pixel locations with respect to the small motion in each of the robot’s axis in the form of a Jacobian. This Jacobian can then be fed into a minimization algorithm to obtain a new position in robot coordinates that would minimize the distance between the goal position’s image projection pixel coordinate and the new robot position’s image projection pixel coordinate. The robot homing procedure in two cameras

For the robot tool path calibration component, the goal is to develop a more intuitive, human understandable, and unified data structure for the representation of waypoints used in the robot’s normal operation. In order to achieve this, we developed a custom data structure that does not explicitly define each robot waypoint as its own set of absolute coordinates, but rather as a “directional graph” of positions relative to each other and one single absolute “home position”. This calibration graph is stored in a human readable JSON file for ease of calibration, and is loaded by a ROS node upon robot system startup. The graph developed for the current robot's operational trajectory. Each node is a specific waypoint along some trajectory.

For the robot's handeye calibration, since there is already a calibration procedure that can handle the bulk of the calibration, our team would package the script such that it can be integrated into the ROS framework for the robot. This existing procedure moves the robot to N points in a rectangular grid near the 2D plane where the mosquito manipulation occurs, detects the robot gripper tooltip position at each of these points, and fits the resulting point correspondences using a Bernstein polynomial fit. Coefficients for this polynomial fit is then saved in a config file. This config file is then opened and read by a dedicated ROS service node that can translate between robot and camera coordinates.

All three of the components above are to be implemented as each its own ROS node, along with a calibration GUI that allows for direct supervision of any calibration process by the operator. Below are GUIs developed in rqt for the robot homing and handeye calibration components. Screenshot of the handeye calibration GUI

Screenshot of the robot homing GUI

Significant dependencies:

1. Availability of work station, robot core components such as cameras, robot arm, etc.
   • While a significant amount of work can be done online, not having the physical hardware available would hinder the project's progress for further system integration and validation. It will also prevent the team from being able to develop a proper vision-based calibration algorithm.
   • Resolved Scheduling calendar and robot usage logs
2. Frequency of hardware changes
   • If during the testing and validation process, other teams implement frequent hardware changes, it may affect our ability to collect meaningful data.
   • Resolved Robot usage logs and frequent communication with lab team
3. Tooltip detection algorithm Only affects maximum deliverables
   • A tooltip detection algorithm is critical to fully automating vision based calibration procedures, but there are workarounds to manually label tooltips.
4. Robot controller PID tuning Only affects maximum deliverables
   • Affects the accuracy achievable with calibration. May or may not be critical to fully automated calibration, further investigation is needed.

1. Milestone name: Update existing calibration procedures
   • Planned Date: 03/07/2022
   • Expected Date: 03/07/2022
   • Status: Complete
2. Milestone name: Prototype Implementation of main calibration components
   • Planned Date: 03/31/2022
   • Expected Date: 03/31/2022
   • Status: Complete
3. Milestone name: System integration and verification testing complete
   • Planned Date: 04/20/2022
   • Expected Date: 04/20/2022
   • Status: Partially complete
4. Milestone name: Finalize system documentation + Final project presentation materials
   • Planned Date: 05/04/2022
   • Expected Date: 05/04/2022
   • Status: Complete

• Project Plan
  • liu_cis_ii_project_plan.pdf
  • liu_cis_ii_project_proposal.pdf
• Project Background Reading
  • liu_cis_ii_background_reading_critical_review.pdf
• Project Checkpoint
  • liu_cis_ii_checkpoint_presentation.pdf
• Paper Seminar Presentations
  • liu_cis_ii_background_presentation_slides.pdf
• Project Final Presentation
  • PDF of Poster
• Project Final Report
  • liu_cis_ii_final_report.pdf
  • links to any appendices or other material

H. Wu, J. Mu, T. Da, M. Xu, R. H. Taylor, I. Iordachita, and G. S. Chirikjian, “Multi-mosquito object detection and 2D pose estimation for automation of PfSPZ malaria vaccine production”, in IEEE 15th International Conference on Automation Science and Engineering (CASE), Vancouver, BC, August 22-26, 2019. pp. 411-417.

W. Li et al., “Automated Mosquito Salivary Gland Extractor for PfSPZ-based Malaria Vaccine Production,” 2021 IEEE International Conference on Robotics and Automation (ICRA), 2021, pp. 866-872, doi: 10.1109/ICRA48506.2021.9560959.

Code repositories: All of the code for the robot, including the code developed as a part of this project, are stored in this repository group on GitLab: https://git.lcsr.jhu.edu/mosquitoproject

The code repositories directly containing work covered by this project are:

https://git.lcsr.jhu.edu/mosquitoproject/rqt_sanaria_robot_controller https://git.lcsr.jhu.edu/mosquitoproject/sanaria_calibration_tools https://git.lcsr.jhu.edu/mosquitoproject/sanaria_transformation_graph https://git.lcsr.jhu.edu/mosquitoproject/sanaria_handeye

Documentation on system design is in the documentation repository: https://git.lcsr.jhu.edu/mosquitoproject/sanaria_docs