Contact Us
CiiS Lab
Johns Hopkins University
112 Hackerman Hall
3400 N. Charles Street
Baltimore, MD 21218
Directions
Lab Director
Russell Taylor
127 Hackerman Hall
rht@jhu.edu
This is an old revision of the document!
Last updated: Date and time
The goal of this project is to evaluate human eye gaze behavior in response robot errors. We hope to understand how human eye gaze may be incorporated into a robot error detection system.
Background and Significance
In the field of human-robot interaction (HRI), researchers focus on how humans interact with robots and the different conclusions we can draw upon these interactions. HRI is a growing field with the potential for many applications in medicine or health care. Robots could assist with kitting in surgical settings, help patients in hospitals by providing bedside assistance, or even interact with COVID-19 patients in the ICU to help minimize the risk of exposure for healthcare workers. Nevertheless, robots are fallible and can make mistakes when executing their actions.
As the potential for robotic applications in the real world begins to grow, it is crucial to study how humans react to robot errors to gain a better understanding of humans' interactions with robots. Studying a human’s physical and social responses to robot errors can inform researchers and roboticists of how not only robot errors may affect the trust humans place in a robot’s actions, but also of ways to make future predictions of robot errors. If a robotic system does not have a built-in error detection system in place, recognizing that an error has occurred from a human’s reaction is critical information; the feedback from such a detection system can allow a robot to minimize the severity of an error, allow a robot to correct its error, or even implement early stopping if the error is detected with enough notice.
A robot error detection system, created by Stiber, detects a robot error in real time by analyzing facial action units (AUs). Initial studies from Stiber show that people react very differently to physical mistakes executed by robots. Other previous works indicate that gaze can be a potential metric for detecting robot errors. Depending on the task at hand and the users present, there may be a variety of human reactions. See Figure 1. The overall workflow of this current system consists of using two cameras aimed at a human’s face, determining the current AUs present, detecting if an error has occurred based on these AUs, and logging the result with the corresponding timestamps. The ML algorithm that detects the error has been trained on a set of data of 19 participants reacting to robotic errors. The data collected was manually coded frame by frame in Microsoft PSI by two independent coders.
Aims
The goal of this project is to introduce human eye gaze as a potential metric of robot error detection to an automatic robot error detection system created by Maia Stiber. The current AU detection system will still function correctly if a user wears glasses. Pupil Labs has released a mobile gaze tracker in the form of a pair of eyeglasses for a human to wear. We intend to have users wear these glasses while interacting with a robot as a robot error occurs to gain a better understanding of their gaze patterns. We hope to learn whether gaze can be an informative metric in robot error detection scenarios. Based on different types of errors, we want to investigate if human eye gaze is consistent; if there is a noticeable pattern that we could use down the line. Furthermore, we are interested in discovering if fixation points will appear that can inform us later in future scenarios. Thus, the aims of this project consist of collecting data to understand human gaze reactions associated with physical robotic error, adding the gaze tracker data as an additional component to the Microsoft PSI system pipeline, and creating a ML algorithm with the data collected so that it will automatically detect the robot error as it occurs.
We plan to integrate the gaze tracking component (shown in green) to the existing Microsoft PSI workflow. We hope to run this independently first to determine how informative the gaze fixations are without the influence of AUs. Some gaze fixation data from the following user study may need to be manually coded; if necessary, two independent coders will be used.
We created an exploratory study design to analyze conceptual and physical robot errors during a human-robot collaborative packing task. We hoped to gather data on where people look during the task and where they look when the robot makes a mistake. While not a true factorial design, the breakdown of our study design conditions is shown in the figure below. We had four arrangements of two conditions, with a different type of robot error in each arrangement.
A total of 6 participants were convenience sampled for our pilot study. They were tasked with completing a packing task with the Kinova robot. The participant used voice commands to direct the robot to pack certain items in its designated box. There was be a planned robot error that the participant was not aware of, allowing for a scenario that could generate a genuine human reaction to an unexpected event. The data was recorded via the Pupil Labs invisible gaze tracker eyeglasses, a microphone, and a verbal questionnaire. The invisible gaze tracker is automatically calibrated. The verbal questionnaire included questions about if the participants witnessed an error, what the error was, and its corresponding severity.
