Contact Us
CiiS Lab
Johns Hopkins University
112 Hackerman Hall
3400 N. Charles Street
Baltimore, MD 21218
Directions

Lab Director
Russell Taylor
127 Hackerman Hall
rht@jhu.edu

Trace:

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
courses:456:2022:projects:456-2022-02:project-02 [2022/05/17 15:48] – [Technical Approach] kpineda3courses:456:2022:projects:456-2022-02:project-02 [2022/05/17 16:30] (current) – [Milestones and Status] kpineda3
Line 32: Line 32:
 We plan to integrate the gaze tracking component (shown in green) to the existing Microsoft PSI workflow. We hope to run this independently first to determine how informative the gaze fixations are without the influence of AUs. Some gaze fixation data from the following user study may need to be manually coded; if necessary, two independent coders will be used. We plan to integrate the gaze tracking component (shown in green) to the existing Microsoft PSI workflow. We hope to run this independently first to determine how informative the gaze fixations are without the influence of AUs. Some gaze fixation data from the following user study may need to be manually coded; if necessary, two independent coders will be used.
  
-{{  :courses:456:2022:projects:456-2022-02:workflow.png?750 |Microsoft PSI System Block Diagram with Gaze Tracker Component}}+{{  :courses:456:2022:projects:456-2022-02:workflow.png?650 |Microsoft PSI System Block Diagram with Gaze Tracker Component}}
  
 =====User Study===== =====User Study=====
Line 38: Line 38:
 We created an exploratory study design to analyze conceptual and physical robot errors during a human-robot collaborative packing task. We hoped to gather data on where people look during the task and where they look when the robot makes a mistake. While not a true factorial design, the breakdown of our study design conditions is shown in the figure below. We had four arrangements of two conditions, with a different type of robot error in each arrangement. We created an exploratory study design to analyze conceptual and physical robot errors during a human-robot collaborative packing task. We hoped to gather data on where people look during the task and where they look when the robot makes a mistake. While not a true factorial design, the breakdown of our study design conditions is shown in the figure below. We had four arrangements of two conditions, with a different type of robot error in each arrangement.
  
-A total of 6 participants were convenience sampled for our pilot study. They were tasked with completing a packing task with the Kinova robot. The participant used voice commands to direct the robot to pack certain items in its designated box. There was be a planned robot error that the participant was not aware of, allowing for a scenario that could generate a genuine human reaction to an unexpected event. The data was recorded via the Pupil Labs invisible gaze tracker eyeglasses, a microphone, and a verbal questionnaire. The invisible gaze tracker is automatically calibrated. The verbal questionnaire included questions about if the participants witnessed an error, what the error was, and its corresponding severity. +A total of 6 participants were convenience sampled for our pilot study. They were tasked with completing a packing task with the Kinova robot. The participant used voice commands to direct the robot to pack certain items in its designated box. There was a planned robot error that the participant was not aware of, allowing for a scenario that could generate a genuine human reaction to an unexpected event. The data was recorded via the Pupil Labs invisible gaze tracker eyeglasses, a microphone, and a verbal questionnaire. The invisible gaze tracker is automatically calibrated. The verbal questionnaire included questions about if the participants witnessed an error, what the error was, and its corresponding severity. 
    
-{{  :courses:456:2022:projects:456-2022-02:user_study.png?750 |User Study Design Condition Arrangements}}   +{{  :courses:456:2022:projects:456-2022-02:user_study.png?650 |User Study Design Condition Arrangements}}   
  
  
Line 46: Line 46:
  
    
-{{  :courses:456:2022:projects:456-2022-02:setup_fig.jpeg?550 |Experimental Setup}}   +{{  :courses:456:2022:projects:456-2022-02:setup_fig.jpeg?650 |Experimental Setup}}   
  
 +The data was recorded via the Pupil Labs invisible gaze tracker eyeglasses and a microphone. The invisible gaze tracker was automatically calibrated but recalibrated to each participants' gaze at the beginning of the study. While no paper questionnaire was used, the experimenters verbally asked the participants after the study if they witnessed an error, what the error was, and its corresponding severity. 
 +The recorded data was downloaded from the Pupil Labs cloud software and manually annotated using the datavyu software, as shown in the image below.  
 +
 +
 +{{  :courses:456:2022:projects:456-2022-02:datav.png?650 |Datavyu Video Coding Setup}}   
 +
 +The coder used the fixation circle provided by the pupil labs software to determine where the participant was looking. The coder then identified what object they were looking at during a timestamp range and labeled the timestamps accordingly to that object. They then matched up the corresponding time stamps to the fixation IDs with those associated timestamps in the data provided by pupil labs.   
 =====Results===== =====Results=====
 +
 +While debriefing our participants, we learned that they did not always perceive some of the robot errors as an error. During the wrong object condition, they did not perceive it as a robot error that the wrong object was selected. It seemed to be more of an inherent error related to the study. Also, the error of the box not being able to close due to the selected object being too large once again was not perceived as a robot error, but rather an error by whoever requested those specific items to be packed for the box of that size.  
 +
 +In analyzing our manually coded video data, we confirmed that gaze appears to be goal-oriented, as the literature indicates. Furthermore, gaze appeared to linger in an area where an error occurred (back at the object if the error involved missing the object, or at the gripper if the object was dropped). There also appeared to be occasional gaze shifts during the study. These mostly consisted of shifts back to the set of instructions or the tablet after an error occurred, or between the object involved in the error and the robot/gripper.  
 +
 +Since this analysis of results is qualitative, we decided our study could be improved if we implemented a more quantitative metric for gaze analysis to better categorize the gaze fixations. Thus, we began looking into how to measure gaze velocity during gaze shifts in real-time for our current workflow which only consisted of 2D gaze data with respect to the camera.  
 +
 +=====Limitations=====
 +
 +In order to measure gaze velocity, we needed to perform 3D localization of gaze within the world coordinate system. Thus, accurate detection of AR markers is essential for this process. However, we experienced several technical difficulties while learning how to measure gaze fixation velocity from the pupil labs gaze tracker. While first using AprilTags, we learned that they were not always detected during post-processing, and thus switched over to ArUco Markers. While these resulted in better detection than the AprilTags, we learned that there were still some issues with detecting the ArUco markers despite their size. We also encountered some inconsistencies in our transformation matrices for the markers/tags across images while using individual markers/tags. Thus, we transitioned to using an ArUco Grid Board detection rather than an individual ArUco marker detection. We are currently in the process of setting up the ArUco Grid Board in our study layout for automatic gaze velocity detection across fixations and hope to conduct our future user studies using this technical approach.
 +
 +\subsection{Limitations}
 +In order to measure gaze velocity, we needed to perform 3D localization of gaze within the world coordinate system. Thus, accurate detection of AR markers is essential for this process. However, we experienced several technical difficulties while learning how to measure gaze fixation velocity from the pupil labs gaze tracker. While first using AprilTags, we learned that they were not always detected during post-processing, and thus switched over to ArUco Markers. While these resulted in better detection than the AprilTags, we learned that there were still some issues with detecting the ArUco markers despite their size. We also encountered some inconsistencies in our transformation matrices for the markers/tags across images while using individual markers/tags. Thus, we transitioned to using an ArUco Grid Board detection rather than an individual ArUco marker detection. We are currently in the process of setting up the ArUco Grid Board in our study layout for automatic gaze velocity detection across fixations, and hope to conduct our future user studies using this technical approach.
 +
 +=====Discussion=====
 +While our qualitative analysis provided some insightful results, we realized it was essential to have a quantitative metric for gaze fixations. Due to the video coding being time-consuming (at least 3 hours of coding per participant), we determined it was necessary to have a quantitative metric that could automatically provide additional information surrounding the fixation data in order to increase the scale of our user study. We hoped to soon run the gaze component independently in Microsoft PSI first to determine how informative the gaze fixations are without the influence of AUs. Once our automatic gaze velocity measure is working and it is integrated into Microsoft PSI, we can consider if implementing an ML algorithm to automatically detect the error would provide accurate results. 
 +
 +=====Future Works=====
 +Our next steps, apart from finishing the physical setup for our new ArUco approach, involve finalizing our user study with errors that can be perceived more clearly and explicitly as errors by the participants (fixing the confusing objects by the participants and clarifying that the robot is a supposed packing expert). We also intend to include a full written questionnaire that will include questions regarding basic demographic information, and if the participants witnessed an error and its corresponding severity. We then hope to run a full-scale user study.   
 +
 ======Dependencies====== ======Dependencies======
 {{:courses:456:2022:projects:456-2022-02:dependency_2.0.png?750}} {{:courses:456:2022:projects:456-2022-02:dependency_2.0.png?750}}
Line 54: Line 81:
 ======Milestones and Status ====== ======Milestones and Status ======
  
-{{:courses:456:2022:projects:456-2022-02:milestone_2.0.png?750}}+{{:courses:456:2022:projects:456-2022-02:milestones3.png?750}}
  
 ======Reports and presentations====== ======Reports and presentations======
Line 70: Line 97:
     * Selected paper: {{https://harplab.github.io/assets/pubs/fja_rss2018_aronson.pdf | Gaze for Error Detection During Human-Robot Shared Manipulation}}     * Selected paper: {{https://harplab.github.io/assets/pubs/fja_rss2018_aronson.pdf | Gaze for Error Detection During Human-Robot Shared Manipulation}}
   * Project Final Presentation   * Project Final Presentation
-    * {{:courses:456:2022:projects:456-2022-02:final_poster_pdf.pdf|PDF of Poster}}+    * {{:courses:456:2022:projects:456-2022-02:poster_pdf_pineda.pdf|PDF of Poster}}
   * Project Final Report   * Project Final Report
     * {{:courses:456:2022:projects:456-2022-02:cisii_final_report_pineda.pdf|Final Report}}     * {{:courses:456:2022:projects:456-2022-02:cisii_final_report_pineda.pdf|Final Report}}
courses/456/2022/projects/456-2022-02/project-02.1652802504.txt.gz · Last modified: by kpineda3




ERC CISST    LCSR    WSE    JHU