Contact Us
CiiS Lab
Johns Hopkins University
112 Hackerman Hall
3400 N. Charles Street
Baltimore, MD 21218
Directions

Lab Director
Russell Taylor
127 Hackerman Hall
rht@jhu.edu

Trace:

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
courses:456:2022:projects:456-2022-02:project-02 [2022/05/17 15:53] – [User Study] kpineda3courses:456:2022:projects:456-2022-02:project-02 [2022/05/17 16:30] (current) – [Milestones and Status] kpineda3
Line 56: Line 56:
 The coder used the fixation circle provided by the pupil labs software to determine where the participant was looking. The coder then identified what object they were looking at during a timestamp range and labeled the timestamps accordingly to that object. They then matched up the corresponding time stamps to the fixation IDs with those associated timestamps in the data provided by pupil labs.    The coder used the fixation circle provided by the pupil labs software to determine where the participant was looking. The coder then identified what object they were looking at during a timestamp range and labeled the timestamps accordingly to that object. They then matched up the corresponding time stamps to the fixation IDs with those associated timestamps in the data provided by pupil labs.   
 =====Results===== =====Results=====
 +
 +While debriefing our participants, we learned that they did not always perceive some of the robot errors as an error. During the wrong object condition, they did not perceive it as a robot error that the wrong object was selected. It seemed to be more of an inherent error related to the study. Also, the error of the box not being able to close due to the selected object being too large once again was not perceived as a robot error, but rather an error by whoever requested those specific items to be packed for the box of that size.  
 +
 +In analyzing our manually coded video data, we confirmed that gaze appears to be goal-oriented, as the literature indicates. Furthermore, gaze appeared to linger in an area where an error occurred (back at the object if the error involved missing the object, or at the gripper if the object was dropped). There also appeared to be occasional gaze shifts during the study. These mostly consisted of shifts back to the set of instructions or the tablet after an error occurred, or between the object involved in the error and the robot/gripper.  
 +
 +Since this analysis of results is qualitative, we decided our study could be improved if we implemented a more quantitative metric for gaze analysis to better categorize the gaze fixations. Thus, we began looking into how to measure gaze velocity during gaze shifts in real-time for our current workflow which only consisted of 2D gaze data with respect to the camera.  
 +
 +=====Limitations=====
 +
 +In order to measure gaze velocity, we needed to perform 3D localization of gaze within the world coordinate system. Thus, accurate detection of AR markers is essential for this process. However, we experienced several technical difficulties while learning how to measure gaze fixation velocity from the pupil labs gaze tracker. While first using AprilTags, we learned that they were not always detected during post-processing, and thus switched over to ArUco Markers. While these resulted in better detection than the AprilTags, we learned that there were still some issues with detecting the ArUco markers despite their size. We also encountered some inconsistencies in our transformation matrices for the markers/tags across images while using individual markers/tags. Thus, we transitioned to using an ArUco Grid Board detection rather than an individual ArUco marker detection. We are currently in the process of setting up the ArUco Grid Board in our study layout for automatic gaze velocity detection across fixations and hope to conduct our future user studies using this technical approach.
 +
 +\subsection{Limitations}
 +In order to measure gaze velocity, we needed to perform 3D localization of gaze within the world coordinate system. Thus, accurate detection of AR markers is essential for this process. However, we experienced several technical difficulties while learning how to measure gaze fixation velocity from the pupil labs gaze tracker. While first using AprilTags, we learned that they were not always detected during post-processing, and thus switched over to ArUco Markers. While these resulted in better detection than the AprilTags, we learned that there were still some issues with detecting the ArUco markers despite their size. We also encountered some inconsistencies in our transformation matrices for the markers/tags across images while using individual markers/tags. Thus, we transitioned to using an ArUco Grid Board detection rather than an individual ArUco marker detection. We are currently in the process of setting up the ArUco Grid Board in our study layout for automatic gaze velocity detection across fixations, and hope to conduct our future user studies using this technical approach.
 +
 +=====Discussion=====
 +While our qualitative analysis provided some insightful results, we realized it was essential to have a quantitative metric for gaze fixations. Due to the video coding being time-consuming (at least 3 hours of coding per participant), we determined it was necessary to have a quantitative metric that could automatically provide additional information surrounding the fixation data in order to increase the scale of our user study. We hoped to soon run the gaze component independently in Microsoft PSI first to determine how informative the gaze fixations are without the influence of AUs. Once our automatic gaze velocity measure is working and it is integrated into Microsoft PSI, we can consider if implementing an ML algorithm to automatically detect the error would provide accurate results. 
 +
 +=====Future Works=====
 +Our next steps, apart from finishing the physical setup for our new ArUco approach, involve finalizing our user study with errors that can be perceived more clearly and explicitly as errors by the participants (fixing the confusing objects by the participants and clarifying that the robot is a supposed packing expert). We also intend to include a full written questionnaire that will include questions regarding basic demographic information, and if the participants witnessed an error and its corresponding severity. We then hope to run a full-scale user study.   
 +
 ======Dependencies====== ======Dependencies======
 {{:courses:456:2022:projects:456-2022-02:dependency_2.0.png?750}} {{:courses:456:2022:projects:456-2022-02:dependency_2.0.png?750}}
Line 61: Line 81:
 ======Milestones and Status ====== ======Milestones and Status ======
  
-{{:courses:456:2022:projects:456-2022-02:milestone_2.0.png?750}}+{{:courses:456:2022:projects:456-2022-02:milestones3.png?750}}
  
 ======Reports and presentations====== ======Reports and presentations======
Line 77: Line 97:
     * Selected paper: {{https://harplab.github.io/assets/pubs/fja_rss2018_aronson.pdf | Gaze for Error Detection During Human-Robot Shared Manipulation}}     * Selected paper: {{https://harplab.github.io/assets/pubs/fja_rss2018_aronson.pdf | Gaze for Error Detection During Human-Robot Shared Manipulation}}
   * Project Final Presentation   * Project Final Presentation
-    * {{:courses:456:2022:projects:456-2022-02:final_poster_pdf.pdf|PDF of Poster}}+    * {{:courses:456:2022:projects:456-2022-02:poster_pdf_pineda.pdf|PDF of Poster}}
   * Project Final Report   * Project Final Report
     * {{:courses:456:2022:projects:456-2022-02:cisii_final_report_pineda.pdf|Final Report}}     * {{:courses:456:2022:projects:456-2022-02:cisii_final_report_pineda.pdf|Final Report}}
courses/456/2022/projects/456-2022-02/project-02.1652802820.txt.gz · Last modified: by kpineda3




ERC CISST    LCSR    WSE    JHU