Trace:

Differences

This shows you the differences between two versions of the page.

--- courses:456:2022:projects:456-2022-10:project-10 [2022/04/29 22:00] – [Technical Approach] achen134
+++ courses:456:2022:projects:456-2022-10:project-10 [2022/05/16 20:18] (current) – [Reports and presentations] achen134
@@ Line 1: / Line 1: @@
 ======Multisensory Navigational Aid for Visual Prosthesis Users======
-**Last updated: 14 April 2022 at 10:32**
+**Last updated: 30 April 2022 at 18:22**
@@ Line 7: / Line 7: @@
   * **Students:** An Chi Chen
-  * **Mentor(s):** Seth Billings, Chi Ewulum
+  * **Mentor(s):** Dr Seth Billings, Chigozie Ewulum
@@ Line 14: / Line 14: @@
 The Argus II is a retinal prosthesis system for patients with late stage Retinitis Pigmentosa (loss or breakdown of retina cells).  The Argus II system is composed of an electrode array that is implanted behind the users' eye as well as a camera mounted onto a pair of glasses along with the required processing system.
-{{:courses:456:2022:projects:456-2022-10:argusii.png?400|}}
+{{ :courses:456:2022:projects:456-2022-10:argusii.png?400 |}}
 Artificial vision is provided to the users by using input from the camera to stimulate the users' viable retinal cells which is then perceived as light patterns by the user.
-{{:courses:456:2022:projects:456-2022-10:artificialvision.png?600|}}
+{{ :courses:456:2022:projects:456-2022-10:artificialvision.png?600 |}}
 There are limitations to this system that hinder the users ability to independently navigate a space.  Some of these limitations are:
@@ Line 43: / Line 43: @@
     - Evaluation of that system's performance in terms of target navigation and obstacle avoidance as a results of the participant testing
-  * **Adjusted Maximum:** (Completed on 1 May 2022)
+  * **Adjusted Maximum:** (Completed on 30 April 2022)
     - Integrated haptic and auditory feedback system with simulated Argus II feedback on virtual reality headset
+======Dependencies======
+There are not many dependencies for this project however they will affect the deliverables if they are not resolved in a timely manner.  For both the haptic and auditory feedback systems the SLAM-based navigational system is required for full integration with the Argus II system.  This also includes the necessary hardware such as cameras.  The SLAM system itself has been developed and is currently in the refinement phase.  The required cameras for the system have been obtained.  The only outstanding hardware are the camera mounts – these are currently being 3D printed.  If these mounts are not completed by 10 April 2022, then the subsequent tasks (testing of gaze and path guidance and object localisation in the real world and participant testing) will be delayed.  If the delay is substantial (past 17 April 2022) it may not be possible to complete the participant testing in this timeframe as there should be at least two weeks of participant testing.
+For the participant testing, the IRB training will need to be completed.  Whether testing can take place also depends on participant availability.  The IRB training can be completed in advance and thus will be done so.  A testing protocol currently exists, and the team members will simply need to be added to it to be able to perform participant testing.  Furthermore, four weeks have been allocated to acquire participants.  Ideally the testing will be done with Argus II users, however if they are not available it is possible to perform participant testing with seeing individuals.  The system developed by APL does also connects to a VR headset which allows seeing individuals to see what Argus II users perceive.  If it is still not possible to acquire the participants by 17 April 2022 then the allocated time for participant testing will be reduced and potentially may not be completed in this project timeline.
+The table below summarises the dependencies mentioned above.
+{{ :courses:456:2022:projects:456-2022-10:dependencies.png?650 |}}
+*Latest Date = If dependency not resolved by this date it will affect the deliverables
+The following table shows the final status of the dependencies at the end of the project timeline (1 May 2022).
+{{ :courses:456:2022:projects:456-2022-10:dependenciesatend.png?650 |}}
+As can be seen the IRB and participant availability dependency were not achieved.  This was due to the fact that the existing IRB protocol only includes recruiting participants from APL or Argus II users.  The developed supplementary feedback systems, though integrated, are not in a complete enough state to be tested with Argus II users at this stage.  In addition, there was not enough time during this project timeline to execute the fallback plan for participant testing which was to test the developed integrated system with APL employees.  As seen above under the deliverables, the maximum deliverable was adjusted due to these time constraints and IRB dependency not being met.
 ======Technical Approach======
@@ Line 52: / Line 70: @@
 The haptic feedback system’s hardware consists of a headband fitted with 8 repositionable haptic actuators which connects to a custom 8-channel haptic driver.  The function and objective for this haptic system is to guide the users’ gaze towards the direction of their target (e.g. a door).  By doing so, this should assist the users in finding their targets and navigating to them more effectively.
-{{:courses:456:2022:projects:456-2022-10:haptic.png?600|}}
+{{ :courses:456:2022:projects:456-2022-10:haptic.png?600 |}}
 The first step in developing this system is to formulate an evaluation method and metrics.  These will be used to determine if the system is intuitive and effective in relaying the directional instructions.  Following so, there are a few options which need to be explored to develop the haptic feedback system.  ERMs and LRAs are two common haptic actuators, thus, it must be determined which type is best suited for this application.  Both have their advantages, and the choice will be dependent on which type provides a clearer, more distinct feedback.  Furthermore, there are various haptic patterns which can be employed to relay a directional instruction.  A static pattern involves only actuating motors on the side the users should direct their gaze.  Whilst a dynamic pattern relays a direction by using perceived motion – the haptic motors are actuated in a specific order, such as right to left to relay a left direction.  How intuitive these haptic patterns are to users will determine which one will be used.
-{{:courses:456:2022:projects:456-2022-10:haptic_pattern.png?300|}}
+{{ :courses:456:2022:projects:456-2022-10:haptic_pattern.png?300 |}}
 It is also worth considering the use of the haptic feedback system for path guidance, as opposed to just gaze direction guidance.  For a path guidance objective, the haptic feedback will continually update as the users move their heads to continually guide the users towards their target.  After the haptic system is developed it will be integrated with the SLAM navigation system to test its real-world gaze direction and path guidance functionalities.
@@ Line 64: / Line 82: @@
 The hardware for the auditory feedback system is a pair of open-ear bone conduction headphones.  These headphones were chosen with the aim to not overwhelm the users’ senses, due to the open-ear nature of these headphones they can still hear noises from the world.  The function and objective for this system is to provide users with the location of potential obstacles using sound.  The name of an obstacle will be played to the users through the headphones and sound like it is coming from the direction of the objects.  This mitigates the need for users to manually scan their surroundings for obstacles before moving.
-{{:courses:456:2022:projects:456-2022-10:auditory.png?500|}}
+{{ :courses:456:2022:projects:456-2022-10:auditory.png?500 |}}
 Similar to the development of the haptic system, the first step in developing the auditory feedback system will be to formulate an evaluation method and metrics which will be used to determine the accuracy and effectiveness of the system’s localisation functionality.  Following so, methods to individualise head related transfer functions (HRTFs) will be explored.  HRTFs are what aids individuals in determining the location of a sound but are unique to each individual as they are dependent on various factors such as the shape of the head.  Thus, generalized HRTFs do not translate well with everyone.  There is especially a discrepancy between determining whether a simulated sound is coming from the front or back as well as up or down.  Currently, individualised HRTFs are not well developed, though there have been various methods suggested for creating personalization.  Some of these methods will be explored as well as determining the possibility of using calibration to personalize a set of generalised HRTFs.  Once the auditory feedback system has been developed it will also be integrated with the SLAM navigation system to test its real-world, real-time object localisation.
@@ Line 74: / Line 92: @@
 **Visual Feedback System**
-This feedback system will not form part of the supplementary system for Argus II users however it is a useful system to use when sighted users test the Argus II feedback system.  This feedback system attempts to emulate what Argus II users see through the use of the virtual reality head-mounted-display, Oculus.  A grid of 6x10 pixels was used as the viewing display – simulating what is seen by Argus II users.  This system was used to display the landmarks that come into the field of view (FOV) of the user.  In order to achieve this the horizontal and vertical angles between where the user is looking and the landmarks were determined (detailed in section 2.4).  If the angles fell within the visible FOV they were displayed at the correct location determined by the horizontal and vertical angles.
+This feedback system will not form part of the supplementary system for Argus II users however it is a useful system to use when sighted users test the Argus II feedback system.  This feedback system attempts to emulate what Argus II users see through the use of the virtual reality head-mounted-display, Oculus.  A grid of 6x10 pixels was used as the viewing display – simulating what is seen by Argus II users.  This system was used to display the landmarks that come into the field of view (FOV) of the user.  In order to achieve this the horizontal and vertical angles between where the user is looking and the landmarks were determined (detailed in section 2.4).  If the angles fell within the visible FOV they were displayed at the correct location determined by the horizontal and vertical angles.  This is illustrated in the figure below.
-======Dependencies======
+{{ :courses:456:2022:projects:456-2022-10:visualfeedbackimage.png?400 |}}
-There are not many dependencies for this project however they will affect the deliverables if they are not resolved in a timely manner.  For both the haptic and auditory feedback systems the SLAM-based navigational system is required for full integration with the Argus II system.  This also includes the necessary hardware such as cameras.  The SLAM system itself has been developed and is currently in the refinement phase.  The required cameras for the system have been obtained.  The only outstanding hardware are the camera mounts – these are currently being 3D printed.  If these mounts are not completed by 10 April 2022, then the subsequent tasks (testing of gaze and path guidance and object localisation in the real world and participant testing) will be delayed.  If the delay is substantial (past 17 April 2022) it may not be possible to complete the participant testing in this timeframe as there should be at least two weeks of participant testing.
-For the participant testing, the IRB training will need to be completed.  Whether testing can take place also depends on participant availability.  The IRB training can be completed in advance and thus will be done so.  A testing protocol currently exists, and the team members will simply need to be added to it to be able to perform participant testing.  Furthermore, four weeks have been allocated to acquire participants.  Ideally the testing will be done with Argus II users, however if they are not available it is possible to perform participant testing with seeing individuals.  The system developed by APL does also connects to a VR headset which allows seeing individuals to see what Argus II users perceive.  If it is still not possible to acquire the participants by 17 April 2022 then the allocated time for participant testing will be reduced and potentially may not be completed in this project timeline.
-The table below summarises the dependencies mentioned above.
-{{:courses:456:2022:projects:456-2022-10:dependencies.png?650|}}
+======Results======
+**Haptic Feedback System**
+Through the iterative testing it was found that static patterns were optimal for relaying the left and right instructions.  This was sufficient and intuitive enough to indicate to users in which direction they should look.  All 4 of the actuators located around the temporal and back of the head were utilised for this group of instructions thus making it possible to indicate a direction located towards the back of the head.
+Dynamic patterns were employed to relay the up and down instructions.  Interestingly, when only the 3 actuators located along the top of the head were used in the dynamic patterns the interpretation of the haptic pattern became unclear – potentially due to the lower density of actuators along the top of the head.  Thus, these patterns make use of all 7 actuators actuated in a specific order.  By using all 7 actuators, the feedback from the haptic pattern felt more distinct and clearer in relaying a directional instruction.  The up pattern starts with the actuators closest to the front of the head and moves towards the back, whilst the down pattern starts at the back of the head and ends in the front.
+{{ :courses:456:2022:projects:456-2022-10:hapticpatterns.png?600 |}}
+During the testing, it was realised that the duration of a haptic pattern should be as short as possible to allow for a fast update time.  If this was not satisfied, there was a noticeable delay between a turning of the head and the next correct haptic instruction which results in an overshoot of head turning movement and creates confusion.  Thus, for the static patterns the durations were set to 0.015 seconds.  This may seem like a short duration but the haptic patterns are repeated continuously until the target gaze direction is reached.  For the dynamic patterns, it is important that the durations are still set at a suitable time which makes them discernible as the directional instruction which they are relaying.  A total duration of 0.18 seconds was found to be a sufficient duration that satisfies these criteria.
+Furthermore, it was found that it is important that the haptic actuators are flush and are placed with a bit of pressure against the head in order to localise them properly.  Additionally, hair density may result in a reduced ability to determine which haptic actuator is on.  This potential issue could be solved by ensuring the haptic actuators are very close to the surface of the head.
+**Auditory Feedback System**
+A tournament to find a best fitting HRTF was set up using Unity with the SOFAlizer plugin [5] which facilitated switching between different HRTF profiles with ease.  The HRTF contenders were obtained from the open-source databases by MIT, CIPIC and RIEC [6, 7, 8].  A total of 30 HRTF profiles were used during this evaluation.  A completely accurate best fitting HRTF profile match was not found.  In essence, of the HRTF profiles examined, none overcame the front/back discrepancy described earlier.  Furthermore, this process was rather tedious as each HRTF had to be tested with the sound source located at varying position – the effort required did not equate to an extremely well fitted HRTF.
+The second approach of using the application that generates an individualised HRTF based off of measurements of the ear was a much more concise process.  After providing the input of a few measurements of the ear and head the application produced a comparable HRTF to that of the first individualisation method; although it was still not perfect.  Nevertheless, as the focus for this project was on the feedback algorithm, and not the accuracy of individualisation of an HRTF profile, the HRTF obtained in the second method was used in the subsequent feedback algorithm testing.
+Spatialised sounds were able to be accurately created and played through the bone conduction headphones.  The position of the landmarks (ArUco markers) located along the side of the user could be successfully located solely based-off of the sound.  Visual feedback provides assistance in locating sound sources from the front and back due to the front/back discrepancy, however, the use of a well-fitted HRTF profile would mitigate this issue.
+The distances, in feet, between the user and the landmark(s) were also announced to the users.  This was computed by simply calculating the norm of the vector between the user body and the landmark.  It was found that if more than three identified obstacles (often will be the case), including their distances, were announced created a larger load on the senses and could potentially disengage the users.  Therefore, in order to reduce the cognitive load, the algorithm was updated such that only obstacles within 10 feet of the body were announced.  It was also found that the listing of landmark names along with their distances is slow and users can move rather quickly.  Thus, the distances of the obstacles from the user were only announced every third iteration of the loop and not every single time the obstacles were listed in order to ensure adequate and accurate auditory feedback is provided.
+**Visual Feedback System**
+The horizontal and vertical FOVs had to be fine-tuned to accurately reflect the landmark positions in real life.  The final FOVs are:
+Horizontal FOV = 80°
+Vertical FOV = 60°.
+The figure below shows that the landmarks within the user’s FOV were successfully placed and shown on the virtual reality headset display.
+{{ :courses:456:2022:projects:456-2022-10:vr.png?400 |}}
-*Latest Date = If dependency not resolved by this date it will affect the deliverables
 ======Milestones and Status ======
@@ Line 94: / Line 138: @@
   - Milestone name:  Formulate evaluation method and metrics
     * Planned Date: 27 February 2022
-    * Latest Date to Be Completed: 13 March 2022
+    * Latest Date to be Completed: 13 March 2022
     * Status: Completed
   - Milestone name:  Chose actuator type (ERM vs LRA)
     * Planned Date: 27 February 2022
-    * Latest Date to Be Completed: 13 March 2022
+    * Latest Date to be Completed: 13 March 2022
     * Status: Completed
   - Milestone name:  Design and evaluate haptic feedback patterns
     * Planned Date: 13 March 2022
-    * Latest Date to Be Completed: 27 March 2022
+    * Latest Date to be Completed: 27 March 2022
-    * Status: In progress (fine-tuning)
+    * Status: Completed
   - Milestone name:  Investigate path guidance
     * Planned Date: 20 March 2022
-    * Latest Date to Be Completed: 10 April 2022
+    * Latest Date to be Completed: 10 April 2022
     * Status: Completed
   - Milestone name:  Integrate with SLAM system
     * Planned Date: 27 March 2022
-    * Latest Date to Be Completed: 17 April 2022
+    * Latest Date to be Completed: 17 April 2022
     * Status: Completed
@@ Line 117: / Line 161: @@
   - Milestone name:  Formulate evaluation method and metrics
     * Planned Date: 27 February 2022
-    * Latest Date to Be Completed: 13 March 2022
+    * Latest Date to be Completed: 13 March 2022
     * Status: Completed
   - Milestone name:  Research methods to implement individualised HRTFs
     * Planned Date: 13 March 2022
-    * Latest Date to Be Completed: 20 March 2022
+    * Latest Date to be Completed: 20 March 2022
     * Status: Completed
   - Milestone name:  Implement chosen methods
     * Planned Date: 20 March 2022
-    * Latest Date to Be Completed: 27 March 2022
+    * Latest Date to be Completed: 27 March 2022
     * Status: Completed
   - Milestone name:  Test and evaluate implemented methods
     * Planned Date: 3 April 2022
-    * Latest Date to Be Completed: 10 April 2022
+    * Latest Date to be Completed: 10 April 2022
     * Status: Completed
   - Milestone name:  Integrate with SLAM system
     * Planned Date: 10 April 2022
-    * Latest Date to Be Completed: 17 April 2022
+    * Latest Date to be Completed: 17 April 2022
-    * Status: In progress (fine-tuning)
+    * Status: Completed
-**Participant Testing**
+**<del>Participant Testing</del>**
-  - Milestone name:  Formulate evaluation method and metrics
+  - Milestone name:  <del>Formulate evaluation method and metrics</del>
     * Planned Date: 10 April 2022
-    * Latest Date to Be Completed: 17 April 2022
+    * Latest Date to be Completed: 17 April 2022
-    * Status: Not started
+    * Status: -
   - Milestone name:  <del>IRB training</del>
     * Planned Date: -
-    * Latest Date to Be Completed: -
+    * Latest Date to be Completed: -
     * Status: -
-  - Milestone name:  Participant acquisition
+  - Milestone name:  <del>Participant acquisition</del>
     * Planned Date: 10 April 2022
-    * Latest Date to Be Completed: 17 April 2022
+    * Latest Date to be Completed: 17 April 2022
-    * Status: Not started
+    * Status: -
-  - Milestone name:  Participant testing
+  - Milestone name:  <del>Participant testing</del>
     * Planned Date: 1 May 2022
-    * Latest Date to Be Completed: 1 May 2022
+    * Latest Date to be Completed: 1 May 2022
-    * Status: Not started
+    * Status: -
+**Visual Feedback System**
+  - Milestone name: Implement visual feedback algorithm
+    * Planned Date: 22 April 2022
+    * Latest Date to be Completed: 25 April 2022
+    * Status: Completed
+  - Milestone name: Integrate with SLAM system
+    * Planned Date: 25 April 2022
+    * Latest Date to be Completed: 28 April 2022
+    * Status: Completed
+  - Milestone name: Integrate with Haptic and Auditory feedback systems
+    * Planned Date: 30 April 2022
+    * Latest Date to be Completed: 30 April 2022
+    * Status: Completed
 ======Reports and presentations======
@@ Line 167: / Line 225: @@
   * Project Checkpoint
     * {{ :courses:456:2022:projects:456-2022-10:checkpoint_presentation.pdf | Project Checkpoint Presentation}}
-  * Project Final Presentation
+  * Project Final Poster
-    * {{:courses:456:2022:projects:456-2022-01:final_poster_pdf.pdf|PDF of Poster}}
+    * {{ :courses:456:2022:projects:456-2022-10:poster.pdf | Final Poster}}
   * Project Final Report
-    * {{:courses:456:2022:projects:456-2022-01:final_report.pdf|Final Report}}
+    * {{ :courses:456:2022:projects:456-2022-10:finalreport.pdf | Final Report}}
-    * {{ :courses:456:2022:projects:456-2022-10:documentation.pdf | Code Documentation}}
+    * {{ :courses:456:2022:projects:456-2022-10:documentation.pdf | Documentation}}
 ======Project Bibliography=======

courses/456/2022/projects/456-2022-10/project-10.1651269643.txt.gz · Last modified: 2022/04/29 22:00 by achen134