======Multisensory Navigational Aid for Visual Prosthesis Users====== **Last updated: 30 April 2022 at 18:22** ======Summary====== The Argus II retinal prosthesis system offers limited visual feedback to the users making independent navigation challenging. Thus, this project aims to develop two supplementary systems, namely haptic and auditory, and to determine the effectiveness of the augmented system in aiding the users with path navigation and obstacle localisation. * **Students:** An Chi Chen * **Mentor(s):** Dr Seth Billings, Chigozie Ewulum ======Background, Specific Aims, and Significance====== The Argus II is a retinal prosthesis system for patients with late stage Retinitis Pigmentosa (loss or breakdown of retina cells). The Argus II system is composed of an electrode array that is implanted behind the users' eye as well as a camera mounted onto a pair of glasses along with the required processing system. {{ :courses:456:2022:projects:456-2022-10:argusii.png?400 |}} Artificial vision is provided to the users by using input from the camera to stimulate the users' viable retinal cells which is then perceived as light patterns by the user. {{ :courses:456:2022:projects:456-2022-10:artificialvision.png?600 |}} There are limitations to this system that hinder the users ability to independently navigate a space. Some of these limitations are: * Varying perception of light patterns * Some can distinguish variable light intensities * Others can only distinguish on/off states * Limited field of vision Thus the goals for this project are the following: * Develop haptic and auditory feedback system that work in conjunction with the Argus II retinal prosthesis system * The supplementary systems must assist users with target navigation and object localisation * The supplementary systems should be intuitive and maintain a low cognitive load ======Deliverables====== * **Minimum:** (Completed on 27 March 2022) - Operational haptic feedback system that can intuitively guide users' gaze direction * **Expected:** (Completed on 4 April 2022) - Operational haptic feedback system - Operational auditory feedback system that is able to assist users with obstacle localisation * **Proposed Maximum:** (Expected by 1 May 2022, not completed) - Retinal prosthesis system that is integrated with haptic and auditory feedback that has been user tested - Evaluation of that system's performance in terms of target navigation and obstacle avoidance as a results of the participant testing * **Adjusted Maximum:** (Completed on 30 April 2022) - Integrated haptic and auditory feedback system with simulated Argus II feedback on virtual reality headset ======Dependencies====== There are not many dependencies for this project however they will affect the deliverables if they are not resolved in a timely manner. For both the haptic and auditory feedback systems the SLAM-based navigational system is required for full integration with the Argus II system. This also includes the necessary hardware such as cameras. The SLAM system itself has been developed and is currently in the refinement phase. The required cameras for the system have been obtained. The only outstanding hardware are the camera mounts – these are currently being 3D printed. If these mounts are not completed by 10 April 2022, then the subsequent tasks (testing of gaze and path guidance and object localisation in the real world and participant testing) will be delayed. If the delay is substantial (past 17 April 2022) it may not be possible to complete the participant testing in this timeframe as there should be at least two weeks of participant testing. For the participant testing, the IRB training will need to be completed. Whether testing can take place also depends on participant availability. The IRB training can be completed in advance and thus will be done so. A testing protocol currently exists, and the team members will simply need to be added to it to be able to perform participant testing. Furthermore, four weeks have been allocated to acquire participants. Ideally the testing will be done with Argus II users, however if they are not available it is possible to perform participant testing with seeing individuals. The system developed by APL does also connects to a VR headset which allows seeing individuals to see what Argus II users perceive. If it is still not possible to acquire the participants by 17 April 2022 then the allocated time for participant testing will be reduced and potentially may not be completed in this project timeline. The table below summarises the dependencies mentioned above. {{ :courses:456:2022:projects:456-2022-10:dependencies.png?650 |}} *Latest Date = If dependency not resolved by this date it will affect the deliverables The following table shows the final status of the dependencies at the end of the project timeline (1 May 2022). {{ :courses:456:2022:projects:456-2022-10:dependenciesatend.png?650 |}} As can be seen the IRB and participant availability dependency were not achieved. This was due to the fact that the existing IRB protocol only includes recruiting participants from APL or Argus II users. The developed supplementary feedback systems, though integrated, are not in a complete enough state to be tested with Argus II users at this stage. In addition, there was not enough time during this project timeline to execute the fallback plan for participant testing which was to test the developed integrated system with APL employees. As seen above under the deliverables, the maximum deliverable was adjusted due to these time constraints and IRB dependency not being met. ======Technical Approach====== **Haptic Feedback System** The haptic feedback system’s hardware consists of a headband fitted with 8 repositionable haptic actuators which connects to a custom 8-channel haptic driver. The function and objective for this haptic system is to guide the users’ gaze towards the direction of their target (e.g. a door). By doing so, this should assist the users in finding their targets and navigating to them more effectively. {{ :courses:456:2022:projects:456-2022-10:haptic.png?600 |}} The first step in developing this system is to formulate an evaluation method and metrics. These will be used to determine if the system is intuitive and effective in relaying the directional instructions. Following so, there are a few options which need to be explored to develop the haptic feedback system. ERMs and LRAs are two common haptic actuators, thus, it must be determined which type is best suited for this application. Both have their advantages, and the choice will be dependent on which type provides a clearer, more distinct feedback. Furthermore, there are various haptic patterns which can be employed to relay a directional instruction. A static pattern involves only actuating motors on the side the users should direct their gaze. Whilst a dynamic pattern relays a direction by using perceived motion – the haptic motors are actuated in a specific order, such as right to left to relay a left direction. How intuitive these haptic patterns are to users will determine which one will be used. {{ :courses:456:2022:projects:456-2022-10:haptic_pattern.png?300 |}} It is also worth considering the use of the haptic feedback system for path guidance, as opposed to just gaze direction guidance. For a path guidance objective, the haptic feedback will continually update as the users move their heads to continually guide the users towards their target. After the haptic system is developed it will be integrated with the SLAM navigation system to test its real-world gaze direction and path guidance functionalities. **Auditory Feedback System** The hardware for the auditory feedback system is a pair of open-ear bone conduction headphones. These headphones were chosen with the aim to not overwhelm the users’ senses, due to the open-ear nature of these headphones they can still hear noises from the world. The function and objective for this system is to provide users with the location of potential obstacles using sound. The name of an obstacle will be played to the users through the headphones and sound like it is coming from the direction of the objects. This mitigates the need for users to manually scan their surroundings for obstacles before moving. {{ :courses:456:2022:projects:456-2022-10:auditory.png?500 |}} Similar to the development of the haptic system, the first step in developing the auditory feedback system will be to formulate an evaluation method and metrics which will be used to determine the accuracy and effectiveness of the system’s localisation functionality. Following so, methods to individualise head related transfer functions (HRTFs) will be explored. HRTFs are what aids individuals in determining the location of a sound but are unique to each individual as they are dependent on various factors such as the shape of the head. Thus, generalized HRTFs do not translate well with everyone. There is especially a discrepancy between determining whether a simulated sound is coming from the front or back as well as up or down. Currently, individualised HRTFs are not well developed, though there have been various methods suggested for creating personalization. Some of these methods will be explored as well as determining the possibility of using calibration to personalize a set of generalised HRTFs. Once the auditory feedback system has been developed it will also be integrated with the SLAM navigation system to test its real-world, real-time object localisation. **Participant Testing** Once both systems have been developed and integrated with the SLAM-based navigational system built by APL the system can be participant tested with Argus II users. Before testing can take place an evaluation method and required metrics will be formulated. These will be used to evaluate the system’s performance in terms of target navigation and obstacle avoidance. The participant testing was not completed during this timeline of the project due to time constraints and IRB dependency not met. **Visual Feedback System** This feedback system will not form part of the supplementary system for Argus II users however it is a useful system to use when sighted users test the Argus II feedback system. This feedback system attempts to emulate what Argus II users see through the use of the virtual reality head-mounted-display, Oculus. A grid of 6x10 pixels was used as the viewing display – simulating what is seen by Argus II users. This system was used to display the landmarks that come into the field of view (FOV) of the user. In order to achieve this the horizontal and vertical angles between where the user is looking and the landmarks were determined (detailed in section 2.4). If the angles fell within the visible FOV they were displayed at the correct location determined by the horizontal and vertical angles. This is illustrated in the figure below. {{ :courses:456:2022:projects:456-2022-10:visualfeedbackimage.png?400 |}} ======Results====== **Haptic Feedback System** Through the iterative testing it was found that static patterns were optimal for relaying the left and right instructions. This was sufficient and intuitive enough to indicate to users in which direction they should look. All 4 of the actuators located around the temporal and back of the head were utilised for this group of instructions thus making it possible to indicate a direction located towards the back of the head. Dynamic patterns were employed to relay the up and down instructions. Interestingly, when only the 3 actuators located along the top of the head were used in the dynamic patterns the interpretation of the haptic pattern became unclear – potentially due to the lower density of actuators along the top of the head. Thus, these patterns make use of all 7 actuators actuated in a specific order. By using all 7 actuators, the feedback from the haptic pattern felt more distinct and clearer in relaying a directional instruction. The up pattern starts with the actuators closest to the front of the head and moves towards the back, whilst the down pattern starts at the back of the head and ends in the front. {{ :courses:456:2022:projects:456-2022-10:hapticpatterns.png?600 |}} During the testing, it was realised that the duration of a haptic pattern should be as short as possible to allow for a fast update time. If this was not satisfied, there was a noticeable delay between a turning of the head and the next correct haptic instruction which results in an overshoot of head turning movement and creates confusion. Thus, for the static patterns the durations were set to 0.015 seconds. This may seem like a short duration but the haptic patterns are repeated continuously until the target gaze direction is reached. For the dynamic patterns, it is important that the durations are still set at a suitable time which makes them discernible as the directional instruction which they are relaying. A total duration of 0.18 seconds was found to be a sufficient duration that satisfies these criteria. Furthermore, it was found that it is important that the haptic actuators are flush and are placed with a bit of pressure against the head in order to localise them properly. Additionally, hair density may result in a reduced ability to determine which haptic actuator is on. This potential issue could be solved by ensuring the haptic actuators are very close to the surface of the head. **Auditory Feedback System** A tournament to find a best fitting HRTF was set up using Unity with the SOFAlizer plugin [5] which facilitated switching between different HRTF profiles with ease. The HRTF contenders were obtained from the open-source databases by MIT, CIPIC and RIEC [6, 7, 8]. A total of 30 HRTF profiles were used during this evaluation. A completely accurate best fitting HRTF profile match was not found. In essence, of the HRTF profiles examined, none overcame the front/back discrepancy described earlier. Furthermore, this process was rather tedious as each HRTF had to be tested with the sound source located at varying position – the effort required did not equate to an extremely well fitted HRTF. The second approach of using the application that generates an individualised HRTF based off of measurements of the ear was a much more concise process. After providing the input of a few measurements of the ear and head the application produced a comparable HRTF to that of the first individualisation method; although it was still not perfect. Nevertheless, as the focus for this project was on the feedback algorithm, and not the accuracy of individualisation of an HRTF profile, the HRTF obtained in the second method was used in the subsequent feedback algorithm testing. Spatialised sounds were able to be accurately created and played through the bone conduction headphones. The position of the landmarks (ArUco markers) located along the side of the user could be successfully located solely based-off of the sound. Visual feedback provides assistance in locating sound sources from the front and back due to the front/back discrepancy, however, the use of a well-fitted HRTF profile would mitigate this issue. The distances, in feet, between the user and the landmark(s) were also announced to the users. This was computed by simply calculating the norm of the vector between the user body and the landmark. It was found that if more than three identified obstacles (often will be the case), including their distances, were announced created a larger load on the senses and could potentially disengage the users. Therefore, in order to reduce the cognitive load, the algorithm was updated such that only obstacles within 10 feet of the body were announced. It was also found that the listing of landmark names along with their distances is slow and users can move rather quickly. Thus, the distances of the obstacles from the user were only announced every third iteration of the loop and not every single time the obstacles were listed in order to ensure adequate and accurate auditory feedback is provided. **Visual Feedback System** The horizontal and vertical FOVs had to be fine-tuned to accurately reflect the landmark positions in real life. The final FOVs are: Horizontal FOV = 80° Vertical FOV = 60°. The figure below shows that the landmarks within the user’s FOV were successfully placed and shown on the virtual reality headset display. {{ :courses:456:2022:projects:456-2022-10:vr.png?400 |}} ======Milestones and Status ====== The milestones for the project as well as the anticipated completion dates are detailed below. The ‘Latest Date to Be Completed By” provides the date after which if a task is still incomplete will results in delays in the planned schedule. **Haptic Feedback System** - Milestone name: Formulate evaluation method and metrics * Planned Date: 27 February 2022 * Latest Date to be Completed: 13 March 2022 * Status: Completed - Milestone name: Chose actuator type (ERM vs LRA) * Planned Date: 27 February 2022 * Latest Date to be Completed: 13 March 2022 * Status: Completed - Milestone name: Design and evaluate haptic feedback patterns * Planned Date: 13 March 2022 * Latest Date to be Completed: 27 March 2022 * Status: Completed - Milestone name: Investigate path guidance * Planned Date: 20 March 2022 * Latest Date to be Completed: 10 April 2022 * Status: Completed - Milestone name: Integrate with SLAM system * Planned Date: 27 March 2022 * Latest Date to be Completed: 17 April 2022 * Status: Completed **Auditory Feedback System** - Milestone name: Formulate evaluation method and metrics * Planned Date: 27 February 2022 * Latest Date to be Completed: 13 March 2022 * Status: Completed - Milestone name: Research methods to implement individualised HRTFs * Planned Date: 13 March 2022 * Latest Date to be Completed: 20 March 2022 * Status: Completed - Milestone name: Implement chosen methods * Planned Date: 20 March 2022 * Latest Date to be Completed: 27 March 2022 * Status: Completed - Milestone name: Test and evaluate implemented methods * Planned Date: 3 April 2022 * Latest Date to be Completed: 10 April 2022 * Status: Completed - Milestone name: Integrate with SLAM system * Planned Date: 10 April 2022 * Latest Date to be Completed: 17 April 2022 * Status: Completed **Participant Testing** - Milestone name: Formulate evaluation method and metrics * Planned Date: 10 April 2022 * Latest Date to be Completed: 17 April 2022 * Status: - - Milestone name: IRB training * Planned Date: - * Latest Date to be Completed: - * Status: - - Milestone name: Participant acquisition * Planned Date: 10 April 2022 * Latest Date to be Completed: 17 April 2022 * Status: - - Milestone name: Participant testing * Planned Date: 1 May 2022 * Latest Date to be Completed: 1 May 2022 * Status: - **Visual Feedback System** - Milestone name: Implement visual feedback algorithm * Planned Date: 22 April 2022 * Latest Date to be Completed: 25 April 2022 * Status: Completed - Milestone name: Integrate with SLAM system * Planned Date: 25 April 2022 * Latest Date to be Completed: 28 April 2022 * Status: Completed - Milestone name: Integrate with Haptic and Auditory feedback systems * Planned Date: 30 April 2022 * Latest Date to be Completed: 30 April 2022 * Status: Completed ======Reports and presentations====== * Project Plan * {{ :courses:456:2022:projects:456-2022-10:multisensory_navigational_aid_for_visual_prosthesis_users.pdf | Project Plan Presentation}} * {{ :courses:456:2022:projects:456-2022-10:cis_ii_project_proposal_plan.pdf | Project Plan Proposal}} * Project Background Reading * [[https://dl.acm.org/doi/pdf/10.1145/3458021 | Reading for Haptic Feedback System]] * [[https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=8798218&tag=1 | Reading for Auditory Feedback System]] * Paper Seminar Presentations * {{ :courses:456:2022:projects:456-2022-10:seminar_presentation.pdf | Seminar Presentation}} * {{ :courses:456:2022:projects:456-2022-10:critical_review.pdf | Critical Review}} * Project Checkpoint * {{ :courses:456:2022:projects:456-2022-10:checkpoint_presentation.pdf | Project Checkpoint Presentation}} * Project Final Poster * {{ :courses:456:2022:projects:456-2022-10:poster.pdf | Final Poster}} * Project Final Report * {{ :courses:456:2022:projects:456-2022-10:finalreport.pdf | Final Report}} * {{ :courses:456:2022:projects:456-2022-10:documentation.pdf | Documentation}} ======Project Bibliography======= * Farvardin M, Afarid M, Attarzadeh A, et al. The Argus-II Retinal Prosthesis Implantation; From the Global to Local Successful Experience. Front Neurosci. 2018;12:584. Published 2018 Sep 5. doi:10.3389/fnins.2018.00584 * Arevalo, J. F., Al Rashaed, S., Alhamad, T. A., Al Kahtani, E., Al-Dhibi, H. A., & KKESH Collaborative Retina Study Group (2021). Argus II retinal prosthesis for retinitis pigmentosa in the Middle East: The 2015 Pan-American Association of Ophthalmology Gradle Lecture. International journal of retina and vitreous, 7(1), 65. https://doi.org/10.1186/s40942-021-00324-6 * Kaul, O.B., Rohs, M., Mogalle, M., & Simon, B. (2021). Around-the-Head Tactile System for Supporting Micro Navigation of People with Visual Impairments. ACM Transactions on Computer-Human Interaction (TOCHI), 28, 1 - 35. * Panëels, S.A., Anastassova, M., Strachan, S., Van, S.P., Sivacoumarane, S., & Bolzmacher, C. (2013). What's around me? Multi-actuator haptic feedback on the wrist. 2013 World Haptics Conference (WHC), 407-412. * Xu, S., Li, Z., & Salvendy, G. (2007). Individualization of Head-Related Transfer Function for Three-Dimensional Virtual Auditory Display: A Review. HCI. * Iida, K. Head-Related Transfer Function and Acoustic Virtual Reality. 1st ed. 2019. * Berger, C.C., González-Franco, M., Tajadura-Jiménez, A., Florêncio, D.A., & Zhang, Z. (2018). Generic HRTFs May be Good Enough in Virtual Reality. Improving Source Localization through Cross-Modal Plasticity. Frontiers in Neuroscience, 12. ======Other Resources and Project Files====== Here give list of other project files (e.g., source code) associated with the project. If these are online give a link to an appropriate external repository or to uploaded media files under this name space (2022-01).