Over 43 million people worldwide live with severe visual impairment[1], facing significant challenges in navigating unfamiliar environments.We present MR.NAVI, a mixed reality system that enhances spatial awareness for visually impaired users through real-time scene understanding and intuitive audio feedback.Our system combines computer vision algorithms for object detection and depth estimation with natural language processing to provide contextual scene descriptions, proactive collision avoidance, and navigation instructions.The distributed architecture processes sensor data through MobileNet for object detection and employs RANSAC-based floor detection with DBSCAN clustering for obstacle avoidance.Integration with public transit APIs enables navigation with public transportation directions.Through our experiments with user studies, we evaluated both scene description and navigation functionalities in unfamiliar environments, showing promising usability and effectiveness.A demo video is available here.Code is available here.
This paper presents MR.NAVI, a mixed-reality navigation assistant designed to enhance spatial awareness for visually impaired individuals using the HoloLens2. The system integrates real-time obstacle avoidance, natural language-based scene descriptions, and public transport navigation by utilizing computer vision techniques and natural language processing. It addresses key challenges faced by visually impaired users in navigating unfamiliar environments, combining functionalities from Electronic Orientation Aids (EOAs), Electronic Travel Aids (ETAs), and Position Locator Devices (PLDs) into a single cohesive system. Key contributions include the implementation of a scene understanding pipeline optimized for real-time operation, a multimodal navigation system, and a user study evaluating the system's usability and effectiveness in real-world scenarios.
This paper employs the following methods:
- Computer Vision
- Natural Language Processing
- Real-Time Processing
The following datasets were used in this research:
- Response Time
- User Ratings
- MR.NAVI effectively enhances navigation for visually impaired users through real-time scene understanding and intuitive audio feedback.
- User studies indicated high satisfaction with the system's scene description functionality.
The authors identified the following limitations:
- Limited ability of the scene description framework to recognize moving objects.
- Obstacle avoidance mode received mixed feedback, particularly on spatial audio cues.
- Number of GPUs: None specified
- GPU Type: None specified
- Compute Requirements: None specified