Robotics and sensing technology Team

Principal Investigator:Professor Wen-June Wang

–

Summary

We are developing an integrated system to facilitate visually impaired people in daily activities. The system will ensure that users have both safe outdoor activities and convenient indoor actions. Using Deep Learning technologies, our team will design and implement navigation devices, whether they are wearable or carried via a robot, to help visually impaired persons to travel, to recognize, and to find objects.

Keywords

Deep learning, Machine Learning, Robots, Wearable Devices, Semantic Segmentation, Ground Truth Generator, Visually Impaired Supporting System, Cloud-based Dialog Navigation Agent, Indoor Living Support

Innovations

• A guiding robot is implemented (see Figure 1) to assist visually impaired people for outdoor navigation.
• Deep learning technology is employed to find obstacles and their distances.
• Street Logos are recognized by the robot eyes. Suggestions based on recognition results further assists users to find new locations.
• A new semantic segmentation tool further helps the recognition system to automatically collect millions of street photos for neural network training (Figure 2).
• A cloud-based dialog navigation agent (CDNA) system is designed and will be integrated with the robot. Through verbal commands, the robot can further assist the user. The cell design for the dialog engine is illustrated in Figure 3.
• A smart indoor living support system is developed with different kinds of sensor technologies. The system assists visually impaired people to understand object locations and setups (see system architecture in Figure 4).
• Via object recognition deep learning modules, the indoor supporting system will assist the user to find personal belongings.

Benefits

• The robot hardware and its neural network system are developed and tested on streets. Preliminary results show that the navigation system can guide blind people to travel in a university campus.
• The robot eye can recognition 14 types of street logos. Preliminary result shows a 73% accuracy with a minimal but sufficient set of training data.
• The ground truth generator had already collected 6 million street photos in 6 major cities in Taiwan. The goal is to collect 10 millions in all cities in Taiwan.
• The CDNA system is developed and tested by using information near a university campus. The dialogue is effective in assisting the visually impaired people.
• Indoor objects and setup can be precisely identified and located. The design of the smart living supporting system is proven to be feasible.