Drones and computer vision Team

Principal Investigator:Professor Jen-Hui Chuang

—

Summary

The objective of the research is to develop an autonomous flight control system for drones, or unmanned aerial vehicle (UAV). We have developed and tested the control system over a computer simulator and physical drones. The system is designed to avoid any obstacles in the air, complete a flight mission, and land safely. For cost-effective training to avoid crashes, we built a flight simulator, which can receive and react to the somatosensory feedbacks. We created 3D models based on the real environments to train the flight control system over the simulator. In addition to the automatic flight control, we also study control of precise locating, navigating, and landing. The plan is to train the autonomous flight control system in the simulated environments and transfer the system over real drones when ready. We anticipate differences between simulated environments and the real outdoor ones, which will be the second part of this research challenges.

Keywords

UAVs, Smart Drones, Smart Sensing, Smart Control and Smart Simulation, Deep Learning

Innovations

• A new network architecture is developed for the analysis of aerial images of a parking lot, which is the world’s first, to quickly identify the available parking spaces whose sizes are smaller than 15×15 pixels.
• Deep learning-based obstacle detection and depth prediction are merged into a single integrated architecture to attain real-time performance.
• With flight control training based on cross-domains and cross-tasks learning, the training results in the simulated environment are expected to be directly applicable to real-world situations.
• The world’s first end-to-end algorithm for static/dynamic obstacle avoidance and navigation　using LIDAR is developed for smart sensing of drones.
• A wearable haptic device is developed to convert remote UAV’s flight and sensor information to somatosensory signals to serve as a flight control aider for piloting.

Benefits

• We have developed a real-time obstacle detection and avoidance system for drones that can effectively avoid static objects or moving pedestrians.
• We are developing the world’s first drone flight simulator which will combine the remote control, AirSim simulator, and VR technology to provide professional and amateur pilots with a good simulation environment for collecting simulated flight data, and thus with no need to worry about power consumption, equipment damage, etc.
• We have developed a prototype of the Remote Tactile and Sensing Suit (RETASENS), which allows the pilot of a drone to perceive the flight attitude and sensor signals from the drone by augmenting and remapping them over human sensory.
• We have published two international journal papers and 13 international conference papers (including a CVPR paper for a depth estimator which is ranked 1st worldwide on KITTI leaderboard for 8 months).