Table of Contents Author Guidelines Submit a Manuscript
Journal of Sensors
Volume 2017, Article ID 3296874, 13 pages
https://doi.org/10.1155/2017/3296874
Review Article

A Review of Deep Learning Methods and Applications for Unmanned Aerial Vehicles

Computer Vision and Aerial Robotics Group, Centre for Automation and Robotics (CAR) UPM-CSIC, Universidad Politécnica de Madrid, Calle José Gutiérrez Abascal 2, 28006 Madrid, Spain

Correspondence should be addressed to Adrian Carrio; se.mpu@oirrac.nairda

Received 28 April 2017; Accepted 18 June 2017; Published 14 August 2017

Academic Editor: Vera Tyrsa

Copyright © 2017 Adrian Carrio et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Linked References

  1. A. G. Ivakhnenko, “Polynomial theory of complex systems,” IEEE Transactions on Systems, Man and Cybernetics, vol. 1, no. 4, pp. 364–378, 1971. View at Publisher · View at Google Scholar · View at Scopus
  2. K. Fukushima, “Neocognitron: a hierarchical neural network capable of visual pattern recognition,” Neural Networks, vol. 1, no. 2, pp. 119–130, 1988. View at Google Scholar · View at Scopus
  3. Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,” Proceedings of the IEEE, vol. 86, no. 11, pp. 2278–2323, 1998. View at Publisher · View at Google Scholar · View at Scopus
  4. J. Deng, W. Dong, R. Socher et al., “ImageNet: a large-scale hierarchical image database,” in Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 248–255, Miami, Fla, USA, June 2009. View at Publisher · View at Google Scholar
  5. Y. Bengio, A. Courville, and P. Vincent, “Representation learning: a review and new perspectives,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 35, no. 8, pp. 1798–1828, 2013. View at Publisher · View at Google Scholar · View at Scopus
  6. J. Schmidhuber, “Deep learning in neural networks: an overview,” Neural Networks, vol. 61, pp. 85–117, 2015. View at Publisher · View at Google Scholar · View at Scopus
  7. J. Gu, Z. Wang, J. Kuen et al., Recent Advances in Convolutional Neural Networks, https://arxiv.org/abs/1512.07108.
  8. L. Tai and M. Liu, “Deep-learning in mobile robotics - from perception to control systems: a survey on why and why not,” CoRR abs/1612.07139. http://arxiv.org/abs/1612.07139.
  9. C. Martinez, C. Sampedro, A. Chauhan, and P. Campoy, “Towards autonomous detection and tracking of electric towers for aerial power line inspection,” in Proceedings of the 2014 International Conference on Unmanned Aircraft Systems, ICUAS 2014, pp. 284–295, May 2014. View at Publisher · View at Google Scholar · View at Scopus
  10. M. A. Olivares-Mendez, C. Fu, P. Ludivig et al., “Towards an autonomous vision-based unmanned aerial system against wildlife poachers,” Sensors, vol. 15, no. 12, pp. 31362–31391, 2015. View at Publisher · View at Google Scholar · View at Scopus
  11. A. Carrio, J. Pestana, J.-L. Sanchez-Lopez et al. et al., “Ubristes: uav-based building rehabilitation with visible and thermal infrared remote sensing,” in Proceedings of the Robot 2015: Second Iberian Robotics Conference, pp. 245–256, Springer International Publishing, 2016.
  12. L. Li, Y. Fan, X. Huang, and L. Tian, “Real-time uav weed scout for selective weed control by adaptive robust control and machine learning algorithm,” in Proceedings of the 2016 ASABE Annual International Meeting, American Society of Agricultural and Biological Engineers, p. 1, 2016.
  13. J. L. Sanchez-Lopez, M. Molina, H. Bavle et al., “A multi-layered component-based approach for the development of aerial robotic systems: The aerostack framework,” Journal of Intelligent & Robotic Systems, pp. 1–27, 2017. View at Google Scholar
  14. A. Graves, “Generating sequences with recurrent neural networks,” arXiv preprint https://arxiv.org/abs/1308.0850.
  15. T. M. Mitchell, Machine Learning, vol. 45 (37), McGraw Hill, Burr Ridge, Ill, USA, 1997.
  16. I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning, MIT Press, Cambridge, Mass, USA, 2016. View at MathSciNet
  17. S. Hochreiter and J. Schmidhuber, “LSTM can solve hard long time lag problems,” in Proceedings of the 10th Annual Conference on Neural Information Processing Systems, NIPS 1996, pp. 473–479, December 1996. View at Scopus
  18. A. Gibson and J. Patterson, Deep Learning, O’Reilly, 2016.
  19. G. E. Hinton, S. Osindero, and Y.-W. Teh, “A fast learning algorithm for deep belief nets,” Neural Computation, vol. 18, no. 7, pp. 1527–1554, 2006. View at Publisher · View at Google Scholar · View at MathSciNet · View at Scopus
  20. Y. Bengio, P. Lamblin, D. Popovici, H. Larochelle et al., “Greedy layer-wise training of deep networks,” Advances in Neural Information Processing Systems, vol. 19, pp. 153–160, 2007. View at Google Scholar
  21. P. Smolensky, “Information processing in dynamical systems: foundations of harmony theory,” Tech. Rep., DTIC Document, 1986. View at Google Scholar
  22. G. E. Hinton, “Training products of experts by minimizing contrastive divergence,” Neural Computation, vol. 14, no. 8, pp. 1771–1800, 2002. View at Publisher · View at Google Scholar · View at Scopus
  23. G. E. Hinton, P. Dayan, B. J. Frey, and R. M. Neal, “The “wake-sleep” algorithm for unsupervised neural networks,” Science, vol. 268, no. 5214, pp. 1158–1161, 1995. View at Publisher · View at Google Scholar · View at Scopus
  24. G. E. Hinton and R. R. Salakhutdinov, “Reducing the dimensionality of data with neural networks,” American Association for the Advancement of Science. Science, vol. 313, no. 5786, pp. 504–507, 2006. View at Publisher · View at Google Scholar · View at MathSciNet · View at Scopus
  25. P. Vincent, H. Larochelle, I. Lajoie, and P. Manzagol, “Stacked denoising autoencoders: learning useful representations in a deep network with a local denoising criterion,” Journal of Machine Learning Research, vol. 11, pp. 3371–3408, 2010. View at Google Scholar · View at MathSciNet · View at Scopus
  26. R. Salakhutdinov and G. Hinton, “Semantic hashing,” International Journal of Approximate Reasoning, vol. 50, no. 7, pp. 969–978, 2009. View at Publisher · View at Google Scholar · View at Scopus
  27. A. Krizhevsky and G. E. Hinton, “Using very deep autoencoders for content-based image retrieval,” in Proceedings of the 19th European Symposium on Artificial Neural Networks (ESANN '11), Bruges, Belgium, April 2011.
  28. J. Kober, J. A. Bagnell, and J. Peters, “Reinforcement learning in robotics: A survey,” International Journal of Robotics Research, vol. 32, no. 11, pp. 1238–1274, 2013. View at Publisher · View at Google Scholar · View at Scopus
  29. R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction, vol. 1, MIT Press, Cambridge, UK, 1998.
  30. V. Mnih, K. Kavukcuoglu, D. Silver et al., “Playing atari with deep reinforcement learning,” arXiv preprint https://arxiv.org/abs/1312.5602.
  31. V. Mnih, K. Kavukcuoglu, D. Silver et al., “Human-level control through deep reinforcement learning,” Nature, vol. 518, no. 7540, pp. 529–533, 2015. View at Publisher · View at Google Scholar · View at Scopus
  32. S. Gu, T. Lillicrap, I. Sutskever, and S. Levine, “Continuous deep q-learning with model-based acceleration,” in Proceedings of the 33rd International Conference on International Conference on Machine Learning, vol. 48, pp. 2829–2838, New York, NY, USA, June 2016, preprint https://arxiv.org/abs/1603.00748.
  33. T. P. Lillicrap, J. J. Hunt, A. Pritzel et al., “Continuous control with deep reinforcement learning,” preprint https://arxiv.org/abs/1509.02971.
  34. S. Levine, C. Finn, T. Darrell, and P. Abbeel, “End-to-end training of deep visuomotor policies,” Journal of Machine Learning Research, vol. 17, no. 39, pp. 1–40, 2016, preprint https://arxiv.org/abs/1504.00702. View at Google Scholar
  35. M. Zhang, Z. McCarthy, C. Finn, S. Levine, and P. Abbeel, “Learning deep neural network policies with continuous memory states,” in Proceedings of the 2016 IEEE International Conference on Robotics and Automation, ICRA 2016, pp. 520–527, May 2016. View at Publisher · View at Google Scholar · View at Scopus
  36. T. Zhang, G. Kahn, S. Levine, and P. Abbeel, “Learning deep control policies for autonomous aerial vehicles with MPC-guided policy search,” in Proceedings of the 2016 IEEE International Conference on Robotics and Automation, ICRA 2016, pp. 528–535, May 2016. View at Publisher · View at Google Scholar · View at Scopus
  37. U. Shah, R. Khawad, and K. M. Krishna, “Deepfly: Towards complete autonomous navigation of mavs with monocular camera,” in Proceedings of the Tenth Indian Conference on Computer Vision, Graphics and Image Processing, ICVGIP 16, pp. 59:1–59:8, New York, NY, USA, 2016.
  38. F. Sadeghi and S. Levine, “Real single-image flight without a single real image,” preprint https://arxiv.org/pdf/1611.04201.pdf.
  39. D. K. Kim and T. Chen, “Deep neural network for real-time autonomous indoor navigation,” preprint https://arxiv.org/abs/1511.04668.
  40. A. Giusti, J. Guzzi, D. C. Ciresan et al., “A machine learning approach to visual perception of forest trails for mobile robots,” IEEE Robotics and Automation Letters, vol. 1, no. 2, pp. 661–667, 2016. View at Publisher · View at Google Scholar · View at Scopus
  41. K. Kelchtermans and T. Tuytelaars, “How hard is it to cross the room? – training (recurrent) neural networks to steer a uav,” preprint https://arxiv.org/abs/1702.07600.
  42. R. Girshick, J. Donahue, T. Darrell, and J. Malik, “Rich feature hierarchies for accurate object detection and semantic segmentation,” in Proceedings of the 27th IEEE Conference on Computer Vision and Pattern Recognition (CVPR '14), pp. 580–587, Columbus, Ohio, USA, June 2014. View at Publisher · View at Google Scholar · View at Scopus
  43. R. Girshick, “Fast R-CNN,” in Proceedings of the 15th IEEE International Conference on Computer Vision (ICCV '15), pp. 1440–1448, December 2015. View at Publisher · View at Google Scholar · View at Scopus
  44. S. Ren, K. He, R. Girshick, and J. Sun, “Faster R-CNN: towards real-time object detection with region proposal networks,” in Advances in Neural Information Processing Systems, vol. 28, pp. 91–99, 2015. View at Google Scholar · View at Scopus
  45. J. Lee, J. Wang, D. Crandall, S. Šabanovic, and G. Fox, “Real-time, cloud-based object detection for unmanned aerial vehicles,” in Proceedings of the 1st IEEE International Conference on Robotic Computing (IRC), pp. 36–43, Taichung, Taiwan, April 2017. View at Publisher · View at Google Scholar
  46. J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You only look once: Unified, real-time object detection,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 779–788, 2016, preprint https://arxiv.org/abs/1506.02640.
  47. J. Redmon and A. Farhadi, “Yolo9000: better, faster, stronger,” preprint https://arxiv.org/abs/1612.08242.
  48. W. Liu, D. Anguelov, D. Erhan et al., “Ssd: Single shot multibox detector,” in Proceedings of the European Conference on Computer Vision, pp. 21–37, Springer, 2016.
  49. W. Li, H. Fu, L. Yu, and A. Cracknell, “Deep learning based oil palm tree detection and counting for high-resolution remote sensing images,” Remote Sensing, vol. 9, no. 1, p. 22, 2017. View at Publisher · View at Google Scholar
  50. S. W. Chen, S. S. Shivakumar, S. Dcunha et al., “Counting apples and oranges with deep learning: a data-driven approach,” IEEE Robotics and Automation Letters, vol. 2, no. 2, pp. 781–788, 2017. View at Publisher · View at Google Scholar
  51. B. Zhou, A. Lapedriza, J. Xiao, A. Torralba, and A. Oliva, “Learning deep features for scene recognition using places database,” in Proceedings of the 28th Annual Conference on Neural Information Processing Systems 2014, NIPS 2014, pp. 487–495, December 2014. View at Scopus
  52. O. A. B. Penatti, K. Nogueira, and J. A. Dos Santos, “Do deep features generalize from everyday objects to remote sensing and aerial scenes domains?” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, CVPRW 2015, pp. 44–51, June 2015. View at Publisher · View at Google Scholar · View at Scopus
  53. F. Hu, G.-S. Xia, J. Hu, and L. Zhang, “Transferring deep convolutional neural networks for the scene classification of high-resolution remote sensing imagery,” Remote Sensing, vol. 7, no. 11, pp. 14680–14707, 2015. View at Publisher · View at Google Scholar · View at Scopus
  54. A. Gangopadhyay, S. M. Tripathi, I. Jindal, and S. Raman, “Sa-cnn: dynamic scene classification using convolutional neural networks,” preprint https://arxiv.org/abs/1502.05243.
  55. C. Hung, Z. Xu, and S. Sukkarieh, “Feature learning based approach for weed classification using high resolution aerial images from a digital camera mounted on a UAV,” Remote Sensing, vol. 6, no. 12, pp. 12037–12054, 2014. View at Publisher · View at Google Scholar · View at Scopus
  56. J. Rebetez, H. F. Satizábal, M. Mota et al., “Augmenting a convolutional neural network with local histograms-a case study in crop classification from high-resolution uav imagery,” in Proceedings of the European Symposium on Artificial Neural Networks, 2016.
  57. J. Delmerico, E. Mueggler, J. Nitsch, and D. Scaramuzza, “Active autonomous aerial exploration for ground robot path planning,” IEEE Robotics and Automation Letters, vol. 2, no. 2, pp. 664–671, 2017. View at Publisher · View at Google Scholar
  58. J. Delmerico, A. Giusti, E. Mueggler, L. M. Gambardella, and D. Scaramuzza, ““on-the-spot training” for terrain classification in autonomous air-ground collaborative teams,” in Proceedings of the International Symposium on Experimental Robotics (ISER), EPFL-CONF-221506, 2016.
  59. T. Taisho, L. Enfu, T. Kanji, and S. Naotoshi, “Mining visual experience for fast cross-view UAV localization,” in Proceedings of the 8th Annual IEEE/SICE International Symposium on System Integration, SII 2015, pp. 375–380, December 2015. View at Publisher · View at Google Scholar · View at Scopus
  60. T.-Y. Lin, Y. Cui, S. Belongie, and J. Hays, “Learning deep representations for ground-to-aerial geolocalization,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, pp. 5007–5015, June 2015. View at Publisher · View at Google Scholar · View at Scopus
  61. F. Aznar, M. Pujol, and R. Rizo, “Visual Navigation for UAV with Map References Using ConvNets,” in Advances in Artificial Intelligence, vol. 9868 of Lecture Notes in Computer Science, pp. 13–22, Springer, 2016. View at Publisher · View at Google Scholar
  62. G. J. Mendis, T. Randeny, J. Wei, and A. Madanayake, “Deep learning based doppler radar for micro UAS detection and classification,” in Proceedings of the MILCOM 2016 - 2016 IEEE Military Communications Conference (MILCOM), pp. 924–929, Baltimore, Md, USA, November 2016. View at Publisher · View at Google Scholar
  63. T. Zhang, G. Kahn, S. Levine, and P. Abbeel, “Learning deep control policies for autonomous aerial vehicles with MPC-guided policy search,” in Proceedings of the 2016 IEEE International Conference on Robotics and Automation (ICRA), pp. 528–535, Stockholm, Sweden, May 2016. View at Publisher · View at Google Scholar
  64. T. Morito, O. Sugiyama, R. Kojima, and K. Nakadai, “Partially shared deep neural network in sound source separation and identification using a uav-embedded microphone array,” in Proceedings of the 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2016, pp. 1299–1304, October 2016. View at Publisher · View at Google Scholar · View at Scopus
  65. S. Jeon, J.-W. Shin, Y.-J. Lee, W.-H. Kim, Y. Kwon, and H.-Y. Yang, “Empirical study of drone sound detection in real-life environment with deep neural networks,” preprint https://arxiv.org/abs/1701.05779.
  66. D. Maturana and S. Scherer, “3D convolutional neural networks for landing zone detection from LiDAR,” in Proceedings of the IEEE International Conference on Robotics and Automation (ICRA '15), pp. 3471–3478, IEEE, Washington, DC, USA, May 2015. View at Publisher · View at Google Scholar · View at Scopus
  67. Y. LeCun, B. E. Boser, J. S. Denker et al., “Handwritten digit recognition with a back-propagation network,” in Advances in Neural Information Processing Systems, D. S. Touretzky, Ed., vol. 2, pp. 396–404, 1990. View at Google Scholar
  68. A. Ghaderi and V. Athitsos, “Selective unsupervised feature learning with convolutional neural network (S-CNN),” in Proceedings of the 2016 23rd International Conference on Pattern Recognition (ICPR), pp. 2486–2490, December 2016. View at Publisher · View at Google Scholar
  69. A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in Proceedings of the 26th Annual Conference on Neural Information Processing Systems (NIPS '12), pp. 1097–1105, Lake Tahoe, Nev, USA, December 2012. View at Scopus
  70. H. Kim, D. Kim, S. Jung, J. Koo, J.-U. Shin, and H. Myung, “Development of a UAV-type jellyfish monitoring system using deep learning,” in Proceedings of the 12th International Conference on Ubiquitous Robots and Ambient Intelligence, URAI 2015, pp. 495–497, October 2015. View at Publisher · View at Google Scholar · View at Scopus
  71. N. V. Kim and M. A. Chervonenkis, “Situation control of unmanned aerial vehicles for road traffic monitoring,” Modern Applied Science, vol. 9, no. 5, pp. 1–13, 2015. View at Publisher · View at Google Scholar · View at Scopus
  72. M. Bejiga, A. Zeggada, A. Nouffidj, and F. Melgani, “A convolutional neural network approach for assisting avalanche search and rescue operations with UAV imagery,” Remote Sensing, vol. 9, no. 2, p. 100, 2017. View at Publisher · View at Google Scholar
  73. A. Sawarkar, V. Chaudhari, R. Chavan, V. Zope, A. Budale, and F. Kazi, “HMD vision-based teleoperating UGV and UAV for hostile environment using deep learning,” CoRR abs/1609.04147. URL http://arxiv.org/abs/1609.04147.
  74. C. Szegedy, W. Liu, Y. Jia et al., “Going deeper with convolutions,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR '15), pp. 1–9, Boston, Mass, USA, June 2015. View at Publisher · View at Google Scholar
  75. The Technion – Israel Institute of Technology, “Technion aerial systems 2016,” in Journal Paper for AUVSI Student UAS Competition, 2016. View at Google Scholar
  76. P. Santana, L. Correia, R. Mendonça, N. Alves, and J. Barata, “Tracking natural trails with swarm-based visual saliency,” Journal of Field Robotics, vol. 30, no. 1, pp. 64–86, 2013. View at Publisher · View at Google Scholar · View at Scopus