Table of Contents Author Guidelines Submit a Manuscript
Mobile Information Systems
Volume 2016, Article ID 4565203, 18 pages
http://dx.doi.org/10.1155/2016/4565203
Research Article

Dynamic Resource Allocation with Integrated Reinforcement Learning for a D2D-Enabled LTE-A Network with Access to Unlicensed Band

Laboratory of Information Communication Networks, School of Information Science and Technology, Hokkaido University, Sapporo, Japan

Received 30 May 2016; Revised 8 September 2016; Accepted 16 October 2016

Academic Editor: Juan C. Cano

Copyright © 2016 Alia Asheralieva and Yoshikazu Miyanaga. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Linked References

  1. A. Asadi, Q. Wang, and V. Mancuso, “A survey on device-to-device communication in cellular networks,” IEEE Communications Surveys and Tutorials, vol. 16, no. 4, pp. 1801–1819, 2014. View at Publisher · View at Google Scholar · View at Scopus
  2. A. Asheralieva and Y. Miyanaga, “Dynamic buffer status-based control for LTE-a network with underlay D2D communication,” IEEE Transactions on Communications, vol. 64, no. 3, pp. 1342–1355, 2016. View at Publisher · View at Google Scholar
  3. A. Asheralieva and Y. Miyanaga, “QoS oriented mode, spectrum and power allocation for D2D communication underlaying LTE-A network,” IEEE Transactions on Vehicular Technology, 2016. View at Publisher · View at Google Scholar
  4. J. Kim, S. Kim, J. Bang, and D. Hong, “Adaptive mode selection in D2D communications considering the bursty traffic model,” IEEE Communications Letters, vol. 20, no. 4, pp. 712–715, 2016. View at Publisher · View at Google Scholar
  5. D. Penda, L. Fu, and M. Johansson, “Energy efficient D2D communications in dynamic TDD systems,” https://arxiv.org/abs/1506.00412.
  6. K. Yang, S. Martin, L. Boukhatem, J. Wu, and X. Bu, “Energy-efficient resource allocation for device-to-device communications overlaying LTE networks,” in Proceedings of the IEEE 82nd Vehicular Technology Conference (VTC Fall '15), pp. 1–6, Boston, Mass, USA, September 2015. View at Publisher · View at Google Scholar
  7. A. Asadi, P. Jacko, and V. Mancuso, “Modeling multi-mode D2D communications in LTE,” Acm Sigmetrics, vol. 42, no. 2, pp. 55–57, 2014. View at Google Scholar
  8. F. Malandrino, C. Casetti, C. F. Chiasserini, and Z. Limani, “Uplink and downlink resource allocation in D2D-enabled heterogeneous networks,” in Proceedings of the IEEE Wireless Communications and Networking Conference Workshops (WCNCW '14), pp. 87–92, April 2014. View at Publisher · View at Google Scholar · View at Scopus
  9. D. Feng, L. Lu, Y.-W. Yi, G. Y. Li, G. Feng, and S. Li, “Device-to-device communications underlaying cellular networks,” IEEE Transactions on Communications, vol. 61, no. 8, pp. 3541–3551, 2013. View at Publisher · View at Google Scholar · View at Scopus
  10. L. Su, Y. Ji, P. Wang, and F. Liu, “Resource allocation using particle swarm optimization for D2D communication underlay of cellular networks,” in Proceedings of the IEEE Wireless Communications and Networking Conference (WCNC '13), pp. 129–133, IEEE, Shanghai, China, April 2013. View at Publisher · View at Google Scholar · View at Scopus
  11. Wi-Fi Alliance, Wi-Fi Peer-to-Peer (P2P) Specification version 1.1, Wi-Fi Alliance Specification, 1, 2010.
  12. Z. Alliance, Zigbee Specification, Document 053474r06 (version), 1, 2006.
  13. Bluetooth Specification, Bluetooth Specification version 1.1, 2001, http://www.bluetooth.com.
  14. A. Asadi and V. Mancuso, “Energy efficient opportunistic uplink packet forwarding in hybrid wireless networks,” in Proceedings of the 4th ACM International Conference on Future Energy Systems (e-Energy '13), pp. 261–262, Berkeley, Calif, USA, May 2013. View at Publisher · View at Google Scholar · View at Scopus
  15. A. Asadi and V. Mancuso, “On the compound impact of opportunistic scheduling and D2D communications in cellular networks,” in Proceedings of the 16th ACM International Conference on Modeling, Analysis and Simulation of Wireless and Mobile Systems (MSWiM '13), pp. 279–287, November 2013. View at Publisher · View at Google Scholar · View at Scopus
  16. A. Asadi and V. Mancuso, “WiFi Direct and LTE D2D in action,” in Proceedings of the 6th IFIP/IEEE Wireless Days Conference (WD '13), pp. 1–8, Valencia, Spain, November 2013. View at Publisher · View at Google Scholar · View at Scopus
  17. Q. Wang and B. Rengarajan, “Recouping opportunistic gain in dense base station layouts through energy-aware user cooperation,” in Proceedings of the IEEE 14th International Symposium on a World of Wireless, Mobile and Multimedia Networks (WoWMoM '13), pp. 1–9, June 2013. View at Publisher · View at Google Scholar · View at Scopus
  18. B. Zhou, S. Ma, J. Xu, and Z. Li, “Group-wise channel sensing and resource pre-allocation for LTE D2D on ISM band,” in Proceedings of the IEEE Wireless Communications and Networking Conference (WCNC '13), pp. 118–122, IEEE, Shanghai, China, April 2013. View at Publisher · View at Google Scholar · View at Scopus
  19. M. Ji, G. Caire, and A. F. Molisch, “Wireless device-to-device caching networks: basic principles and system performance,” IEEE Journal on Selected Areas in Communications, vol. 34, no. 1, pp. 176–189, 2016. View at Publisher · View at Google Scholar
  20. H. Cai, I. Koprulu, and N. B. Shroff, “Exploiting double opportunities for deadline based content propagation in wireless networks,” in Proceedings of the 32nd IEEE Conference on Computer Communications (IEEE INFOCOM '13), pp. 764–772, Turin, Italy, April 2013. View at Publisher · View at Google Scholar · View at Scopus
  21. A. Asadi, P. Jacko, and V. Mancuso, “Modeling multi-mode D2D communications in LTE,” https://arxiv.org/abs/1405.6689.
  22. R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction, MIT Press, Cambridge, Mass, USA, 1998.
  23. S. M. Perlaza, H. Tembine, and S. Lasaulce, “How can ignorant but patient cognitive terminals learn their strategy and utility?” in Proceedings of the IEEE 11th International Workshop on Signal Processing Advances in Wireless Communications (SPAWC '10), June 2010. View at Publisher · View at Google Scholar · View at Scopus
  24. Y. Xing and R. Chandramouli, “Stochastic learning solution for distributed discrete power control game in wireless data networks,” IEEE/ACM Transactions on Networking, vol. 16, no. 4, pp. 932–944, 2008. View at Publisher · View at Google Scholar · View at Scopus
  25. L. Rose, S. Lasaulce, S. M. Perlaza, and M. Debbah, “Learning equilibria with partial information in decentralized wireless networks,” IEEE Communications Magazine, vol. 49, no. 8, pp. 136–142, 2011. View at Publisher · View at Google Scholar · View at Scopus
  26. Y. Xu, Q. Wu, L. Shen, J. Wang, and A. Anpalagan, “Opportunistic spectrum access with spatial reuse: graphical game and uncoupled learning solutions,” IEEE Transactions on Wireless Communications, vol. 12, no. 10, pp. 4814–4826, 2013. View at Publisher · View at Google Scholar · View at Scopus
  27. Y. Xu, Q. Wu, J. Wang, L. Shen, and A. Anpalagan, “Opportunistic spectrum access using partially overlapping channels: graphical game and uncoupled learning,” IEEE Transactions on Communications, vol. 61, no. 9, pp. 3906–3918, 2013. View at Publisher · View at Google Scholar · View at Scopus
  28. D. Kalathil, N. Nayyar, and R. Jain, “Decentralized learning for multiplayer multiarmed bandits,” IEEE Transactions on Information Theory, vol. 60, no. 4, pp. 2331–2345, 2014. View at Publisher · View at Google Scholar · View at MathSciNet · View at Scopus
  29. S. Maghsudi and S. Stańczak, “Channel selection for network-assisted D2D communication via no-regret bandit learning with calibrated forecasting,” IEEE Transactions on Wireless Communications, vol. 14, no. 3, pp. 1309–1322, 2015. View at Publisher · View at Google Scholar
  30. A. Asheralieva and Y. Miyanaga, “An autonomous learning-based algorithm for joint channel and power level selection by D2D pairs in heterogeneous cellular networks,” IEEE Transactions on Communications, vol. 64, no. 9, pp. 3996–4012, 2016. View at Publisher · View at Google Scholar
  31. 3rd Generation Partnership Project, “Physical channels and modulation,” Technical Specification 3GPP TS 36.211 V9.1.0, 2010. View at Google Scholar
  32. 3rd Generation Partnership Project; Technical Specification, “E-UTRA; MAC protocol specification,” 3GPP TS 36.211 V12.5.0, 2015. View at Google Scholar
  33. L. Lei, Z. Zhong, C. Lin, and X. Shen, “Operator controlled device-to-device communications in LTE-advanced networks,” IEEE Wireless Communications, vol. 19, no. 3, pp. 96–104, 2012. View at Publisher · View at Google Scholar · View at Scopus
  34. H. Holma and A. Toskala, LTE for UMTS: Evolution to LTE-Advanced, John Wiley & Sons, New York, NY, USA, 2011.
  35. I. F. Akyildiz, D. M. Gutierrez-Estevez, and E. C. Reyes, “The evolution to 4G cellular systems: LTE-Advanced,” Physical Communication, vol. 3, no. 4, pp. 217–244, 2010. View at Publisher · View at Google Scholar · View at Scopus
  36. Y. Xu, J. Wang, Q. Wu, A. Anpalagan, and Y.-D. Yao, “Opportunistic spectrum access in unknown dynamic environment: a game-theoretic stochastic learning solution,” IEEE Transactions on Wireless Communications, vol. 11, no. 4, pp. 1380–1391, 2012. View at Publisher · View at Google Scholar · View at Scopus
  37. H. Li, “Multi-agent Q-learning of channel selection in multi-user cognitive radio systems: a two by two case,” in Proceedings of the IEEE International Conference on Systems, Man and Cybernetics (SMC '09), pp. 1893–1898, October 2009. View at Publisher · View at Google Scholar · View at Scopus
  38. T. Berthold, Heuristic Algorithms in Global MINLP Solvers, Verlag Dr. Hut, 2014.
  39. S. Burer and A. N. Letchford, “Non-convex mixed-integer nonlinear programming: a survey,” Surveys in Operations Research and Management Science, vol. 17, no. 2, pp. 97–106, 2012. View at Publisher · View at Google Scholar · View at MathSciNet · View at Scopus
  40. M. Fischetti and A. Lodi, “Local branching,” Mathematical Programming, vol. 98, no. 1–3, pp. 23–47, 2003. View at Publisher · View at Google Scholar · View at MathSciNet · View at Scopus
  41. A. Lodi, “The heuristic (dark) side of MIP solvers,” in Hybrid Metaheuristics, vol. 434 of Studies in Computational Intelligence, pp. 273–284, Springer, Berlin, Germany, 2013. View at Publisher · View at Google Scholar
  42. C. D'Ambrosio, A. Frangioni, L. Liberti, and A. Lodi, “A storm of feasibility pumps for nonconvex MINLP,” Mathematical Programming B, vol. 136, no. 2, pp. 375–402, 2012. View at Publisher · View at Google Scholar · View at MathSciNet · View at Scopus
  43. M. Fischetti and D. Salvagnin, “Feasibility pump 2.0,” Mathematical Programming Computation, vol. 1, no. 2-3, pp. 201–222, 2009. View at Publisher · View at Google Scholar · View at MathSciNet · View at Scopus
  44. C. D'Ambrosio, A. Frangioni, L. Liberti, and A. Lodi, “A storm of feasibility pumps for nonconvex MINLP,” Mathematical Programming, vol. 136, no. 2, pp. 375–402, 2012. View at Publisher · View at Google Scholar · View at MathSciNet · View at Scopus
  45. S. P. Boyd and L. Vandenberghe, Convex Optimization, Cambridge University Press, 2004. View at Publisher · View at Google Scholar · View at MathSciNet
  46. S. Leyffer, “Integrating SQP and branch-and-bound for mixed integer nonlinear programming,” Computational Optimization and Applications, vol. 18, no. 3, pp. 295–309, 2001. View at Publisher · View at Google Scholar · View at MathSciNet · View at Scopus
  47. R. Sun, M. Hong, and Z.-Q. Luo, “Joint downlink base station association and power control for max-min fairness: computation and complexity,” IEEE Journal on Selected Areas in Communications, vol. 33, no. 6, pp. 1040–1054, 2015. View at Publisher · View at Google Scholar · View at Scopus
  48. Q. Kuang, W. Utschick, and A. Dotzler, “Optimal joint user association and resource allocation in heterogeneous networks via sparsity pursuit,” https://arxiv.org/abs/1408.5091.
  49. K. Shen and W. Yu, “Distributed pricing-based user association for downlink heterogeneous cellular networks,” IEEE Journal on Selected Areas in Communications, vol. 32, no. 6, pp. 1100–1113, 2014. View at Publisher · View at Google Scholar · View at Scopus
  50. M. Peng, X. Xie, Q. Hu, J. Zhang, and H. V. Poor, “Contract-based interference coordination in heterogeneous cloud radio access networks,” IEEE Journal on Selected Areas in Communications, vol. 33, no. 6, pp. 1140–1153, 2015. View at Publisher · View at Google Scholar · View at Scopus
  51. D. Fooladivanda and C. Rosenberg, “Joint resource allocation and user association for heterogeneous wireless cellular networks,” IEEE Transactions on Wireless Communications, vol. 12, no. 1, pp. 248–257, 2013. View at Publisher · View at Google Scholar · View at Scopus
  52. M. Sanjabi, M. Razaviyayn, and Z.-Q. Luo, “Optimal joint base station assignment and beamforming for heterogeneous networks,” IEEE Transactions on Signal Processing, vol. 62, no. 8, pp. 1950–1961, 2014. View at Publisher · View at Google Scholar · View at MathSciNet · View at Scopus
  53. Q. Han, B. Yang, X. Wang, K. Ma, C. Chen, and X. Guan, “Hierarchical-game-based uplink power control in femtocell networks,” IEEE Transactions on Vehicular Technology, vol. 63, no. 6, pp. 2819–2835, 2014. View at Publisher · View at Google Scholar · View at Scopus
  54. IEEE Standards Association, IEEE 802®: Local and Metropolitan Area Network Standards: IEEE 802.11 Standard, 2012.
  55. OPNET, http://www.opnet.com.
  56. “Evolved Universal Terrestrial Radio Access (E-UTRA) and Evolved Universal Terrestrial Radio Access Network (E-UTRAN),” 3GPP TS 36.300, version V9.4.0, 2010.
  57. M. Bennis, S. Guruacharya, and D. Niyato, “Distributed learning strategies for interference mitigation in femtocell networks,” in Proceedings of the IEEE Global Telecommunications Conference (GLOBECOM '11), pp. 1–5, Houston, Tex, USA, December 2011. View at Publisher · View at Google Scholar