Subscribe
Issue No.11 - Nov. (2013 vol.12)
pp: 2155-2166
Xianfu Chen , VTT Technical Research Centre of Finland, Oulu
Zhifeng Zhao , Zhejiang University, Hangzhou
Honggang Zhang , Zhejiang University, Hangzhou
ABSTRACT
As the scarce spectrum resource is becoming overcrowded, cognitive radio indicates great flexibility to improve the spectrum efficiency by opportunistically accessing the authorized frequency bands. One of the critical challenges for operating such radios in a network is how to efficiently allocate transmission powers and frequency resource among the secondary users (SUs) while satisfying the quality-of-service constraints of the primary users. In this paper, we focus on the noncooperative power allocation problem in cognitive wireless mesh networks formed by a number of clusters with the consideration of energy efficiency. Due to the SUs' dynamic and spontaneous properties, the problem is modeled as a stochastic learning process. We first extend the single-agent $(Q)$-learning to a multiuser context, and then propose a conjecture-based multiagent $(Q)$-learning algorithm to achieve the optimal transmission strategies with only private and incomplete information. An intelligent SU performs $(Q)$-function updates based on the conjecture over the other SUs' stochastic behaviors. This learning algorithm provably converges given certain restrictions that arise during the learning procedure. Simulation experiments are used to verify the performance of our algorithm and demonstrate its effectiveness of improving the energy efficiency.
INDEX TERMS
Resource management, Interference, Games, Stochastic processes, Signal to noise ratio, Algorithm design and analysis, Wireless communication,reinforcement learning, Cognitive radio, resource allocation, algorithm/protocol design and analysis
CITATION
Xianfu Chen, Zhifeng Zhao, Honggang Zhang, "Stochastic Power Adaptation with Multiagent Reinforcement Learning for Cognitive Wireless Mesh Networks", IEEE Transactions on Mobile Computing, vol.12, no. 11, pp. 2155-2166, Nov. 2013, doi:10.1109/TMC.2012.178
REFERENCES
 [1] Federal Communication Commission, "Commission Seeks Public Comment on Spectrum Policy Task Force Report," Technical Report, ET Docket No. 02-135, Nov. 2002. [2] I.F. Akyildiz, W.-Y. Lee, and K.R. Chowdhury, "CRAHNs: Cognitive Radio Ad Hoc Networks," Ad Hoc Networks, vol. 7, no. 5, pp. 810-836, July 2009. [3] J. Mitola and G.Q. Maguire, "Cognitive Radios: Making Software Radios More Personal," IEEE Personal Comm., vol. 6, no. 4, pp. 13-18, Aug. 1999. [4] S. Haykin, "Cognitive Radio: Brain-Empowered Wireless Communications," IEEE J. Selected Areas in Comm., vol. 23, no. 2, pp. 201-220, Feb. 2005. [5] T. Chen, H. Zhang, G.M. Maggio, and I. Chlamtac, "CogMesh: A Cluster-Based Cognitive Radio Network," Proc. IEEE Second Int'l Symp. New Frontiers in Dynamic Spectrum Access Networks (DySPAN), Apr. 2007. [6] Y. Shi and Y.T. Hou, "A Distributed Optimization Algorithm for Multi-Hop Cognitive Radio Networks," Proc. IEEE INFOCOM, Apr. 2008. [7] J. Mietzner, L. Lampe, and R. Schober, "Distributed Transmit Power Allocation for Relay-Assisted Cognitive-Radio Systems," IEEE Trans. Wireless Comm., vol. 8, pp. 5187-5201, Oct. 2009. [8] F. Wang, M. Krunz, and S. Cui, "Price-Based Spectrum Management in Cognitive Radio Networks," IEEE J. Selected Topics in Signal Processing, vol. 2, no. 1, pp. 74-87, Feb. 2008. [9] A.T. Hoang, Y.-C. Liang, and M.H. Islam, "Power Control and Channel Allocation in Cognitive Radio Networks with Primary Users' Cooperation," IEEE Trans. Mobile Computing, vol. 9, no. 3, pp. 348-360, Mar. 2010. [10] Y.T. Hou, Y. Shi, and H.D. Sherali, "Optimal Spectrum Sharing for Multi-Hop Software Defined Radio Networks," Proc. IEEE INFOCOM, May 2007. [11] S. Gao, L. Qian, and D. Vaman, "Distributed Energy Efficient Spectrum Access in Cognitive Radio Wireless Ad Hoc Networks," IEEE Trans. Wireless Comm., vol. 8, no. 10, pp. 5202-5213, Oct. 2009. [12] Y. Wu and D.H.K. Tsang, "Distributed Power Allocation Algorithm for Spectrum Sharing Cognitive Radio Networks with QoS Guarantee," Proc. IEEE INFOCOM, Apr. 2009. [13] J. Hu and M.P. Wellman, "Nash $Q$ -Learning for General-Sum Stochastic Games," J. Machine Learning Research, vol. 4, pp. 1039-1069, Dec. 2003. [14] R.S. Sutton and A.G. Barto, Reinforcement Learning: An Introduction. MIT, 1998. [15] Y. Xing and R. Chandramouli, "Stochastic Learning Solution for Distributed Discrete Power Control Game in Wireless Data Networks," IEEE/ACM Trans. Networking, vol. 16, no. 4, pp. 932-944, Aug. 2008. [16] F. Fu and M. van der Schaar, "Learning to Compete for Resources in Wireless Stochastic Games," IEEE Trans. Vehicular Technology, vol. 58, no. 4, pp. 1904-1919, May 2009. [17] J. Lunden, V. Koivunen, S.R. Kulkarni, and H.V. Poor, "Reinforcement Learning Based Distributed Multiagent Sensing Policy for Cognitive Radio Networks," Proc. IEEE Fifth Symp. New Frontiers in Dynamic Spectrum Access Networks (DySPAN), May 2011. [18] H. Li, "Multiagent $Q$ -Learning for Aloha-Like Spectrum Access in Cognitive Radio Systems," EURASIP J. Wireless Comm. and Networking, vol. 2010, article 56, 2010. [19] S. Filippi, O. Cappe, and A. Garivier, "Optimally Sensing a Single Channel Without Prior Information: The Tiling Algorithm and Regret Bounds," IEEE J. Selected Topics in Signal Processing, vol. 5, no. 1, pp. 68-76, Feb. 2011. [20] A. Greenwald and K. Hall, "Correlated-$Q$ Learning," Proc. 20th Int'l Conf. Machine Learning (ICML), Aug. 2003. [21] E.R. Gomes and R. Kowalczyk, "Dynamic Analysis of Multiagent $Q$ -Learning with $\epsilon$ -Greedy Exploration," Proc. 26th Int'l Conf. Machine Learning (ICML), June 2009. [22] C.J.C.H. Watkins and P. Dayan, "$Q$ -Learning," Machine Learning, vol. 8, pp. 279-292, 1992. [23] F. Meshkati, M. Chiang, H.V. Poor, and S.C. Schwartz, "A Game-Theoretic Approach to Energy-Efficient Power Control in Multicarrier CDMA Systems," IEEE J. Selected Areas in Comm., vol. 24, no. 6, pp. 1115-1129, June 2006. [24] D. Fudenberg and J. Tirole, Game Theory. MIT, 1992. [25] C.U. Saraydar, N.B. Mandayam, and D.J. Goodman, "Efficient Power Control via Pricing in Wireless Data Networks," IEEE Trans. Comm., vol. 50, no. 2, pp. 291-303, Feb. 2002. [26] S. Huang, X. Liu, and Z. Ding, "Decentralized Cognitive Radio Control Based on Inference from Primary Link Control Information," IEEE J. Selected Areas Comm., vol. 29, no. 2, pp. 394-406, Feb. 2011. [27] M.P. Wellman and J. Hu, "Conjectural Equilibrium in Multiagent Learning," Machine Learning, vol. 33, pp. 179-200, Dec. 1998. [28] A. Jean-Marie and M. Tidball, "Adapting Behaviors through a Learning Process," J. Economic Behavior and Organization, vol. 60, no. 3, pp. 399-422, July 2006. [29] C. Szepesvari and M.L. Littman, "A Unified Analysis of Value-Function-Based Reinforcement Learning Algorithms," Neural Computation, vol. 11, no. 8, pp. 2017-2060, Nov. 1999.