Spectrum management of cognitive radio using multi-agent reinforcement learning

ABSTRACT

Wireless cognitive radio (CR) is a newly emerging paradigm that attempts to opportunistically transmit in licensed frequencies, without affecting the pre-assigned users of these bands. To enable this functionality, such a radio must predict its operational parameters, such as transmit power and spectrum. These tasks, collectively called spectrum management, is difficult to achieve in a dynamic distributed environment, in which CR users may only take local decisions, and react to the environmental changes. In this paper, we introduce a multi-agent reinforcement learning approach based spectrum management. Our approach uses value functions to evaluate the desirability of choosing different transmission parameters, and enables efficient assignment of spectrums and transmit powers by maximizing long-term reward. We then investigate various real-world scenarios, and compare the communication performance using different sets of learning parameters. We also apply Kanerva-based function approximation to improve our approach's ability to handle large cognitive radio networks and evaluate its effect on communication performance. We conclude that our reinforcement learning based spectrum management can significantly reduce the interference to the licensed users, while maintaining a high probability of successful transmissions in a cognitive radio ad hoc network.

References

1. I. F. Akyildiz, W.-Y. Lee, and K. R. Chowdhury. Crahns: Cognitive radio ad hoc networks. Ad Hoc Networks Journal (Elsevier), 7(5):810--836, July 2009.
2. I. F. Akyildiz, W.-Y. Lee, and K. R. Chowdhury. Spectrum management in cognitive radio ad hoc networks. IEEE Network, 23(4):6--12, July 2009.
3. I. F. Akyildiz, W.-Y. Lee, M. C. Vuran, and S. Mohanty. Next generation / dynamic spectrum access / cognitive radio wireless networks: a survey. Computer Networks Journal (Elsevier), 50:2127--2159, September 2006.
4. J. Albus. Brains, Behaviour, and Robotics. McGraw-Hill, 1981.
5. L. Baird. Residual algorithms: Reinforcement learning with function approximation. In Proc. of the 12th Intl. Conf. on Machine Learning. Morgan Kaufmann, 1995.
6. FCC. Spectrum policy task force report. ET Docket No. 02--155, Nov. 2002.
7. G. J. Gordon. Stable function approximation in dynamic programming. In Proc. of Intl. Conf. on Machine Learning, 1995.
8. S. Haykin. Cognitive radio: Brain-empowered wireless communications. IEEE Journal on Selected Areas in Communications, 23(2):201--220, February 2005.
9. G. Hinton. Distributed representations. Technical Report, Department of Computer Science, Carnegie-Mellon University, Pittsburgh, 1984.
10. IEEE Std 802.11b Specification. Ieee std 802.11b-1999/cor 1--2001, 2001.
11. P. Kanerva. Sparse Distributed Memory. MIT Press, 1988.
12. J. Mitola III. Cognitive radio for flexible mobile multimedia communication. In Proc. IEEE International Workshop on Mobile Multimedia Communications (MoMuC) 1999, pages 3--10, November 1999.
13. J. Proakis. Digital Communications. McGraw-Hill Science/Engineering/Math, August 2000.
14. B. Ratitch and D. Precup. Sparse distributed memories for on-line value-based reinforcement learning. In Proc. of the European Conf. on Machine Learning, 2004.
15. G. L. Stüber. Principles of mobile communication (2nd ed.). Kluwer Academic Publishers, 2001.
16. R. Sutton. Generalization in reinforcement learning: Successful examples using sparse coarse coding. In Proc. of Conf. on Neural Information Processing Systems, 1995.
17. R. Sutton and A. Barto. Reinforcement Learning: An Introduction. Bradford Books, 1998.
18. M. D. Waltz and K. S. Fu. A heuristic approach to reinforcment learning control systems. In IEEE Transactions on Automatic Control, 10:390--398., 1965.
19. C. Watkins. Learning from delayed rewards. Ph.D thesis, Cambridge Univeristy, Cambridge, England, 1989.
20. M. Wooldridge. An introduction to multiagent systems. In John Wiley Sons Ltd, ISBN 0-471-49691-X, 2002.

Index Terms

• General and reference

  • Cross-computing tools and techniques

    • Performance

• Information systems

  • Information systems applications

• Social and professional topics

  • Professional topics

    • Management of computing and information systems

      • Software management

        • Software selection and adaptation

• Software and its engineering

  • Software creation and management

    • Software verification and validation

      • Empirical software validation

  • Software organization and properties

    • Extra-functional properties

      • Software performance

    • Software functional properties

      • Formal methods