Call admission control in cellular networks: a reinforcement learning solution

ABSTRACT

In this paper, we address the call admission control (CAC) problem in a cellular network that handles several classes of traffic with different resource requirements. The problem is formulated as a semi-Markov decision process (SMDP) problem. We use a real-time reinforcement learning (RL) [neuro-dynamic programming (NDP)] algorithm to construct a dynamic call admission control policy. We show that the policies obtained using our TQ-CAC and NQ-CAC algorithms, which are two different implementations of the RL algorithm, provide a good solution and are able to earn significantly higher revenues than classical solutions such as guard channel. A large number of experiments illustrates the robustness of our policies and shows how they improve quality of service (QoS) and reduce call-blocking probabilities of handoff calls even with variable traffic conditions.

REFERENCES

• 1: 1. Yoon CH, Un CK. Performance of personal portable radio telephone systems with and without guard channels, IEEE Journal on Selected Areas in Communications (JSAC'1993) August 1993; 11: 911-917.
• 2: 2. Tekinay S, Jabbari B. Handover and channel assignment in mobile cellular networks, IEEE Communications Magazine November 1991; 29.
• 3: 3. Katzela I, Naghshineh M. Channel-assignment schemes for cellular mobile telecommunications systems, IEEE Personal Communications Magazine, June 1996.
• 4: 4. Nie J, Haykin S. A Q-learning based dynamic channel assignment technique for mobile communication systems, IEEE Transactions on Vehicular Technology September 1999; 48: No. 5.
• 5: 5. Singh SP, Bertsekas DP. Reinforcement learning for dynamic channel allocation in cellular telephone systems, In Mozer M et al. (eds.), Neural Information Processing Systems (NIPS), Vol. 9, 1997; MIT Press, 974-980.
• 6: 6. Marbach P, Mihatsch O, Tsitsikils JN. Call admission control and routing in integrated services networks using neuro-dynamic programming, IEEE Journal on Selected Areas in Communications (JSAC'2000), February 2000; 18: No. 2, 197-208.
• 7: 7. Tong H, Brown TX. Adaptive call admission control under quality of service constraint: a reinforcement learning solution, IEEE Journal on Selected Areas in Communications (JSAC'2000), February 2000; 18: No. 2, 209-221.
• 8: 8. Ramjee R, Nagarajan R, Towsley D. On optimal call admission control in cellular networks, IEEE INFOCOM, March 1996; 43-50, San Francisco, CA.
• 9: 9. Littman M, Boyan J. Packet routing in dynamically changing networks: a reinforcement learning approach, Advances in Neural Information Processing Systems (NIPS), 1994; Vol. 6: 671-678, San Francisco, CA.
• 10: 10. Banerjee N, Das S. Fast determination of QoS-based multicast routes in wireless networking using genetic algorithms, ICC 2001, 2001; Vol. 8: 2588-2592, Helsinki, Finland.
• 11: 11. Mitra M, Reiman I, Wang J. Robust dynamic admission control for unified cell and call QoS in statistical multiplexers, IEEE Journal on Selected Areas in Communications (JSAC'1998), 1998; 16: No. 5, 692-707.
• 12: Thomas M. Mitchell, Machine Learning, McGraw-Hill Higher Education, 1997
• 13: Andrew G. Barto , Steven J. Bradtke , Satinder P. Singh, Learning to act using real-time dynamic programming, Artificial Intelligence, v.72 n.1-2, p.81-138, Jan. 1995 [doi>10.1016/0004-3702(94)00011-O]
• 14: Christopher J. C. H. Watkins , Peter Dayan, Technical Note: $\backslash cal\; Q$-Learning, Machine Learning, v.8 n.3-4, p.279-292, May 1992 [doi>10.1007/BF00992698]
• 15: 15. Ritter M, Tran-Gia P (Eds.) COST242, Multi-rate models for dimensioning and performance evaluation of multiservice networks, June 1994.
• 16: Peter Marbach , Oliver Mihatsch , Miriam Schulte , John N. Tsitsiklis, Reinforcement learning for call admission control and routing in integrated service networks, Proceedings of the 1997 conference on Advances in neural information processing systems 10, p.922-928, July 1998, Denver, Colorado, United States
• 17: 17. Tong H. Adaptive Admission Control for Broadband Communications, PhD Thesis, Summer 1999; University of Colorado, Boulder.
• 18: 18. Brown TX. Low-power wireless communication via reinforcement learning. In Solla SA, Leen TK, Muller K-R (Eds.), Advances in Neural Information Processing Systems (NIPS), 2000; 12: 893-899, MIT Press.
• 19: Sidi-Mohammed Senouci , André-Luc Beylot , Guy Pujolle, Call Admission Control for Multimedia Cellular Networks Using Neuro-dynamic Programming, Proceedings of the Second International IFIP-TC6 Networking Conference on Networking Technologies, Services, and Protocols; Performance of Computer and Communication Networks; and Mobile and Wireless Communications, p.1208-1213, May 19-24, 2002
• 20: 20. Beylot A-L, Boumerdassi S, Pujolle G, NACR: a new adaptive channel reservation in cellular communication systems, Telecommunication Systems, 2001; 17: Nos. 1-2, 233-241, Kluwer.
• 21: 21. Chao C, Chen W. Connection admission control for mobile multiple-class personal communications networks, IEEE Journal on Selected Areas in Communications (JSAC'1997), October 1997; 15: No. 8, 1618-1626.

INDEX TERMS

Primary Classification:
  C. Computer Systems Organization
  C.2 COMPUTER-COMMUNICATION NETWORKS
      C.2.2 Network Protocols
          Subjects: Applications (SMTP, FTP, etc.)

Additional Classification:
  C. Computer Systems Organization
  C.1 PROCESSOR ARCHITECTURES
      C.1.3 Other Architecture Styles
          Subjects: Cellular architecture (e.g., mobile)
  C.2 COMPUTER-COMMUNICATION NETWORKS
      C.2.1 Network Architecture and Design
          Subjects: Wireless communication

  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.6 Learning
          Subjects: Concept learning; Connectionism and neural nets

General Terms:
Algorithms, Experimentation, Performance