Cooperation in stochastic games through communication

ABSTRACT

The application of reinforcement learning principles to the search of equilibrium policies in stochastic games (SGs) has met with some success ([3], [4], [2]). The key insight of this approach is that each agent can learn his own ß-discounted reward equilibrium policy by keeping track of Q-values of all the agents including himself, and considering the Q-value matrix for each state as his payoff matrix. Each agent sees what actions other agents take, and what payoffs they receive. There is some evidence that in practice, agents that do not observe the actions and payoffs of other agents (hereby denoted as imperfectly observing agents), can still learn adversarial equilibrium (AE) policies in general-sum SGs ([1]) using naive Q-learning. Considering the Prisoners' Dilemma stage game (Table 1) as an abstraction of a SG, this implies that, even by ignoring other agents' play, agents still learn to play DD, which is the adversarial equilibrium joint action. The payoff received in DD can be thought of as each agent's security level.

REFERENCES

• 1: Michael H. Bowling, Convergence Problems of General-Sum Multiagent Reinforcement Learning, Proceedings of the Seventeenth International Conference on Machine Learning, p.89-94, June 29-July 02, 2000
• 2: A. Greenwald and K. Hall. Correlated-Q learning. Twentieth International Conference on Machine Learning, pages 242--249, 2003.
• 3: Junling Hu , Michael P. Wellman, Nash q-learning for general-sum stochastic games, The Journal of Machine Learning Research, 4, p.1039-1069, 12/1/2003
• 4: Michael L. Littman, Friend-or-Foe Q-learning in General-Sum Games, Proceedings of the Eighteenth International Conference on Machine Learning, p.322-328, June 28-July 01, 2001