skip to main content
10.5555/1402383.1402428acmconferencesArticle/Chapter ViewAbstractPublication PagesaamasConference Proceedingsconference-collections
research-article

Analysis of an evolutionary reinforcement learning method in a multiagent domain

Published: 12 May 2008 Publication History

Abstract

Many multiagent problems comprise subtasks which can be considered as reinforcement learning (RL) problems. In addition to classical temporal difference methods, evolutionary algorithms are among the most promising approaches for such RL problems. The relative performance of these approaches in certain subdomains (e. g. multiagent learning) of the general RL problem remains an open question at this time. In addition to theoretical analysis, benchmarks are one of the most important tools for comparing different RL methods in certain problem domains. A recently proposed multiagent RL benchmark problem is the RoboCup Keepaway benchmark. This benchmark is one of the most challenging multiagent learning problems because its state-space is continuous and high dimensional, and both the sensors and the actuators are noisy. In this paper we analyze the performance of the neuroevolutionary approach called Evolutionary Acquisition of Neural Topologies (EANT) in the Keepaway benchmark, and compare the results obtained using EANT with the results of other algorithms tested on the same benchmark.

References

[1]
P. J. Angeline, G. M. Saunders, and J. B. Pollack. An evolutionary algorithm that constructs recurrent neural networks. IEEE Transactions on Neural Networks, 5:54--65, 1994.
[2]
J. E. Baker. Adaptive selection methods for genetic algorithms. In Proceedings of the 1st International Conference on Genetic Algorithms, pages 101--111, Mahwah, NJ, USA, 1985. Lawrence Erlbaum Associates, Inc.
[3]
P. Bentley and S. Kumar. Three ways to grow designs: A comparison of embryogenies for an evolutionary design problem. In Proceedings of the Genetic and Evolutionary Computation Conference, volume 1, pages 35--43, Orlando, Florida, USA, 13--17 July 1999. Morgan Kaufmann.
[4]
J. C. Bongard and R. Pfeifer. Repeated structure and dissociation of genotypic and phenotypic complexity in artificial ontogeny. In Proceedings of the Genetic and Evolutionary Computation Conference, pages 829--836, 2001.
[5]
F. Gomez, J. Schmidhuber, and R. Miikkulainen. Efficient non-linear control through neuroevolution. In Proceedings of the European Conference on Machine Learning, 2006.
[6]
F. Gruau. Neural Network Synthesis Using Cellular Encoding and the Genetic Algorithm. PhD thesis, Ecole Normale Superieure de Lyon, Laboratoire de l'Informatique du Parallelisme, France, January 1994.
[7]
Y. Kassahun. Towards a Unified Approach to Learning and Adaptation. PhD thesis, Technical Report 0602, Institute of Computer Science and Applied Mathematics, Christian-Albrechts University, Kiel, Germany, Feb 2006.
[8]
Y. Kassahun, M. Edgington, J. H. Metzen, G. Sommer, and F. Kirchner. A common genetic encoding for both direct and indirect encodings of networks. In Proceedings of the Genetic and Evolutionary Computation Conference, pages 1029--1036, 2007.
[9]
Y. Kassahun, J. H. Metzen, J. de Gea, M. Edgington, and F. Kirchner. A general framework for encoding and evolving neural networks. In Proceedings of the 30th Annual German Conference on Artificial Intelligence, pages 205--219, 9 2007.
[10]
H. Kitano. Designing neural networks using genetic algorithms with graph generation system. Complex Systems, 4:461--476, 1990.
[11]
A. Lindenmayer. Mathematical models for cellular interactions in development, parts I and II. Journal of Theoretical Biology, 18:280--315, 1968.
[12]
S. Nolfi and D. Floreano. Evolutionary Robotics. The Biology, Intelligence, and Technology of Self-Organizing Machines. MIT Press, Massachusetts, London, 2000.
[13]
S. Nolfi and D. Parisi. Growing neural networks. Technical Report PCIA-91-15, Institute of Psychology, Rome, 1991.
[14]
J. Reisinger and R. Miikkulainen. Acquiring evolvability through adaptive representations. In Proceedings of the Genetic and Evolutionary Computation Conference, pages 1045--1052, New York, NY, USA, 2007. ACM Press.
[15]
H.-P. P. Schwefel. Evolution and Optimum Seeking: The Sixth Generation. John Wiley & Sons, Inc., New York, NY, USA, 1993.
[16]
B. Sendhoff and M. Kreutz. Variable encoding of modular neural networks for time series prediction. In Congress on Evolutionary Computation, pages 259--266, 1999.
[17]
N. Siebel, J. Krause, and G. Sommer. Efficient learning of neural networks with evolutionary algorithms. In Proceedings of the 29th German Symposium for Pattern Recognition, pages 466--475, 2007.
[18]
K. O. Stanley. Efficient Evolution of Neural Networks through Complexification. PhD thesis, Artificial Intelligence Laboratory. The University of Texas at Austin., Austin, USA, Aug 2004.
[19]
P. Stone, G. Kuhlmann, M. E. Taylor, and Y. Liu. Keepaway soccer: From machine learning testbed to benchmark. In RoboCup-2005: Robot Soccer World Cup IX, volume 4020, pages 93--105. Springer Verlag, Berlin, 2006.
[20]
P. Stone, R. S. Sutton, and G. Kuhlmann. Reinforcement learning for RoboCup-soccer Keepaway. Adaptive Behavior, 13(3):165--188, 2005.
[21]
R. Sutton and A. Barto. Reinforcement Learning: An Introduction. MIT Press, Massachusetts, London, 1998.
[22]
M. E. Taylor, S. Whiteson, and P. Stone. Comparing evolutionary and temporal difference methods in a reinforcement learning domain. In Proceedings of the Genetic and Evolutionary Computation Conference, pages 1321--1328, 2006.
[23]
J. Vaario, A. Onitsuka, and K. Shimohara. Formation of neural structures. In Proceedings of the Fourth European Conference on Articial Life, pages 214--223, 1997.
[24]
S. Whiteson and P. Stone. Evolutionary function approximation for reinforcement learning. Journal of Machine Learning Research, 7:877--917, May 2006.
[25]
S. Whiteson, M. E. Taylor, and P. Stone. Empirical studies in action selection for reinforcement learning. Adaptive Behavior, 15(1):33--50, March 2007.
[26]
D. Whitley, S. Dominic, R. Das, and C. W. Anderson. Genetic reinforcement learning for neurocontrol problems. Machine Learning, 13:259--284, 1993.
[27]
X. Yao. Evolving artificial neural networks. Proceedings of the IEEE, 87(9):1423--1447, 1999.

Cited By

View all
  • (2021)Evolutionary Machine Learning: A SurveyACM Computing Surveys10.1145/346747754:8(1-35)Online publication date: 4-Oct-2021
  • (2018)Model parameter adaptive instance-based policy optimization for episodic control tasks of nonholonomic systemsProceedings of the Genetic and Evolutionary Computation Conference Companion10.1145/3205651.3208295(1426-1433)Online publication date: 6-Jul-2018
  • (2014)On Diversity, Teaming, and Hierarchical PoliciesRevised Selected Papers of the 17th European Conference on Genetic Programming - Volume 859910.1007/978-3-662-44303-3_7(75-86)Online publication date: 23-Apr-2014
  • Show More Cited By

Index Terms

  1. Analysis of an evolutionary reinforcement learning method in a multiagent domain

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    AAMAS '08: Proceedings of the 7th international joint conference on Autonomous agents and multiagent systems - Volume 1
    May 2008
    565 pages
    ISBN:9780981738109

    Sponsors

    Publisher

    International Foundation for Autonomous Agents and Multiagent Systems

    Richland, SC

    Publication History

    Published: 12 May 2008

    Check for updates

    Author Tags

    1. neuroevolution
    2. reinforcement learning

    Qualifiers

    • Research-article

    Conference

    AAMAS08
    Sponsor:

    Acceptance Rates

    Overall Acceptance Rate 1,155 of 5,036 submissions, 23%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)6
    • Downloads (Last 6 weeks)0
    Reflects downloads up to 17 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2021)Evolutionary Machine Learning: A SurveyACM Computing Surveys10.1145/346747754:8(1-35)Online publication date: 4-Oct-2021
    • (2018)Model parameter adaptive instance-based policy optimization for episodic control tasks of nonholonomic systemsProceedings of the Genetic and Evolutionary Computation Conference Companion10.1145/3205651.3208295(1426-1433)Online publication date: 6-Jul-2018
    • (2014)On Diversity, Teaming, and Hierarchical PoliciesRevised Selected Papers of the 17th European Conference on Genetic Programming - Volume 859910.1007/978-3-662-44303-3_7(75-86)Online publication date: 23-Apr-2014
    • (2012)CMA-TWEANNProceedings of the 14th annual conference on Genetic and evolutionary computation10.1145/2330163.2330288(903-910)Online publication date: 7-Jul-2012
    • (2010)Learning complementary multiagent behaviorsRoboCup 200910.5555/2167873.2167887(153-165)Online publication date: 1-Jan-2010
    • (2009)Evolving an autonomous agent for non-Markovian reinforcement learningProceedings of the 11th Annual conference on Genetic and evolutionary computation10.1145/1569901.1570034(971-978)Online publication date: 8-Jul-2009
    • (2009)Fuzzy CMAC with automatic state partition for reinforcementlearningProceedings of the first ACM/SIGEVO Summit on Genetic and Evolutionary Computation10.1145/1543834.1543891(421-428)Online publication date: 12-Jun-2009
    • (2008)Evolving Neural Networks for Online Reinforcement LearningProceedings of the 10th International Conference on Parallel Problem Solving from Nature --- PPSN X - Volume 519910.5555/2951659.2951714(518-527)Online publication date: 13-Sep-2008

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media