skip to main content
10.1145/1276958.1277158acmconferencesArticle/Chapter ViewAbstractPublication PagesgeccoConference Proceedingsconference-collections
Article

Generating large-scale neural networks through discovering geometric regularities

Published: 07 July 2007 Publication History

Abstract

Connectivity patterns in biological brains exhibit many repeating motifs. This repetition mirrors inherent geometric regularities in the physical world. For example, stimuli that excite adjacent locations on the retina map to neurons that are similarly adjacent in the visual cortex. That way, neural connectivity can exploit geometric locality in the outside world by employing local connections in the brain. If such regularities could be discovered by methods that evolve artificial neural networks (ANNs), then they could be similarly exploited to solve problems that would otherwise require optimizing too many dimensions to solve. This paper introduces such a method, called Hypercube-based Neuroevolution of Augmenting Topologies (HyperNEAT), which evolves a novel generative encoding called connective Compositional Pattern Producing Networks (connective CPPNs) to discover geometric regularities in the task domain. Connective CPPNs encode connectivity patterns as concepts that are independent of the number of inputs or outputs, allowing functional large-scale neural networks to be evolved. In this paper, this approach is tested in a simple visual task for which it effectively discovers the correct underlying regularity, allowing the solution to both generalize and scale without loss of function to an ANN of over eight million connections.

References

[1]
L. Altenberg. Evolving better representations through selective genome growth. In Proceedings of the IEEE World Congress on Computational Intelligence, pages 182--187, Piscataway, NJ, 1994. IEEE Press.
[2]
P. J. Angeline. Morphogenic evolutionary computations: Introduction, issues and examples. In J. R. McDonnell, R. G. Reynolds, and D. B. Fogel, editors, Evolutionary Programming IV: The Fourth Annual Conference on Evolutionary Programming, pages 387--401. MIT Press, 1995.
[3]
P. J. Bentley and S. Kumar. The ways to grow designs: A comparison of embryogenies for an evolutionary design problem. In Proceedings of the Genetic and Evolutionary Computation Conference (GECCO-1999), pages 35--43, San Francisco, 1999. Kaufmann.
[4]
J. C. Bongard. Evolving modular genetic regulatory networks. In Proceedings of the 2002 Congress on Evolutionary Computation, 2002.
[5]
D. B. Chklovskii and A. A. Koulakov. MAPS IN THE BRAIN: What can we learn from them? Annual Review of Neuroscience, 27:369--392, 2004.
[6]
P. M. Churchland. Some reductive strategies in cognitive neurobiology. Mind, 95:279--309, 1986.
[7]
D. Federici. Evolving a neurocontroller through a process of embryogeny. In S. Schaal, A. J. Ijspeert, A. Billard, S. Vijayakumar, J. Hallam, and Jean-Arcady, editors, Proceedings of the Eighth International Conference on Simulation and Adaptive Behavior (SAB--2004), pages 373--384, Cambridge, MA, 2004. MIT Press.
[8]
G. J. Goodhill and M. A. Carreira-Perpinn. Cortical columns. In L. Nadel, editor, Encyclopedia of Cognitive Science, volume 1, pages 845--851. MacMillan Publishers Ltd., London, 2002.
[9]
G. S. Hornby and J. B. Pollack. Creating high-level components with a generative representation for body-brain evolution. Artificial Life, 8(3), 2002.
[10]
A. Lindenmayer. Adding continuous components to L-systems. In G. Rozenberg and A. Salomaa, editors, L Systems, Lecture Notes in Computer Science 15, pages 53--68. Springer--Verlag, Heidelberg, Germany, 1974.
[11]
A. P. Martin. Increasing genomic complexity by gene duplication and the origin of vertebrates. The American Naturalist, 154(2):111--128, 1999.
[12]
J. F. Miller. Evolving a self-repairing, self-regulating, French flag organism. In Proceedings of the Genetic and Evolutionary Computation Conference (GECCO-2004), Berlin, 2004. Springer Verlag.
[13]
E. Mjolsness, D. H. Sharp, and J. Reinitz. A connectionist model of development. Journal of Theoretical Biology, 152:429--453, 1991.
[14]
O. Sporns. Network analysis, complexity, and brain function. Complexity, 8(1):56--60, 2002.
[15]
K. O. Stanley. Comparing artificial phenotypes with natural biological patterns. In Proceedings of the Genetic and Evolutionary Computation Conference (GECCO) Workshop Program, New York, NY, 2006. ACM Press.
[16]
K. O. Stanley. Exploiting regularity without development. In Proceedings of the AAAI Fall Symposium on Developmental Systems, Menlo Park, CA, 2006. AAAI Press.
[17]
K. O. Stanley. Compositional pattern producing networks: A novel abstraction of development. Genetic Programming and Evolvable Machines Special Issue on Developmental Systems, 2007. To appear.
[18]
K. O. Stanley, B. D. Bryant, and R. Miikkulainen. Real-time neuroevolution in the NERO video game. IEEE Transactions on Evolutionary Computation Special Issue on Evolutionary Computation and Games, 9(6):653--668, 2005.
[19]
K. O. Stanley and R. Miikkulainen. Evolving neural networks through augmenting topologies. Evolutionary Computation, 10:99--127, 2002.
[20]
K. O. Stanley and R. Miikkulainen. A taxonomy for artificial embryogeny. Artificial Life, 9(2):93--130, 2003.
[21]
K. O. Stanley and R. Miikkulainen. Competitive coevolution through evolutionary complexification. Journal of Artificial Intelligence Research, 21:63--100, 2004.
[22]
A. Turing. The chemical basis of morphogenesis. Philosophical Transactions of the Royal Society B, 237:37--72, 1952.
[23]
J. D. Watson, N. H. Hopkins, J. W. Roberts, J. A. Steitz, and A. M. Weiner. Molecular Biology of the Gene Fourth Edition. The Benjamin Cummings Publishing Company, Inc., Menlo Park, CA, 1987.
[24]
M. J. Zigmond, F. E. Bloom, S. C. Landis, J. L. Roberts, and L. R. Squire, editors. Fundamental Neuroscience. Academic Press, London, 1999.

Cited By

View all
  • (2024)Comparing Robot Controller Optimization Methods on Evolvable MorphologiesEvolutionary Computation10.1162/evco_a_0033432:2(105-124)Online publication date: 3-Jun-2024
  • (2023)Emerging Modularity During the Evolution of Neural NetworksJournal of Artificial Intelligence and Soft Computing Research10.2478/jaiscr-2023-001013:2(107-126)Online publication date: 11-Mar-2023
  • (2022)Hill-Climb-Assembler Encoding: Evolution of Small/Mid-Scale Artificial Neural Networks for Classification and Control ProblemsElectronics10.3390/electronics1113210411:13(2104)Online publication date: 5-Jul-2022
  • Show More Cited By

Index Terms

  1. Generating large-scale neural networks through discovering geometric regularities

      Recommendations

      Comments

      Information & Contributors

      Information

      Published In

      cover image ACM Conferences
      GECCO '07: Proceedings of the 9th annual conference on Genetic and evolutionary computation
      July 2007
      2313 pages
      ISBN:9781595936974
      DOI:10.1145/1276958
      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

      Sponsors

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      Published: 07 July 2007

      Permissions

      Request permissions for this article.

      Check for updates

      Author Tags

      1. HyperNEAT
      2. NEAT
      3. compositional pattern producing networks
      4. large-scale artificial neural networks

      Qualifiers

      • Article

      Conference

      GECCO07
      Sponsor:

      Acceptance Rates

      GECCO '07 Paper Acceptance Rate 266 of 577 submissions, 46%;
      Overall Acceptance Rate 1,669 of 4,410 submissions, 38%

      Contributors

      Other Metrics

      Bibliometrics & Citations

      Bibliometrics

      Article Metrics

      • Downloads (Last 12 months)21
      • Downloads (Last 6 weeks)1
      Reflects downloads up to 16 Feb 2025

      Other Metrics

      Citations

      Cited By

      View all
      • (2024)Comparing Robot Controller Optimization Methods on Evolvable MorphologiesEvolutionary Computation10.1162/evco_a_0033432:2(105-124)Online publication date: 3-Jun-2024
      • (2023)Emerging Modularity During the Evolution of Neural NetworksJournal of Artificial Intelligence and Soft Computing Research10.2478/jaiscr-2023-001013:2(107-126)Online publication date: 11-Mar-2023
      • (2022)Hill-Climb-Assembler Encoding: Evolution of Small/Mid-Scale Artificial Neural Networks for Classification and Control ProblemsElectronics10.3390/electronics1113210411:13(2104)Online publication date: 5-Jul-2022
      • (2022)Adaptive Neural Network Structure Optimization Algorithm Based on Dynamic NodesCurrent Issues in Molecular Biology10.3390/cimb4402005644:2(817-832)Online publication date: 7-Feb-2022
      • (2022)A neuroevolutionary method for knowledge space constructionComputer Science and Information Systems10.2298/CSIS210820004S19:2(763-781)Online publication date: 2022
      • (2022)Neuro-Evolutionary Direct Policy Search for Multiobjective Optimal ControlIEEE Transactions on Neural Networks and Learning Systems10.1109/TNNLS.2021.307196033:10(5926-5938)Online publication date: Oct-2022
      • (2022)A comprehensive survey on optimizing deep learning models by metaheuristicsArtificial Intelligence Review10.1007/s10462-021-09992-055:2(829-894)Online publication date: 1-Feb-2022
      • (2021)A Systematic Literature Review of the Successors of “NeuroEvolution of Augmenting Topologies”Evolutionary Computation10.1162/evco_a_0028229:1(1-73)Online publication date: Mar-2021
      • (2021)Hill Climb Modular Assembler EncodingKnowledge-Based Systems10.1016/j.knosys.2021.107493232:COnline publication date: 28-Nov-2021
      • (2021)Evolutionary design of neural network architectures: a review of three decades of researchArtificial Intelligence Review10.1007/s10462-021-10049-555:3(1723-1802)Online publication date: 27-Jul-2021
      • Show More Cited By

      View Options

      Login options

      View options

      PDF

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      Figures

      Tables

      Media

      Share

      Share

      Share this Publication link

      Share on social media