skip to main content
10.1145/3319619.3326815acmconferencesArticle/Chapter ViewAbstractPublication PagesgeccoConference Proceedingsconference-collections
research-article

Prediction of neural network performance by phenotypic modeling

Published: 13 July 2019 Publication History

Abstract

Surrogate models are used to reduce the burden of expensive-to-evaluate objective functions in optimization. By creating models which map genomes to objective values, these models can estimate the performance of unknown inputs, and so be used in place of expensive objective functions. Evolutionary techniques such as genetic programming or neuroevolution commonly alter the structure of the genome itself. A lack of consistency in the genotype is a fatal blow to data-driven modeling techniques: interpolation between points is impossible without a common input space. However, while the dimensionality of genotypes may differ across individuals, in many domains, such as controllers or classifiers, the dimensionality of the input and output remains constant. In this work we leverage this insight to embed differing neural networks into the same input space. To judge the difference between the behavior of two neural networks, we give them both the same input sequence, and examine the difference in output. This difference, the phenotypic distance, can then be used to situate these networks into a common input space, allowing us to produce surrogate models which can predict the performance of neural networks regardless of topology. In a robotic navigation task, we show that models trained using this phenotypic embedding perform as well or better as those trained on the weight values of a fixed topology neural network. We establish such phenotypic surrogate models as a promising and flexible approach which enables surrogate modeling even for representations that undergo structural changes.

References

[1]
C. C. Aggarwal, A. Hinneburg, and D. A. Keim. 2001. On the Surprising Behavior of Distance Metrics in High Dimensional Space. In Database Theory - ICDT 2001: 8th International Conference (LNCS). London, UK.
[2]
W.J. Conover and R. L. Iman. 1979. On Multiple-comparisons Procedures. Technical Report LA-7677-MS. Los Alamos Sci. Lab.
[3]
S. J. Daniels, A. A. M. Rahat, R. M. Everson, G. R. Tabor, and J. E. Fieldsend. 2018. A Suite of Computationally Expensive Shape Optimisation Problems Using Computational Fluid Dynamics. In International Conference on Parallel Problem Solving from Nature.
[4]
R. Dawkins. 1982. The Extended Phenotype. Oxford University Press Oxford.
[5]
K. De Grave, J. Ramon, and L. De Raedt. 2008. Active Learning for High Throughput Screening. In International Conference on Discovery Science.
[6]
S. Doncieux and J.-B. Mouret. 2010. Behavioral Diversity Measures for Evolutionary Robotics. In IEEE Congress on Evolutionary Computation.
[7]
A. Forrester, A. Sobester, and A. Keane. 2008. Engineering Design via Surrogate Modelling. John Wiley & Sons.
[8]
J. M. Gablonsky and C. T. Kelley. 2001. A Locally-Biased form of the DIRECT Algorithm. Journal of Global Optimization.
[9]
A. Gaier, A. Asteroth, and J.-B. Mouret. 2018. Data-efficient Neuroevolution with Kernel-Based Surrogate Models. In Proceedings of the Genetic and Evolutionary Computation Conference.
[10]
T. Hildebrandt and J. Branke. 2015. On Using Surrogates with Genetic Programming. Evolutionary Computation.
[11]
Y. Jin. 2011. Surrogate-assisted Evolutionary Computation: Recent Advances and Future Challenges. Swarm and Evolutionary Computation.
[12]
Y. Jin, H. Wang, T. Chugh, D. Guo, and K. Miettinen. 2018. Data-driven Evolutionary Optimization: An Overview and Case Studies. IEEE Transactions on Evolutionary Computation.
[13]
M. G. Kendall and J. D. Gibbons. 1990. Rank Correlation Methods. Oxford University Press, London.
[14]
J. R. Koza. 1994. Genetic programming. MIT Press.
[15]
W. H. Kruskal and W. A. Wallis. 1952. Use of Ranks in One-Criterion Variance Analysis. J. Amer. Statist. Assoc.
[16]
A. Moraglio, K. Krawiec, and C. G. Johnson. 2012. Geometric Semantic Genetic Programming. In International Conference on Parallel Problem Solving from Nature.
[17]
J.-B. Mouret. 2011. Encouraging Behavioral Diversity in Evolutionary Robotics: An Empirical Study. Evolutionary Computation.
[18]
J.-B. Mouret and J. Clune. 2015. Illuminating Search Spaces by Mapping Elites. arXiv:1504.04909v1.
[19]
Y. S. Ong, P. B. Nair, and A. J. Keane. 2003. Evolutionary Optimization of Computationally Expensive Problems via Surrogate Modeling. AIAA Journal.
[20]
T. Pohlert. 2018. PMCMRplus: Calculate Pairwise Multiple Comparisons of Mean Rank Sums Extended - R package, version 1.4.1.
[21]
R Core Team. 2018. R: A Language and Environment for Statistical Computing.
[22]
C. E. Rasmussen. 2004. Gaussian Processes in Machine Learning. In Advanced Lectures on Machine Learning. Springer.
[23]
J. Snoek, H. Larochelle, and R. P. Adams. 2012. Practical Bayesian Optimization of Machine Learning Algorithms. In Advances in Neural Information Processing Systems.
[24]
K. O. Stanley. 2006. Exploiting Regularity Without Development. In Proceedings of the AAAI Fall Symposium on Developmental Systems. AAAI Press.
[25]
K. O. Stanley and R. Miikkulainen. 2002. Evolving neural networks through augmenting topologies. Evolutionary Computation.
[26]
J. Stork, M. Zaefferer, and T. Bartz-Beielstein. 2019. Improving NeuroEvolution Efficiency by Surrogate Model-Based Optimization with Phenotypic Distance Kernels. In Applications of Evolutionary Computation.
[27]
J. Stork, M. Zaefferer, A. Fischbach, and T. Bartz-Beielstein. 2017. Surrogate-Assisted Learning of Neural Networks. In Proceedings 27. Workshop Computational Intelligence.
[28]
W. N. Venables and B. D. Ripley. 2002. Modern Applied Statistics with S. Springer.
[29]
H. Wang, Y. Jin, and J. O. Jansen. 2016. Data-driven Surrogate-assisted Multi-objective Evolutionary Optimization of a Trauma System. IEEE Transactions on Evolutionary Computation.
[30]
M. Zaefferer. 2019. Combinatorial Efficient Global Optimization in R - CEGO v2.3.0. https://cran.r-project.org/package=CEGO accessed: 2019-03-19.
[31]
M. Zaefferer, J. Stork, O. Flasch, and T. Bartz-Beielstein. 2018. Linear Combination of Distance Measures for Surrogate Models in Genetic Programming. In Parallel Problem Solving from Nature - PPSN XV. Coimbra, Portugal.

Cited By

View all
  • (2024)NeuroLGP-SM: Scalable Surrogate-Assisted Neuroevolution for Deep Neural Networks2024 IEEE Congress on Evolutionary Computation (CEC)10.1109/CEC60901.2024.10612039(1-8)Online publication date: 30-Jun-2024
  • (2024)Accelerated NAS via Pretrained Ensembles and Multi-fidelity Bayesian OptimizationArtificial Neural Networks and Machine Learning – ICANN 202410.1007/978-3-031-72332-2_17(245-260)Online publication date: 17-Sep-2024
  • (2023)Discovering and Exploiting Sparse Rewards in a Learned Behavior SpaceEvolutionary Computation10.1162/evco_a_00343(1-28)Online publication date: 6-Oct-2023
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
GECCO '19: Proceedings of the Genetic and Evolutionary Computation Conference Companion
July 2019
2161 pages
ISBN:9781450367486
DOI:10.1145/3319619
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 July 2019

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. distance metrics
  2. neural networks
  3. surrogate models

Qualifiers

  • Research-article

Conference

GECCO '19
Sponsor:
GECCO '19: Genetic and Evolutionary Computation Conference
July 13 - 17, 2019
Prague, Czech Republic

Acceptance Rates

Overall Acceptance Rate 1,669 of 4,410 submissions, 38%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)4
  • Downloads (Last 6 weeks)1
Reflects downloads up to 07 Jan 2025

Other Metrics

Citations

Cited By

View all
  • (2024)NeuroLGP-SM: Scalable Surrogate-Assisted Neuroevolution for Deep Neural Networks2024 IEEE Congress on Evolutionary Computation (CEC)10.1109/CEC60901.2024.10612039(1-8)Online publication date: 30-Jun-2024
  • (2024)Accelerated NAS via Pretrained Ensembles and Multi-fidelity Bayesian OptimizationArtificial Neural Networks and Machine Learning – ICANN 202410.1007/978-3-031-72332-2_17(245-260)Online publication date: 17-Sep-2024
  • (2023)Discovering and Exploiting Sparse Rewards in a Learned Behavior SpaceEvolutionary Computation10.1162/evco_a_00343(1-28)Online publication date: 6-Oct-2023
  • (2022)Augmenting Novelty Search with a Surrogate Model to Engineer Meta-diversity in Ensembles of ClassifiersApplications of Evolutionary Computation10.1007/978-3-031-02462-7_27(418-434)Online publication date: 15-Apr-2022
  • (2020)Understanding the Behavior of Reinforcement Learning AgentsBioinspired Optimization Methods and Their Applications10.1007/978-3-030-63710-1_12(148-160)Online publication date: 16-Nov-2020

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media