research-article

Empirical analysis of a genetic algorithm-based stress test technique

Author:

Vahid GarousiAuthors Info & Claims

GECCO '08: Proceedings of the 10th annual conference on Genetic and evolutionary computation

Pages 1743 - 1750

https://doi.org/10.1145/1389095.1389433

Published: 12 July 2008 Publication History

Abstract

Evolutionary testing denotes the use of evolutionary algorithms, e.g., Genetic Algorithms (GAs), to support various test automation tasks. Since evolutionary algorithms are heuristics, their performance and output efficiency can vary across multiple runs. Therefore, there is a strong need to empirically investigate the capacity of evolutionary test techniques to achieve the desired objectives (e.g., generate stress test cases) and their scalability in terms of the complexity of the System Under Test (SUT), the inputs, and the control parameters of the search algorithms. In a previous work, we presented a GA-based UML-driven, stress test technique aimed at increasing chances of discovering faults related to network traffic in distributed real-time software. This paper reports a carefully-designed empirical study which was conducted to analyze and improve the applicability, efficiency and effectiveness of the above GA-based stress test technique. Detailed stages and objectives of the empirical analysis are reported. The findings of the study are furthermore used to better calibrate the parameters of the GA-based stress test technique.

References

[1]

J. J. P. Tsai, Y. Bi, S. J. H. Yang, and R. A. W. Smith, Distributed Real-Time Systems: Monitoring, Visualization, Debugging, and Analysis: John Wiley & Sons, 1996.

Digital Library

[2]

R. Kuhn, "Sources of Failure in the Public Switched Telephone Network," IEEE Computer, vol. 30, no. 4, pp. 31--36, 1997.

Digital Library

[3]

Object Management Group (OMG), "UML 2.1.1 Superstructure Specification," 2007.

[4]

V. Garousi, L. Briand, and Y. Labiche, "Traffic-aware Stress Testing of Distributed Real-Time Systems Based on UML Models using Genetic Algorithms," Elsevier Journal of Systems and Software, Special Issue on Model-Based Software Testing, vol. 81, no. 2, pp. 161--185, 2008.

Digital Library

[5]

V. Garousi, L. Briand, and Y. Labiche, "Traffic-aware Stress Testing of Distributed Systems based on UML Models," Proc. of Int. Conf. on Software Engineering, pp. 391--400, 2006.

Digital Library

[6]

M. Harman, "The current State and Future of Search-Based Software Engineering," Proc. of Int. Conf. on Software Engineering, Future of Software Engineering, pp. 342--357, 2007.

Digital Library

[7]

L. Briand, "A Critical Analysis of Empirical Research in Software Testing," Keynote address, Int. Symp. on Empirical Software Engineering and Measurement, 2007.

Digital Library

[8]

Y. Min, X. Jin, X. Su, and B. Peng, "Empirical Analysis of the Spatial Genetic Algorithm on Small-World Networks," Proc. of Int. Conf. on Computational Science, pp. 1032--1039, 2006.

Digital Library

[9]

G. Rudolph, "Convergence Analysis of Canonical Genetic Algorithms," IEEE Transactions on Neural Networks, vol. 5, no. 1, pp. 96--101, 1994.

Digital Library

[10]

K. Sastry, M. Pelikan, and D. E. Goldberg, "Empirical Analysis of Ideal Recombination on Random Decomposable Problems," Genetic and Evolutionary Computation Conference, pp. 1388--1395, 2007.

Digital Library

[11]

G. Antoniol, M. Di Penta, and M. Harman, "Search-Based Techniques Applied to Optimization of Project Planning for a Massive Maintenance Project," Proc. of Int. Conf. on Software Maintenance, pp. 240--249, 2005.

Digital Library

[12]

M. Harman, Y. Hassoun, K. Lakhotia, P. McMinn, and J. Wegener, "The Impact of Input Domain Reduction on Search-Based Test Data Generation," Proc. of Int. Symp. on the Foundations of Software Engineering, pp. 155--164, 2007

Digital Library

[13]

M. Harman and P. McMinn, "A Theoretical & Empirical Analysis of Evolutionary Testing and Hill Climbing for Structural Test Data Generation " Proc. of Int. Symp. on Software Testing and Analysis pp. 73--83, 2007.

Digital Library

[14]

M. Xiao, M. E. M. Reformat, and J. Miller, "Empirical Evaluation of Optimization Algorithms when used in Goal-Oriented Automated Test Data Generation Techniques," Empirical Software Engineering, vol. 12, no. 2, pp. 183--239, 2007

Digital Library

[15]

K. De Jong, "The Analysis of the Behavior of a Class of Genetic Adaptive Systems," Ph.D. Dissertation, Dept. of Computer Science, University of Michigan, Ann Arbor, 1975.

Digital Library

[16]

J. J. Greffenstette, "Optimization of Control Parameters for Genetic Algorithms," IEEE Trans. on Systems, Man, and Cybernetics, vol. 16, no. 1, pp. 122--128, 1986.

Digital Library

[17]

V. Garousi, "GARUS (Genetic Algorithm-based test Requirement tool for real-time distribUted Systems)," http://www.enel.ucalgary.ca/~vgarousi/tools/GARUS, 2006.

[18]

V. Garousi, "Traffic-aware Stress Testing of Distributed Real-Time Systems Based on UML Models using Genetic Algorithms," Ph.D. Thesis, Department of Systems and Computer Engineering, Carleton University, 2006.

Digital Library

[19]

M. Wall, "GAlib: A C++ Library of Genetic Algorithm Components," Documentation version 2.4, Massachusetts Institute of Technology 1996.

[20]

S. J. Louis and G. J. E. Rawlins, "Predicting Convergence Time for Genetic Algorithms," Technical Report 370, Computer Science Department, Indiana University 1993.

[21]

R. L. Haupt and S. E. Haupt, Practical Genetic Algorithms: Wiley-Interscience, 1998.

Digital Library

[22]

J. J. Grefenstette and H. G. Cobb, "Genetic Algorithms for Tracking Changing Environments," Proceeding of International Conference on Genetic Algorithms, pp. 523--530, 1993.

Digital Library

Cited By

Moghadam MSaadatmand MBorg MBohlin MLisper B(2021)An autonomous performance testing framework using self-adaptive fuzzy reinforcement learningSoftware Quality Journal10.1007/s11219-020-09532-z30:1(127-159)Online publication date: 10-Mar-2021
https://doi.org/10.1007/s11219-020-09532-z
Moghadam MDumas MPfahl DApel SRusso A(2019)Machine learning-assisted performance testingProceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3338906.3342484(1187-1189)Online publication date: 12-Aug-2019
https://dl.acm.org/doi/10.1145/3338906.3342484
Gois NPorfirio PCoelho A(2017)A Multi-objective Metaheuristic Approach to Search-Based Stress Testing2017 IEEE International Conference on Computer and Information Technology (CIT)10.1109/CIT.2017.19(55-62)Online publication date: Aug-2017
https://doi.org/10.1109/CIT.2017.19
Show More Cited By

Index Terms

Empirical analysis of a genetic algorithm-based stress test technique
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging

Recommendations

A Genetic Algorithm-Based Stress Test Requirements Generator Tool and Its Empirical Evaluation

Genetic algorithms (GAs) have been applied previously to UML-driven stress test requirements generation with the aim of increasing chances of discovering faults relating to network traffic in distributed real-time systems. However, since evolutionary ...
Test Case Prioritization Technique Based on Genetic Algorithm
ICICIS '11: Proceedings of the 2011 International Conference on Internet Computing and Information Services

With the rapid development of information technology, software testing, as a software quality assurance, is becoming more and more important. In the software life cycle, each time the code has changed need to be regression testing. The huge test case ...
A genetic algorithm calibration method based on convergence due to genetic drift

The selection of Genetic Algorithm (GA) parameters is a difficult problem, and if not addressed adequately, solutions of good quality are unlikely to be found. A number of approaches have been developed to assist in the calibration of GAs, however there ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

GECCO '08: Proceedings of the 10th annual conference on Genetic and evolutionary computation

July 2008

1814 pages

ISBN:9781605581309

DOI:10.1145/1389095

Conference Chair:
Conor Ryan
University of Limerick, Ireland
,
Editor:
Maarten Keijzer
Chordiant Software International B.V., The Netherlands

Copyright © 2008 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 12 July 2008

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

GECCO08

Sponsor:

GECCO08: Genetic and Evolutionary Computation Conference

July 12 - 16, 2008

GA, Atlanta, USA

Acceptance Rates

Overall Acceptance Rate 1,669 of 4,410 submissions, 38%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

14
Total Citations
View Citations
464
Total Downloads

Downloads (Last 12 months)5
Downloads (Last 6 weeks)1

Reflects downloads up to 08 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Moghadam MSaadatmand MBorg MBohlin MLisper B(2021)An autonomous performance testing framework using self-adaptive fuzzy reinforcement learningSoftware Quality Journal10.1007/s11219-020-09532-z30:1(127-159)Online publication date: 10-Mar-2021
https://doi.org/10.1007/s11219-020-09532-z
Moghadam MDumas MPfahl DApel SRusso A(2019)Machine learning-assisted performance testingProceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3338906.3342484(1187-1189)Online publication date: 12-Aug-2019
https://dl.acm.org/doi/10.1145/3338906.3342484
Gois NPorfirio PCoelho A(2017)A Multi-objective Metaheuristic Approach to Search-Based Stress Testing2017 IEEE International Conference on Computer and Information Technology (CIT)10.1109/CIT.2017.19(55-62)Online publication date: Aug-2017
https://doi.org/10.1109/CIT.2017.19
Bernardo Gois Fde Farias PCoelho ABarbosa T(2016)Improving stress search based testing using a hybrid metaheuristic approach2016 XLII Latin American Computing Conference (CLEI)10.1109/CLEI.2016.7833374(1-11)Online publication date: Oct-2016
https://doi.org/10.1109/CLEI.2016.7833374
Zhen Ming Jiang Hassan A(2015)A Survey on Load Testing of Large-Scale Software SystemsIEEE Transactions on Software Engineering10.1109/TSE.2015.244534041:11(1091-1118)Online publication date: 10-Nov-2015
https://dl.acm.org/doi/10.1109/TSE.2015.2445340
Rahman MMustayen AMekhilef SSaidur R(2015)The Optimization of Solar Drying of Grain by Using a Genetic AlgorithmInternational Journal of Green Energy10.1080/15435075.2014.89010612:12(1222-1231)Online publication date: 11-Jul-2015
https://doi.org/10.1080/15435075.2014.890106
Amannejad YGarousi VIrving RSahaf Z(2014)A Search-Based Approach for Cost-Effective Software Test Automation Decision Support and an Industrial Case StudyProceedings of the 2014 IEEE International Conference on Software Testing, Verification, and Validation Workshops10.1109/ICSTW.2014.34(302-311)Online publication date: 31-Mar-2014
https://dl.acm.org/doi/10.1109/ICSTW.2014.34
Harman MMansouri SZhang Y(2012)Search-based software engineeringACM Computing Surveys10.1145/2379776.237978745:1(1-61)Online publication date: 7-Dec-2012
https://dl.acm.org/doi/10.1145/2379776.2379787
Garvin BCohen MDwyer M(2011)Evaluating improvements to a meta-heuristic search for constrained interaction testingEmpirical Software Engineering10.1007/s10664-010-9135-716:1(61-102)Online publication date: 1-Feb-2011
https://dl.acm.org/doi/10.1007/s10664-010-9135-7
Zhang YAlba EDurillo JEldh SHarman MPelikan MBranke J(2010)Today/future importance analysisProceedings of the 12th annual conference on Genetic and evolutionary computation10.1145/1830483.1830733(1357-1364)Online publication date: 7-Jul-2010
https://dl.acm.org/doi/10.1145/1830483.1830733
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten