research-article

How does regression test prioritization perform in real-world software evolution?

Authors:

Lingming Zhang,

Lu ZhangAuthors Info & Claims

ICSE '16: Proceedings of the 38th International Conference on Software Engineering

Pages 535 - 546

https://doi.org/10.1145/2884781.2884874

Published: 14 May 2016 Publication History

Abstract

In recent years, researchers have intensively investigated various topics in test prioritization, which aims to re-order tests to increase the rate of fault detection during regression testing. While the main research focus in test prioritization is on proposing novel prioritization techniques and evaluating on more and larger subject systems, little effort has been put on investigating the threats to validity in existing work on test prioritization. One main threat to validity is that existing work mainly evaluates prioritization techniques based on simple artificial changes on the source code and tests. For example, the changes in the source code usually include only seeded program faults, whereas the test suite is usually not augmented at all. On the contrary, in real-world software development, software systems usually undergo various changes on the source code and test suite augmentation. Therefore, it is not clear whether the conclusions drawn by existing work in test prioritization from the artificial changes are still valid for real-world software evolution. In this paper, we present the first empirical study to investigate this important threat to validity in test prioritization. We reimplemented 24 variant techniques of both the traditional and time-aware test prioritization, and investigated the impacts of software evolution on those techniques based on the version history of 8 real-world Java programs from GitHub. The results show that for both traditional and time-aware test prioritization, test suite augmentation significantly hampers their effectiveness, whereas source code changes alone do not influence their effectiveness much.

References

[1]

C. Kaner, "Improving the maintainability of automated test suites," in Proc. QW, 1997.

[2]

P. K. Chittimalli and M. J. Harrold, "Recomputing coverage information to assist regression testing," IEEE TSE, vol. 35, no. 4, pp. 452--469, 2009.

Digital Library

[3]

G. Rothermel, R. H. Untch, C. Chu, and M. J. Harrold, "Test case prioritization: an empirical study," in ICSM, pp. 179--188, 1999.

Digital Library

[4]

W. E. Wong, J. R. Horgan, S. London, and H. Agrawal, "A study of effective regression testing in practice," in Proc. ISSRE, pp. 264--274, 1997.

Digital Library

[5]

M. J. Harrold, "Testing evolving software," JSS, vol. 47, no. 2--3, pp. 173--181, 1999.

Digital Library

[6]

G. Rothermel and M. Harrold, "A safe, efficient regression test selection technique," TOSEM, vol. 6, no. 2, pp. 173--210, 1997.

Digital Library

[7]

B. Jiang, Z. Zhang, W. K. Chan, and T. H. Tse, "Adaptive random test case prioritization," in ASE, pp. 257--266, 2009.

Digital Library

[8]

Z. Li, M. Harman, and R. Hierons, "Search algorithms for regression test case prioritisation," TSE, vol. 33, no. 4, pp. 225--237, 2007.

Digital Library

[9]

L. Zhang, D. Hao, L. Zhang, G. Rothermel, and H. Mei, "Bridging the gap between the total and additional test-case prioritization strategies," in ICSE, pp. 192--201, 2013.

Digital Library

[10]

D. Hao, L. Zhang, L. Zhang, G. Rothermel, and H. Mei, "A unified test case prioritization approach," TOSEM, vol. 10, pp. 1--31, 2014.

Digital Library

[11]

S. e Zehra Haidry and T. Miller, "Using dependency structures for prioritization of functional test suites," TSE, vol. 39, no. 2, pp. 258--275, 2013.

Digital Library

[12]

H. Mei, D. Hao, L. Zhang, L. Zhang, J. Zhou, and G. Rothermel, "A static approach to prioritizing junit test cases," TSE, vol. 38, no. 6, pp. 1258--1275, 2012.

Digital Library

[13]

C. Zhang, A. Groce, and M. A. Alipour, "Using test case reduction and prioritization to improve symbolic execution," in Proc. ISSTA, pp. 160--170, 2014.

Digital Library

[14]

L. Zhang, D. Marinov, and S. Khurshid, "Faster mutation testing inspired by test prioritization and reduction," in Proc. ISSTA, pp. 235--245, 2013.

Digital Library

[15]

H. Do, S. Mirarab, L. Tahvildari, and G. Rothermel, "The effects of time constraints on test case prioritization: A series of controlled experiments," TSE, vol. 36, no. 5, pp. 593--617, 2010.

Digital Library

[16]

L. Zhang, S.-S. Hou, C. Guo, T. Xie, and H. Mei, "Time-aware test-case prioritization using integer linear programming," in ISSTA, pp. 213--224, 2009.

Digital Library

[17]

M. Harman, S. A. Mansouri, and Y. Zhang, "Search-based software engineering: Trends, techniques and applications," CSUR, vol. 45, no. 1, p. 11, 2012.

Digital Library

[18]

M. Harman and B. F. Jones, "Search based software engineering," IST, vol. 43, no. 14, pp. 833--839, 2001.

[19]

K. R. Walcott, M. L. Soffa, G. M. Kapfhammer, and R. S. Roos, "Time aware test suite prioritization," in ISSTA, pp. 1--11, 2006.

Digital Library

[20]

S. Alspaugh, K. R. Walcott, M. Belanich, G. M. Kapfhammer, and M. L. Soffa, "Efficient time-aware prioritization with knapsack solvers," in Workshop on Empirical Assessment of Software Engineering Languages and Technologies, pp. 17--31, 2007.

Digital Library

[21]

L. Zhang, M. Kim, and S. Khurshid, "Localizing failure-inducing program edits based on spectrum information," in ICSM, pp. 23--32, 2011.

Digital Library

[22]

L. Zhang, M. Kim, and S. Khurshid, "Faulttracer: a change impact and regression fault analysis tool for evolving java programs," in FSE, p. 40, 2012.

[23]

A. Shi, A. Gyori, M. Gligoric, A. Zaytsev, and D. Marinov, "Balancing trade-offs in test-suite reduction," in Proc. FSE, pp. 246--256, 2014.

Digital Library

[24]

A. Shi, T. Yung, A. Gyori, and D. Marinov, "Comparing and combining test-suite reduction and regression test selection," in FSE, pp. 237--247, 2015.

Digital Library

[25]

A. Gyori, A. Shi, F. Hariri, and D. Marinov, "Reliable testing: Detecting state-polluting tests to prevent test dependency," in ISSTA, pp. 223--233, 2015.

Digital Library

[26]

J. H. Andrews, L. C. Briand, and Y. Labiche, "Is mutation an appropriate tool for testing experiments?," in ICSE, pp. 402--411, 2005.

Digital Library

[27]

H. Do and G. Rothermel, "A controlled experiment assessing test case prioritization techniques via mutation faults," in ICSM, pp. 411--420, 2005.

Digital Library

[28]

R. Just, D. Jalali, L. Inozemtseva, M. D. Ernst, R. Holmes, and G. Fraser, "Are mutants a valid substitute for real faults in software testing," in Proc. FSE, pp. 654--665, 2014.

Digital Library

[29]

H. Do and G. Rothermel, "On the use of mutation faults in empirical assessments of test case prioritization techniques," TSE, vol. 32, no. 9, pp. 733--752, 2006.

Digital Library

[30]

A. Arcuri and L. Briand, "A practical guide for using statistical tests to assess randomized algorithms in software engineering," in ICSE, pp. 1--10, 2011.

Digital Library

[31]

R. K. Saha, L. Zhang, S. Khurshid, and D. E. Perry, "An information retrieval approach for regression test prioritization based on program changes," 2015. To appear.

[32]

R. Just, D. Jalali, and M. D. Ernst, "Defects4j: A database of existing faults to enable controlled testing studies for java programs," in ISSTA, pp. 437--440, 2014.

Digital Library

[33]

T. H. Wonnacott and R. J. Wonnacott, Introductory statistics, vol. 19690. Wiley New York, 1972.

[34]

L. Zhang, S.-S. Hou, J.-J. Hu, T. Xie, and H. Mei, "Is operator-based mutant selection superior to random mutant selection?," in ICSE, pp. 435--444, 2010.

Digital Library

[35]

Y. Qi, X. Mao, Y. Lei, Z. Dai, and C. Wang, "The strength of random search on automated program repair," in ICSE, pp. 254--265, 2014.

Digital Library

[36]

F. Wilcoxon, "Individual comparisons by ranking methods," in Breakthroughs in Statistics, pp. 196--202, 1992.

[37]

S. Yoo and M. Harman, "Regression testing minimization, selection and prioritization: a survey," STVR, vol. 22, no. 2, pp. 67--120, 2012.

Digital Library

[38]

J. Black, E. Melachrinoudis, and D. Kaeli, "Bi-criteria models for all-uses test suite reduction," in ICSE, pp. 106--115, 2004.

Digital Library

[39]

G. Rothermel, M. J. Harrold, J. Von Ronne, and C. Hong, "Empirical studies of test-suite reduction," STVR, vol. 12, no. 4, pp. 219--249, 2002.

[40]

L. Zhang, D. Marinov, L. Zhang, and S. Khurshid, "An empirical study of junit test-suite reduction," in ISSRE, pp. 170--179, 2011.

Digital Library

[41]

T. Ball, "On the limit of control flow analysis for regression test selection," in ISSTA, pp. 134--142, 1998.

Digital Library

[42]

J. Chen, Y. Bai, D. Hao, Y. Xiong, H. Zhang, L. Zhang, and B. Xie, "A text-vector based approach to test case prioritization," in ICST, 2016. To appear.

[43]

Y. Lou, D. Hao, and L. Zhang, "Mutation-based test-case prioritization in software evolution," in ISSRE, pp. 46--57, 2015.

Digital Library

[44]

D. Hao, L. Zhang, L. Zang, Y. Wang, X. Wu, and T. Xie, "To be optimal or not in test-case prioritization," TSE, 2015. To appear.

[45]

S. Yoo, M. Harman, P. Tonella, and A. Susi, "Clustering test cases to achieve effective and scalable prioritisation incorporating expert knowledge," in Proc. ISSTA, pp. 201--212, 2009.

Digital Library

[46]

S. Elbaum, A. Malishevsky, and G. Rothermel, "Test case prioritization: A family of empirical studies," TSE, vol. 28, no. 2, pp. 159--182, 2002.

Digital Library

[47]

S. Elbaum, A. Malishevsky, and G. Rothermel, "Prioritizing test cases for regression testing," in ISSTA, pp. 102--112, 2000.

Digital Library

[48]

P. Tonella, P. Avesani, and A. Susi, "Using the case-based ranking methodology for test case prioritization," in Proc. ICSM, pp. 123--133, 2006.

Digital Library

[49]

C. D. Nguyen, A. Marchetto, and P. Tonella, "Test case prioritization for audit testing of evolving web services using information retrieval techniques," in Proc. ICWS, pp. 636--643, 2011.

Digital Library

[50]

R. K. Saha, L. Zhang, S. Khurshid, and D. E. Perry, "An information retrieval approach for regression test prioritization based on program changes," in Proc. ICSE, p. to appear, 2015.

Digital Library

[51]

H. Do, G. Rothermel, and A. Kinneer, "Empirical studies of test case prioritization in a JUnit testing environment," in ISSRE, pp. 113--124, 2004.

Digital Library

[52]

J. A. Jones and M. J. Harrold, "Test-suite reduction and prioritization for modified condition/decision coverage," in ICSM, pp. 92--101, 2001.

Digital Library

[53]

L. Zhang, J. Zhou, D. Hao, L. Zhang, and H. Mei, "Prioritizing JUnit test cases in absence of coverage information," in ICSM, pp. 19--28, 2009.

[54]

H. Mei, D. Hao, L. Zhang, L. Zhang, J. Zhou, and G. Rothermel, "A static approach to prioritizing junit test cases," TSE, vol. 38, no. 6, pp. 1258--1275, 2012.

Digital Library

[55]

L. Mei, W. K. Chan, and T. H. Tse, "Data flow testing of service-oriented workflow applications," in ICSE, pp. 371--380, 2008.

Digital Library

[56]

L. Mei, Z. Zhang, W. K. Chan, and T. H. Tse, "Test case prioritization for regression testing of service-oriented business applications," in WWW, pp. 901--910, 2009.

Digital Library

[57]

D. Xu and J. Ding, "Prioritizing state-based aspect tests," in Proc. ICST, pp. 265--274, 2010.

Digital Library

[58]

S. W. Thomas, H. Hemmati, A. E. Hassan, and D. Blostein, "Static test case prioritization using topic models," ESE, vol. 19, no. 1, pp. 182--212, 2014.

Digital Library

[59]

S. Elbaum, A. Malishevsky, and G. Rothermel, "Incorporating varying test costs and fault severities into test case prioritization," in ICSE, pp. 329--338, 2001.

Digital Library

[60]

H. Park, H. Ryu, and J. Baik, "Historical value-based approach for cost-cognizant test case prioritization to improve the effectiveness of regression testing," in SSIRI, pp. 39--46, 2008.

Digital Library

[61]

S.-S. Hou, L. Zhang, T. Xie, and J. Sun, "Quota-constrained test-case prioritization for regression testing of service-centric systems," in ICSM, pp. 257--266, 2008.

[62]

J. M. Kim and A. Porter, "A history-based test prioritization technique for regression testing in resource constrained environments," in ICSE, pp. 119--129, 2002.

Digital Library

[63]

H. Do, S. Mirarab, L. Tahvildari, and G. Rothermel, "An empirical study of the effect of time constraints on the cost-benefits of regression testing," in FSE, pp. 71--82, 2008.

Digital Library

[64]

A. Malishevsky, J. R. Ruthru, G. Rothermel, and S. Elbaum, "Cost-cognizant test case prioritization," tech. rep., Department Computer Science and Engineering of University of Nebraska, 2006.

[65]

H. Do and G. Rothermel, "Using sensitivity analysis to create simplified economic models for regression testing," in Proc. ISSTA, pp. 51--62, 2008.

Digital Library

[66]

H. Do and G. Rothermel, "An empirical study of regression testing techniques incorporating context and lifecycle factors and improved cost-benefit models," in Proc. FSE, pp. 141--151, 2006.

Digital Library

[67]

G. Rothermel, R. J. Untch, C. Chu, and M. J. Harrold, "Prioritizing test cases for regression testing," TSE, vol. 27, no. 10, pp. 929--948, 2001.

Digital Library

Cited By

Kıran SEmre İTaşdelen S(2024)Prioritization of Regression Test Cases Based on Machine Learning MethodsGazi University Journal of Science10.35378/gujs.1446469(1-1)Online publication date: 29-Dec-2024
https://doi.org/10.35378/gujs.1446469
Baz AHuang MShi AFilkov VRay BZhou M(2024)Prioritizing Tests for Improved RuntimeProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695298(2273-2278)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695298
Lou YYang JBenton SHao DTan LChen ZZhang LZhang L(2024)When Automated Program Repair Meets Regression Testing—An Extensive Study on Two Million PatchesACM Transactions on Software Engineering and Methodology10.1145/367245033:7(1-23)Online publication date: 13-Jun-2024
https://dl.acm.org/doi/10.1145/3672450
Show More Cited By

Recommendations

Regression test suite prioritization using system models

During regression testing, a modified system is often retested using an existing test suite. Since the size of the test suite may be very large, testers are interested in detecting faults in the modified system as early as possible during this retesting ...
State-based models in regression test suite prioritization

Testing software products is very expensive and time consuming, especially for large software systems with extensive regression testing. During regression testing, a modified system is often re-tested using an existing test suite. Since test suites can ...
Automated regression test suite prioritization using system models

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ICSE '16: Proceedings of the 38th International Conference on Software Engineering

May 2016

1235 pages

ISBN:9781450339001

DOI:10.1145/2884781

General Chair:
Laura Dillon
Michigan State University
,
Program Chairs:
Willem Visser
Stellenbosch University, South Africa
,
Laurie Williams
North Carolina State University

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

ACM: Association for Computing Machinery
SIGSOFT: ACM Special Interest Group on Software Engineering
IEEE-CS\TCSE: TC on Software Engineering
IEEE-CS\DATC: IEEE Computer Society

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 14 May 2016

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Research-article

Funding Sources

National Natural Science Foundation of China
National Basic Research Program of China (973)

Conference

ICSE '16

Sponsor:

ACM
SIGSOFT
IEEE-CS\TCSE
IEEE-CS\DATC

ICSE '16: 38th International Conference on Software Engineering

May 14 - 22, 2016

Texas, Austin

Acceptance Rates

Overall Acceptance Rate 276 of 1,856 submissions, 15%

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

80
Total Citations
View Citations
670
Total Downloads

Downloads (Last 12 months)25
Downloads (Last 6 weeks)1

Reflects downloads up to 02 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Kıran SEmre İTaşdelen S(2024)Prioritization of Regression Test Cases Based on Machine Learning MethodsGazi University Journal of Science10.35378/gujs.1446469(1-1)Online publication date: 29-Dec-2024
https://doi.org/10.35378/gujs.1446469
Baz AHuang MShi AFilkov VRay BZhou M(2024)Prioritizing Tests for Improved RuntimeProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695298(2273-2278)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695298
Lou YYang JBenton SHao DTan LChen ZZhang LZhang L(2024)When Automated Program Repair Meets Regression Testing—An Extensive Study on Two Million PatchesACM Transactions on Software Engineering and Methodology10.1145/367245033:7(1-23)Online publication date: 13-Jun-2024
https://dl.acm.org/doi/10.1145/3672450
Bouzenia IPradel MRoychoudhury APaiva AAbreu RStorey M(2024)Resource Usage and Optimization Opportunities in Workflows of GitHub ActionsProceedings of the IEEE/ACM 46th International Conference on Software Engineering10.1145/3597503.3623303(1-12)Online publication date: 20-May-2024
https://dl.acm.org/doi/10.1145/3597503.3623303
Lin HLi BLiu LWang LChang J(2024)OTCP-ISVM: Online Test Case Prioritization Based on Incremental Support Vector Machine2024 IEEE 24th International Conference on Software Quality, Reliability and Security (QRS)10.1109/QRS62785.2024.00018(84-94)Online publication date: 1-Jul-2024
https://doi.org/10.1109/QRS62785.2024.00018
Wang HYi PParladorio JLam WMarinov DXie T(2024)Hierarchy-Aware Regression Test Prioritization2024 IEEE 35th International Symposium on Software Reliability Engineering (ISSRE)10.1109/ISSRE62328.2024.00041(343-354)Online publication date: 28-Oct-2024
https://doi.org/10.1109/ISSRE62328.2024.00041
Baz AGligoric MShi A(2024)Impact of JVM Configurations on Test Runtime2024 IEEE International Conference on Software Maintenance and Evolution (ICSME)10.1109/ICSME58944.2024.00032(249-261)Online publication date: 6-Oct-2024
https://doi.org/10.1109/ICSME58944.2024.00032
Zhang ZChen JGu YLi ZSosu R(2024)Exploiting DBSCAN and Combination Strategy to Prioritize the Test Suite in Regression TestingIET Software10.1049/2024/99429592024(1-14)Online publication date: 4-Apr-2024
https://doi.org/10.1049/2024/9942959
Fallahzadeh ERigby PAdams B(2024)Contrasting test selection, prioritization, and batch testing at scaleEmpirical Software Engineering10.1007/s10664-024-10589-830:1Online publication date: 29-Nov-2024
https://dl.acm.org/doi/10.1007/s10664-024-10589-8
Feng QLiu SJi HMa XLiang P(2024)An empirical study of untangling patterns of two-class dependency cyclesEmpirical Software Engineering10.1007/s10664-023-10438-029:2Online publication date: 12-Mar-2024
https://doi.org/10.1007/s10664-023-10438-0
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten