skip to main content
10.1145/2593501.2593506acmconferencesArticle/Chapter ViewAbstractPublication PagesicseConference Proceedingsconference-collections
Article

Selecting manual regression test cases automatically using trace link recovery and change coverage

Published: 31 May 2014 Publication History

Abstract

Regression tests ensure that existing functionality is not impaired by changes to an existing software system. However, executing complete test suites often takes much time. Therefore, a subset of tests has to be found that is sufficient to validate whether the system under test is still valid after it has been changed. This test case selection is especially important if regression tests are executed manually, since manual execution is time intensive and costly. To select manual test cases, many regression testing techniques exist that aim on achieving coverage of changed or even new code. Many of these techniques base on coverage data from prior test runs or logical properties such as annotated pre and post conditions in the source code. However, coverage information becomes outdated if a system is changed extensively. Also annotated logical properties are often not available in industrial software systems. We present an approach for test selection that is based on static analyses of the test suite and the system's source code. We combine trace link recovery using latent semantic indexing with the metric change coverage, which assesses the coverage of source code changes. The proposed approach works automatically without the need to execute tests beforehand or annotate source code. Furthermore, we present a first evaluation of the approach.

References

[1]
G. Antoniol, B. Caprile, A. Potrich, and P. Tonella. Design-code traceability recovery: selecting the basic linkage properties. Sci. Comput. Program., 40(2–3):213 – 234, 2001.
[2]
E. Ben Charrada, A. Koziolek, and M. Glinz. Identifying outdated requirements based on source code changes. In RE ’2012, 2012.
[3]
J. Bible, G. Rothermel, and D. S. Rosenblum. A comparative study of coarse- and fine-grained safe regression test-selection techniques. ACM Trans. Softw. Eng. Methodol., 10(2), 2001.
[4]
V. Channakeshava, V. K. Shanbhag, A. Panigrahi, R. Sisodia, and S. Lakshmanan. Safe subset-regression test selection for managed code. In ISEC ’08, 2008.
[5]
Y.-F. Chen, D. Rosenblum, and K.-P. Vo. Testtube: a system for selective regression testing. In ICSE ’94, 1994.
[6]
J. Cleland-Huang, R. Settimi, E. Romanova, B. Berenbach, and S. Clark. Best practices for automated traceability. Computer, 40(6):27–35, 2007.
[7]
S. Deerwester, S. T. Dumais, G. W. Furnas, T. K. Landauer, and R. Harshman. Indexing by latent semantic analysis. J. Am. Soc. Inf. Sci. Technol., 41(6):391–407, 1990.
[8]
S. Eder, B. Hauptmann, M. Junker, E. Juergens, R. Vaas, and K.-H. Prommer. Did we test our changes? assessing alignment between tests and development in practice. In AST ’2013, 2013.
[9]
S. Eder, M. Junker, E. Jurgens, B. Hauptmann, R. Vaas, and K. Prommer. How much does unused code matter for maintenance? In ICSE ’2012, 2012.
[10]
E. Engström, P. Runeson, and M. Skoglund. A systematic review on regression test selection techniques. Inf. Softw. Technol., 52(1):14–30, 2010.
[11]
D. Falessi, G. Cantone, and G. Canfora. Empirical principles and an industrial case study in retrieving equivalent requirements via natural language processing techniques. IEEE Trans. Softw. Eng., 39(1):18–44, 2013.
[12]
M. J. Harrold, J. A. Jones, T. Li, D. Liang, A. Orso, M. Pennings, S. Sinha, S. A. Spoon, and A. Gujarathi. Regression test selection for java software. SIGPLAN Not., 36(11), 2001.
[13]
J. H. Hayes, A. Dekhtyar, and S. K. Sundaram. Advancing candidate link generation for requirements tracing: The study of methods. IEEE Trans. Softw. Eng., 32(1):4–19, 2006.
[14]
D. Jiménez, E. Ferretti, V. Vidal, P. Rosso, and C. F. Enguix. The influence of semantics in ir using lsi and k-means clustering techniques. In ISICT ’03, 2003.
[15]
E. Juergens, B. Hummel, F. Deissenboeck, M. Feilkas, C. Schlögel, and A. Wübbeke. Regression test selection of manual system tests in practice. In CSMR ’11, 2011.
[16]
M. Lormans and A. van Deursen. Can lsi help reconstructing requirements traceability in design and test? In CSMR ’06, 2006.
[17]
A. D. Lucia, F. Fasano, R. Oliveto, and G. Tortora. Recovering traceability links in software artifact management systems using information retrieval methods. ACM Trans. Softw. Eng. Methodol., 16(4), 2007.
[18]
R. Oliveto, M. Gethers, D. Poshyvanyk, and A. De Lucia. On the equivalence of information retrieval methods for automated traceability link recovery. In ICPC ’2010, 2010.
[19]
M. F. Porter. An algorithm for suffix stripping. Program: Electronic Library & Information Systems, 40(3):211––218, 1980.
[20]
G. Rothermel, R. Untch, C. Chu, and M. Harrold. Prioritizing test cases for regression testing. Software Engineering, IEEE Transactions on, 27(10), 2001.
[21]
O. Traub, S. Schechter, and M. D. Smith. Ephemeral instrumentation for lightweight program profiling. Technical report, School of engineering and Applied Sciences, Harvard University, 2000.
[22]
D. Willmor and S. M. Embury. A safe regression test selection technique for database-driven applications. In ICSM ’05, 2005.
[23]
W. Wong, J. Horgan, S. London, and H. Agrawal. A study of effective regression testing in practice. In ISSRE ’97, 1997.
[24]
W. Zhao, L. Zhang, Y. Liu, J. Sun, and F. Yang. SNIAFL: Towards a static noninteractive approach to feature location. ACM Trans. Softw. Eng. Methodol., 15(2):195–226, 2006.

Cited By

View all
  • (2024)Optimization of Automated and Manual Software Tests in Industrial Practice: A Survey and Historical AnalysisIEEE Transactions on Software Engineering10.1109/TSE.2024.341819150:8(2005-2020)Online publication date: Aug-2024
  • (2022)Guided Bug Crush: Assist Manual GUI Testing of Android Apps via Hint MovesProceedings of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491102.3501903(1-14)Online publication date: 29-Apr-2022
  • (2021)How can manual testing processes be optimized? developer survey, optimization guidelines, and case studiesProceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3468264.3473922(1281-1291)Online publication date: 20-Aug-2021
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
AST 2014: Proceedings of the 9th International Workshop on Automation of Software Test
May 2014
63 pages
ISBN:9781450328586
DOI:10.1145/2593501
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

In-Cooperation

  • TCSE: IEEE Computer Society's Tech. Council on Software Engin.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 31 May 2014

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Regression tests
  2. manual system tests
  3. software maintenance
  4. test coverage
  5. test selection
  6. trace link recovery

Qualifiers

  • Article

Conference

ICSE '14
Sponsor:

Upcoming Conference

ICSE 2025

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)3
  • Downloads (Last 6 weeks)0
Reflects downloads up to 20 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Optimization of Automated and Manual Software Tests in Industrial Practice: A Survey and Historical AnalysisIEEE Transactions on Software Engineering10.1109/TSE.2024.341819150:8(2005-2020)Online publication date: Aug-2024
  • (2022)Guided Bug Crush: Assist Manual GUI Testing of Android Apps via Hint MovesProceedings of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491102.3501903(1-14)Online publication date: 29-Apr-2022
  • (2021)How can manual testing processes be optimized? developer survey, optimization guidelines, and case studiesProceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3468264.3473922(1281-1291)Online publication date: 20-Aug-2021
  • (2021)Traceability recovery between bug reports and test cases-a Mozilla Firefox case studyAutomated Software Engineering10.1007/s10515-021-00287-w28:2Online publication date: 7-Jul-2021
  • (2015)Configuring latent semantic indexing for requirements tracingProceedings of the Second International Workshop on Requirements Engineering and Testing10.5555/2820704.2820713(27-33)Online publication date: 16-May-2015
  • (2015)Configuring Latent Semantic Indexing for Requirements TracingProceedings of the 2015 IEEE/ACM 2nd International Workshop on Requirements Engineering and Testing10.1109/RET.2015.13(27-33)Online publication date: 18-May-2015
  • (2014)Which Features Do My Users (Not) Use?Proceedings of the 2014 IEEE International Conference on Software Maintenance and Evolution10.1109/ICSME.2014.71(446-450)Online publication date: 29-Sep-2014

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media