skip to main content
10.1145/1390630.1390640acmconferencesArticle/Chapter ViewAbstractPublication PagesisstaConference Proceedingsconference-collections
research-article

Effective and scalable software compatibility testing

Published: 20 July 2008 Publication History

Abstract

Today's software systems are typically composed of multiple components, each with different versions. Software compatibility testing is a quality assurance task aimed at ensuring that multi-component based systems build and/or execute correctly across all their versions' combinations, or configurations. Because there are complex and changing interdependencies between components and their versions, and because there are such a large number of configurations, it is generally infeasible to test all potential configurations. Consequently, in practice, compatibility testing examines only a handful of default or popular configurations to detect problems; as a result costly errors can and do escape to the field.
This paper presents a new approach to compatibility testing, called Rachet. We formally model the entire configuration space for software systems and use the model to generate test plans to sample a portion of the space. In this paper, we test all direct dependencies between components and execute the test plan efficiently in parallel. We present empirical results obtained by applying our approach to two large-scale scientific middleware systems. The results show that for these systems Rachet scaled well and discovered incompatibilities between components, and that testing only direct dependences did not compromise test quality.

References

[1]
E. Anderson, Z. Bai, J. Dongarra, A. Greenbaum, A. McKenney, J. D. Croz, S. Hammarling, J. Demmel, C. H. Bischof, and D. C. Sorensen. LAPACK: A portable linear algebra library for high--performance computers. In Proc. of Supercomputing '90, pages 2--11, Nov. 1990.
[2]
S. Balay, K. Buschelman, V. Eijkhout, W. D. Gropp, D. Kaushik, M. G. Knepley, L. C. McInnes, B. F. Smith, and H. Zhang. PETSc users manual. Technical Report ANL-95/11 -- Revision 2.3.2, Argonne National Laboratory, Sep. 2006.
[3]
M. B. Cohen, M. B. Dwyer, and J. Shi. Coverage and adequacy in software product line testing. In Proc. of ROSATEA '06, pages 53--63, 2006.
[4]
K. Czarnecki, S. Helsen, and U. Eisenecker. Formalizing cardinality-based feature models and their specialization. Software Process: Improvement and Practice, 10(1):7--29, Jan./Mar. 2005.
[5]
J. Dongarra, J. D. Croz, S. Hammarling, and I. S. Duff. A set of level 3 basic linear algebra subprograms. ACM Transactions on Mathematical Software, 16(1):1--17, Mar. 1990.
[6]
A. Duarte, G. Wagner, F. Brasileiro, and W. Cirne. Multi-environment software testing on the Grid. In Proc. of the 2006 Workshop on Parallel and Distributed Systems: Testing and Debugging, Jul. 2006.
[7]
W. E. Lewis. Software Testing and Continuous Quality Improvement. CRC Press LLC, Apr. 2000.
[8]
F. Mancinelli, J. Boender, R. di Cosmo, J. Vouillon, B. Durak, X. Leroy, and R. Treinen. Managing the complexity of large free and open source package-based software distributions. In Proc. of the 21st Int'l Conf. on Automated Software Engineering, pages 199--208, Sep. 2006.
[9]
A. Memon, A. Porter, C. Yilmaz, A. Nagarajan, D. C. Schmidt, and B. Natarajan. Skoll: Distributed continuous quality assurance. In Proc. of the 26th Int'l Conf. on Software Engineering, pages 459--468, May 2004.
[10]
PostgreSQL BuildFarm. http://www.pgbuildfarm.org.
[11]
R. Sabharwal. Grid infrastructure deployment using SmartFrog technology. In Proc. of the 2006 Int'l Conf. on Networking and Services, Jul. 2006.
[12]
A. Sussman. Building complex coupled physical simulations on the Grid with InterComm. Engineering with Computers, 22(3-4):311--323, Dec. 2006.
[13]
T. Syrjanen. A rule-based formal model for software configuration. Technical Report A55, Helsinki University of Technology, Dec. 1999.
[14]
C. Tucker, D. Shuffelton, R. Jhala, and S. Lerner. OPIUM: Optimal package install/uninstall manager. In Proc. of the 29th Int'l Conf. on Software Engineering, pages 178--188, May 2007.
[15]
G. V. Vaughan, B. Elliston, T. Tromey, and I. L. Taylor. GNU Autoconf, Automake, and Libtool. Sams, 1st edition, 2000.
[16]
VMware Inc. Streamlining software testing with IBM Rational and VMware: Test lab automation solution -- whitepaper, 2003.
[17]
I.-C. Yoon, A. Sussman, A. Memon, and A. Porter. Direct-dependency-based software compatibility testing. In Proc. of the 22st Int'l Conf. on Automated Software Engineering, Nov. 2007.

Cited By

View all
  • (2024)Distributed Recommendation Systems: Survey and Research DirectionsACM Transactions on Information Systems10.1145/369478343:1(1-38)Online publication date: 6-Sep-2024
  • (2023)An automated evaluation of broker compatibility for the Message Queuing Telemetry Transport protocolJournal of Software: Evolution and Process10.1002/smr.241035:7Online publication date: 2-Jul-2023
  • (2022)Overview of Information System Testing Technology Under the “CLOUD + MIcroservices” ModeComputer and Communication Engineering10.1007/978-3-031-17422-3_6(62-74)Online publication date: 24-Sep-2022
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
ISSTA '08: Proceedings of the 2008 international symposium on Software testing and analysis
July 2008
324 pages
ISBN:9781605580500
DOI:10.1145/1390630
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 20 July 2008

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. component-based software system
  2. software compatibility testing

Qualifiers

  • Research-article

Conference

ISSTA '08
Sponsor:

Acceptance Rates

Overall Acceptance Rate 58 of 213 submissions, 27%

Upcoming Conference

ISSTA '25

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)50
  • Downloads (Last 6 weeks)9
Reflects downloads up to 07 Jan 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Distributed Recommendation Systems: Survey and Research DirectionsACM Transactions on Information Systems10.1145/369478343:1(1-38)Online publication date: 6-Sep-2024
  • (2023)An automated evaluation of broker compatibility for the Message Queuing Telemetry Transport protocolJournal of Software: Evolution and Process10.1002/smr.241035:7Online publication date: 2-Jul-2023
  • (2022)Overview of Information System Testing Technology Under the “CLOUD + MIcroservices” ModeComputer and Communication Engineering10.1007/978-3-031-17422-3_6(62-74)Online publication date: 24-Sep-2022
  • (2021)Augmented Reality Applications Development using Android-Based Marker-less Method as Tourism Promotion Media: A Case StudyProceedings of the 2021 International Conference on Computer, Control, Informatics and Its Applications10.1145/3489088.3489118(200-204)Online publication date: 5-Oct-2021
  • (2021)Salp Swarm Optimizer for Modeling Software Reliability Prediction ProblemsNeural Processing Letters10.1007/s11063-021-10607-653:6(4451-4487)Online publication date: 1-Dec-2021
  • (2018)ICAT: An IoT Device Compatibility Testing Tool2018 25th Asia-Pacific Software Engineering Conference (APSEC)10.1109/APSEC.2018.00087(668-672)Online publication date: Dec-2018
  • (2018)Utilizing Faults and Time to Finish Estimating the Number of Software Test Workers Using Artificial Neural Networks and Genetic ProgrammingInformation Systems and Technologies to Support Learning10.1007/978-3-030-03577-8_67(613-624)Online publication date: 25-Oct-2018
  • (2016)Coordinated Collaborative Testing of Shared Software Components2016 IEEE International Conference on Software Testing, Verification and Validation (ICST)10.1109/ICST.2016.38(364-374)Online publication date: Apr-2016
  • (2014)Enabling collaborative testing across shared software componentsProceedings of the 17th international ACM Sigsoft symposium on Component-based software engineering10.1145/2602458.2602468(55-64)Online publication date: 27-Jun-2014
  • (2014)Easing software component repository evolutionProceedings of the 36th International Conference on Software Engineering10.1145/2568225.2568262(756-766)Online publication date: 31-May-2014
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media