ABSTRACT
Immediacy and continuity of feedback are desirable properties during programming. Automated tests are a widely used practice to gain feedback on whether test authors' expectations are consistent with an implementation. With growing test suites, feedback becomes less immediate and is obtained less frequently because of that. The objective of test prioritization is to choose an order of tests that catches errors as early as possible, ideally within a time frame that we can consider live.
Research in test prioritization often relies on dynamic analysis, which is expensive to obtain. Newer approaches focus on most recently edited source code locations and propose IR (information retrieval) approaches that regard a change to the software as query against a collection of tests.
We study the capability of the IR approach to reduce testing time in the presence of faults using the example of open-source Python projects, identify trade-offs in classical TF-IDF-based IR frameworks, and propose different approaches that consider lexical and semantic context of a change, including topic modeling.
We conclude that even simple IR strategies achieve immediate error detection, especially when tests themselves were edited alongside program code. We further discuss applications of this approach in live programming environments, where change granularity does not leave sufficient time to run a test suite entirely.
- David M. Blei, Andrew Y. Ng, and Michael I. Jordan. 2003. Latent Dirichlet Allocation. J. Mach. Learn. Res. 3 (March 2003), 993--1022. Google Scholar
- Qi Luo, Kevin Moran, and Denys Poshyvanyk. 2016. A Large-Scale Empirical Comparison of Static and Dynamic Test Case Prioritization Techniques. In Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE 2016). ACM, New York, NY, USA, 559--570. Google ScholarDigital Library
- Toni Mattis. 2018. Mining Concepts from Code Using Community Detection in CoOccurrence Graphs. In Conference Companion of the 2Nd International Conference on Art, Science, and Engineering of Programming (Programming'18 Companion). ACM, New York, NY, USA, 232--233. Google ScholarDigital Library
- Toni Mattis, Patrick Rein, Stefan Ramson, Jens Lincke, and Robert Hirschfeld. 2017. Towards Concept-Aware Programming Environments for Guiding Software Modularity. In Proceedings of the 3rd ACM SIGPLAN International Workshop on Programming Experience (PX/17.2). ACM, Vancouver, BC, Canada, 36--45. Google ScholarDigital Library
- David Rauch, Patrick Rein, Stefan Ramson, Jens Lincke, and Robert Hirschfeld. 2019. Babylonian-Style Programming. The Art, Science, and Engineering of Programming 3, 3 (Feb. 2019), 9:1--9:39.Google ScholarCross Ref
- Patrick Rein, Stefan Lehmann, Toni Mattis, and Robert Hirschfeld. 2016. How Live Are Live Programming Systems?: Benchmarking the Response Times of Live Programming Environments. In Proceedings of the Programming Experience 2016 (PX/16) Workshop (PX/16). ACM, New York, NY, USA, 1--8. Google ScholarDigital Library
- Stephen Robertson and Hugo Zaragoza. 2009. The Probabilistic Relevance Framework: BM25 and Beyond. Found. Trends Inf. Retr. 3, 4 (April 2009), 333--389. Google ScholarDigital Library
- S. E Robertson, S Walker, and M Beaulieu. 2000. Experimentation as a Way of Life: Okapi at TREC. Information Processing & Management 36, 1 (Jan. 2000), 95--108. Google ScholarDigital Library
- Amir Saeidi, Jurriaan Hage, Ravi Khadka, and Slinger Jansen. 2019. Applications of Multi-View Learning Approaches for Software Comprehension. The Art, Science, and Engineering of Programming 3, 3 (2019), (to appear).Google Scholar
- Ripon K. Saha, Matthew Lease, Sarfraz Khurshid, and Dewayne E. Perry. 2013. Improving Bug Localization Using Structured Information Retrieval. In Proceedings of the 28th IEEE/ACM International Conference on Automated Software Engineering (ASE'13). IEEE Press, Piscataway, NJ, USA, 345--355. Google ScholarDigital Library
- Ripon K. Saha, Lingming Zhang, Sarfraz Khurshid, and Dewayne E. Perry. 2015. An Information Retrieval Approach for Regression Test Prioritization Based on Program Changes. In Proceedings of the 37th International Conference on Software Engineering - Volume 1 (ICSE '15). IEEE Press, Piscataway, NJ, USA, 268--279. Google ScholarDigital Library
- David Schuler and Andreas Zeller. 2009. Javalanche: Efficient Mutation Testing for Java. In Proceedings of the the 7th Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on The Foundations of Software Engineering (ESEC/FSE '09). ACM, New York, NY, USA, 297--298. Google ScholarDigital Library
- Stephen W. Thomas, Hadi Hemmati, Ahmed E. Hassan, and Dorothea Blostein. 2014. Static Test Case Prioritization Using Topic Models. Empirical Software Engineering 19, 1 (Feb. 2014), 182--212. Google ScholarDigital Library
Index Terms
Faster feedback through lexical test prioritization
Recommendations
Static test case prioritization using topic models
Software development teams use test suites to test changes to their source code. In many situations, the test suites are so large that executing every test for every source code change is infeasible, due to time and resource constraints. Development ...
Faster mutation testing inspired by test prioritization and reduction
ISSTA 2013: Proceedings of the 2013 International Symposium on Software Testing and AnalysisMutation testing is a well-known but costly approach for determining test adequacy. The central idea behind the approach is to generate mutants, which are small syntactic transformations of the program under test, and then to measure for a given test ...
Test-Suite Reduction and Prioritization for Modified Condition/Decision Coverage
Software testing is particularly expensive for developers of high-assurance software, such as software that is produced for commercial airborne systems. One reason for this expense is the Federal Aviation Administration's requirement that test suites be ...
Comments