skip to main content
10.1145/3328433.3328455acmotherconferencesArticle/Chapter ViewAbstractPublication PagesprogrammingConference Proceedingsconference-collections
research-article

Faster feedback through lexical test prioritization

Published:01 April 2019Publication History

ABSTRACT

Immediacy and continuity of feedback are desirable properties during programming. Automated tests are a widely used practice to gain feedback on whether test authors' expectations are consistent with an implementation. With growing test suites, feedback becomes less immediate and is obtained less frequently because of that. The objective of test prioritization is to choose an order of tests that catches errors as early as possible, ideally within a time frame that we can consider live.

Research in test prioritization often relies on dynamic analysis, which is expensive to obtain. Newer approaches focus on most recently edited source code locations and propose IR (information retrieval) approaches that regard a change to the software as query against a collection of tests.

We study the capability of the IR approach to reduce testing time in the presence of faults using the example of open-source Python projects, identify trade-offs in classical TF-IDF-based IR frameworks, and propose different approaches that consider lexical and semantic context of a change, including topic modeling.

We conclude that even simple IR strategies achieve immediate error detection, especially when tests themselves were edited alongside program code. We further discuss applications of this approach in live programming environments, where change granularity does not leave sufficient time to run a test suite entirely.

References

  1. David M. Blei, Andrew Y. Ng, and Michael I. Jordan. 2003. Latent Dirichlet Allocation. J. Mach. Learn. Res. 3 (March 2003), 993--1022. Google ScholarGoogle Scholar
  2. Qi Luo, Kevin Moran, and Denys Poshyvanyk. 2016. A Large-Scale Empirical Comparison of Static and Dynamic Test Case Prioritization Techniques. In Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE 2016). ACM, New York, NY, USA, 559--570. Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. Toni Mattis. 2018. Mining Concepts from Code Using Community Detection in CoOccurrence Graphs. In Conference Companion of the 2Nd International Conference on Art, Science, and Engineering of Programming (Programming'18 Companion). ACM, New York, NY, USA, 232--233. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. Toni Mattis, Patrick Rein, Stefan Ramson, Jens Lincke, and Robert Hirschfeld. 2017. Towards Concept-Aware Programming Environments for Guiding Software Modularity. In Proceedings of the 3rd ACM SIGPLAN International Workshop on Programming Experience (PX/17.2). ACM, Vancouver, BC, Canada, 36--45. Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. David Rauch, Patrick Rein, Stefan Ramson, Jens Lincke, and Robert Hirschfeld. 2019. Babylonian-Style Programming. The Art, Science, and Engineering of Programming 3, 3 (Feb. 2019), 9:1--9:39.Google ScholarGoogle ScholarCross RefCross Ref
  6. Patrick Rein, Stefan Lehmann, Toni Mattis, and Robert Hirschfeld. 2016. How Live Are Live Programming Systems?: Benchmarking the Response Times of Live Programming Environments. In Proceedings of the Programming Experience 2016 (PX/16) Workshop (PX/16). ACM, New York, NY, USA, 1--8. Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. Stephen Robertson and Hugo Zaragoza. 2009. The Probabilistic Relevance Framework: BM25 and Beyond. Found. Trends Inf. Retr. 3, 4 (April 2009), 333--389. Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. S. E Robertson, S Walker, and M Beaulieu. 2000. Experimentation as a Way of Life: Okapi at TREC. Information Processing & Management 36, 1 (Jan. 2000), 95--108. Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. Amir Saeidi, Jurriaan Hage, Ravi Khadka, and Slinger Jansen. 2019. Applications of Multi-View Learning Approaches for Software Comprehension. The Art, Science, and Engineering of Programming 3, 3 (2019), (to appear).Google ScholarGoogle Scholar
  10. Ripon K. Saha, Matthew Lease, Sarfraz Khurshid, and Dewayne E. Perry. 2013. Improving Bug Localization Using Structured Information Retrieval. In Proceedings of the 28th IEEE/ACM International Conference on Automated Software Engineering (ASE'13). IEEE Press, Piscataway, NJ, USA, 345--355. Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. Ripon K. Saha, Lingming Zhang, Sarfraz Khurshid, and Dewayne E. Perry. 2015. An Information Retrieval Approach for Regression Test Prioritization Based on Program Changes. In Proceedings of the 37th International Conference on Software Engineering - Volume 1 (ICSE '15). IEEE Press, Piscataway, NJ, USA, 268--279. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. David Schuler and Andreas Zeller. 2009. Javalanche: Efficient Mutation Testing for Java. In Proceedings of the the 7th Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on The Foundations of Software Engineering (ESEC/FSE '09). ACM, New York, NY, USA, 297--298. Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. Stephen W. Thomas, Hadi Hemmati, Ahmed E. Hassan, and Dorothea Blostein. 2014. Static Test Case Prioritization Using Topic Models. Empirical Software Engineering 19, 1 (Feb. 2014), 182--212. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Faster feedback through lexical test prioritization

          Recommendations

          Comments

          Login options

          Check if you have access through your login credentials or your institution to get full access on this article.

          Sign in
          • Published in

            cover image ACM Other conferences
            Programming '19: Companion Proceedings of the 3rd International Conference on the Art, Science, and Engineering of Programming
            April 2019
            201 pages
            ISBN:9781450362573
            DOI:10.1145/3328433

            Copyright © 2019 ACM

            Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

            Publisher

            Association for Computing Machinery

            New York, NY, United States

            Publication History

            • Published: 1 April 2019

            Permissions

            Request permissions about this article.

            Request Permissions

            Check for updates

            Qualifiers

            • research-article

          PDF Format

          View or Download as a PDF file.

          PDF

          eReader

          View online with eReader.

          eReader