research-article

How is exploratory testing used? A state-of-the-practice survey

Authors:

Mika V. Mäntylä,

Jürgen MünchAuthors Info & Claims

ESEM '14: Proceedings of the 8th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement

Article No.: 5, Pages 1 - 10

https://doi.org/10.1145/2652524.2652531

Published: 18 September 2014 Publication History

Abstract

Context: Exploratory Testing has experienced a rise in popularity in the industry with the emergence of agile development practices, yet it remains unclear, in which domains and how it is used in practice.

Goal: To study how software engineers understand and apply the principles of exploratory testing, as well as the specific advantages and difficulties they experience.

Method: We conducted an online survey in the period June to August 2013 among Estonian and Finnish software developers and testers.

Results: Our main findings are that the majority of testers, developers, and test managers using ET, (1) apply ET to usability-critical, performance-critical, security-critical and safety-critical software to a high degree; (2) use ET very flexibly in all types of test levels, activities, and phases; (3) perceive ET as an approach that supports creativity during testing and that is effective and efficient; and (4) find that ET is not easy to use and has little tool support.

Conclusions: The high degree of application of ET in critical domains is particularly interesting and indicates a need for future research to obtain a better understanding of the effects of ET in these domains. In addition, our findings suggest that more support to ET users should be given (guidance and tools).

References

[1]

J. Bach. Session-Based Test Management. Software Testing and Quality Engineering Magazine, 2000.

[2]

J. Bach. Exploratory Testing. In: The Testing Software engineer, 2nd ed., E. van Veenendaal (Ed.) Den Bosch: UTN Publishers, pp. 253--265, 2004.

[3]

P. Bourque, R. E. Fairley (ed.). SWEBOK V3.0: Guide to the Software Engineering Body of Knowledge. IEEE Computer Society, 2014.

[4]

F. J. Fowler, Jr. Improving Survey Questions. Design and Evaluation. Sage, 1995.

[5]

F. J. Fowler, Jr. Survey Research Methods. Sage, third edition, 2002.

[6]

J. Itkonen, M. V. Mäntylä. Are test cases needed? Replicated comparison between exploratory and test-case-based software testing. Empirical Software Engineering, pp. 1--40, 2013. DOI 10.1007/s10664-013-9266-8 (published online)

Digital Library

[7]

J. Itkonen, M. V. Mäntylä, C. Lassenius. Defect Detection Efficiency: Test Case Based vs. Exploratory Testing, First International Symposium on Empirical Software Engineering and Measurement (ESEM 2007), pp. 61--70, 2007.

Digital Library

[8]

J. Itkonen, M. V. Mantyla, C. Lassenius. The role of the tester's knowledge in exploratory software testing. IEEE Transactions on Software Engineering, 39(5): 707--724, 2013.

Digital Library

[9]

J. Itkonen, K. Rautiainen. Exploratory testing: A multiple case study. Proceedings of the 4th International Symposium on Empirical Software Engineering (ISESE 2005). Noosa Heads, Queensland, Australia. IEEE, pp. 84--93, 2005.

[10]

C. Kaner, J. Falk, and H. Q. Nguyen. Testing Computer Software (Second Edition), Van Nostrand Reinhold, New York, 1993.

Digital Library

[11]

C. Kaner. A Tutorial in Exploratory Testing. Tutorial presented at QUEST2008. (Available online at: http://www.kaner.com/pdfs/QAIExploring.pdf, accessed: 26 Jan 2014)

[12]

E. MacGregor, Y. Hsieh, P. Kruchten. Cultural patterns in software process mishaps: incidents in global projects. ACM SIGSOFT Software Engineering Notes, 30(4):1--5, 2005

Digital Library

[13]

L. H. O. do Nascimento, P. D. L. Machado. An experimental evaluation of approaches to feature testing in the mobile phone applications domain. In: Workshop on Domain specific approaches to software test automation: in conjunction with the 6th ESEC/FSE joint meeting (DOSTA '07). ACM, New York, NY, USA, pp. 27--33, 2007.

Digital Library

[14]

S. L. Pfleeger, B. A. Kitchenham. Principles of Survey Research, Part 1: Turning Lemons into Lemonade. Software Engineering Notes, 26(6): 16--18, 2001.

Digital Library

[15]

P. Runeson. A survey of unit testing practices. IEEE Software, 23.4: 22--29, 2006.

Digital Library

[16]

S. Saukkoriipi, I. Tervonen. Team Exploratory Testing Sessions. ISRN Software Engineering, vol. 2012, Article ID 324838, 20 pages, 2012.

[17]

W. R. Shadish, T. D. Cook, and D. T. Campbell. Experimental and Quasi-Experimental Designs for Generalized Causal Inference. Houghton Mifflin, 2002.

[18]

S. M. A. Shah, C. Gencel, U. S. Alvi, K. Petersen. Towards a hybrid testing process unifying exploratory testing and scripted testing. Journal of Software: Evolution and Process, 26: 220--250, 2014.

Digital Library

[19]

J. Våga, S. Amland. Managing high-speed web testing. In: Software quality and software testing in internet times, D. Meyerhoff, B. Laibarra, R. van der Pouw Kraan, A. Wallet (Eds.). Springer-Verlag New York, Inc., New York, NY, USA, pp. 23--30, 2002.

Digital Library

[20]

J. A. Whittaker. Exploratory Software Testing: Tips, Tricks, Tours, and Techniques to Guide Test Design. Addison Wesley Professional, 2009.

Digital Library

[21]

H. Yin. Survey on Exploratory Testing. Master's Thesis at University of Tartu, 2014. (Available online at: http://comserv.cs.ut.ee/forms/ati_report/datasheet.php?id=39151&year=2014)

Cited By

Su YLiao DXing ZHuang QXie MLu QXu XRoychoudhury APaiva AAbreu RStorey M(2024)Enhancing Exploratory Testing by Large Language Model and Knowledge GraphProceedings of the IEEE/ACM 46th International Conference on Software Engineering10.1145/3597503.3639157(1-12)Online publication date: 20-May-2024
https://dl.acm.org/doi/10.1145/3597503.3639157
Faatz APandit K(2024)Exploratory Testing for Safety-Critical Large-Scale Embedded Systems Such as SpacecraftsFrom Multimedia Communications to the Future Internet10.1007/978-3-031-71874-8_15(219-232)Online publication date: 13-Sep-2024
https://doi.org/10.1007/978-3-031-71874-8_15
Costa IOliveira SElgrably IGuerra ASoares ECosta I(2023)Using Active Methodologies for Teaching and Learning of Exploratory Test Design and ExecutionEducation Sciences10.3390/educsci1302011513:2(115)Online publication date: 21-Jan-2023
https://doi.org/10.3390/educsci13020115
Show More Cited By

Index Terms

How is exploratory testing used? A state-of-the-practice survey
1. Social and professional topics
  1. Professional topics
    1. Management of computing and information systems
2. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging

Recommendations

What's going on? Age, distraction, and multitasking during online survey taking

Participants in an online survey revealed what other activities they engaged in while taking it.Younger people were more likely than older people to multitask.The relationship between age, multitasking, and sense of distraction was curvilinear.Most ...
The Role of the Tester's Knowledge in Exploratory Software Testing

We present a field study on how testers use knowledge while performing exploratory software testing (ET) in industrial settings. We video recorded 12 testing sessions in four industrial organizations, having our subjects think aloud while performing ...
Comparing a hybrid testing process with scripted and exploratory testing: an experimental study with practitioners
Agile Processes in Software Engineering and Extreme Programming

This study presents an experimental study comparing the testing quality of a Hybrid Testing (HT) process with the commonly used approaches in industry: Scripted Testing (ST) and Exploratory Testing (ET). The study was conducted in an international IT ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ESEM '14: Proceedings of the 8th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement

September 2014

461 pages

ISBN:9781450327749

DOI:10.1145/2652524

General Chair:
Maurizio Morisio
Politecnico di Torino, Italy
,
Program Chairs:
Tore Dybå
Sintef, Norway
,
Marco Torchiano
Politecnico di Torino, Italy

Copyright © 2014 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 18 September 2014

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Estonian Research Council

Conference

ESEM '14

Sponsor:

SIGSOFT

ESEM '14: 2014 ACM-IEEE International Symposium on Empirical Software Engineering and Measurement

September 18 - 19, 2014

Torino, Italy

Acceptance Rates

ESEM '14 Paper Acceptance Rate 23 of 123 submissions, 19%;

Overall Acceptance Rate 130 of 594 submissions, 22%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

39
Total Citations
View Citations
700
Total Downloads

Downloads (Last 12 months)75
Downloads (Last 6 weeks)5

Reflects downloads up to 03 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Su YLiao DXing ZHuang QXie MLu QXu XRoychoudhury APaiva AAbreu RStorey M(2024)Enhancing Exploratory Testing by Large Language Model and Knowledge GraphProceedings of the IEEE/ACM 46th International Conference on Software Engineering10.1145/3597503.3639157(1-12)Online publication date: 20-May-2024
https://dl.acm.org/doi/10.1145/3597503.3639157
Faatz APandit K(2024)Exploratory Testing for Safety-Critical Large-Scale Embedded Systems Such as SpacecraftsFrom Multimedia Communications to the Future Internet10.1007/978-3-031-71874-8_15(219-232)Online publication date: 13-Sep-2024
https://doi.org/10.1007/978-3-031-71874-8_15
Costa IOliveira SElgrably IGuerra ASoares ECosta I(2023)Using Active Methodologies for Teaching and Learning of Exploratory Test Design and ExecutionEducation Sciences10.3390/educsci1302011513:2(115)Online publication date: 21-Jan-2023
https://doi.org/10.3390/educsci13020115
Coutinho JAndrade WMachado P(2023)Implementing Exploratory Testing in an Agile Context: A Study Based on Design Science ResearchProceedings of the 8th Brazilian Symposium on Systematic and Automated Software Testing10.1145/3624032.3624042(67-76)Online publication date: 25-Sep-2023
https://dl.acm.org/doi/10.1145/3624032.3624042
Kirinuki HTajima MHaruto T(2023)LatteArt: A Platform for Recording and Analyzing Exploratory Testing2023 IEEE Conference on Software Testing, Verification and Validation (ICST)10.1109/ICST57152.2023.00048(443-453)Online publication date: Apr-2023
https://doi.org/10.1109/ICST57152.2023.00048
Su YHan ZXing ZXu XZhu LLu Q(2023)SoapOperaTG: A Tool for System Knowledge Graph Based Soap Opera Test Generation2023 IEEE/ACM 45th International Conference on Software Engineering: Companion Proceedings (ICSE-Companion)10.1109/ICSE-Companion58688.2023.00023(51-54)Online publication date: May-2023
https://doi.org/10.1109/ICSE-Companion58688.2023.00023
Doorn NVos TMarín B(2023)Towards understanding students’ sensemaking of test case designData & Knowledge Engineering10.1016/j.datak.2023.102199146:COnline publication date: 1-Jul-2023
https://dl.acm.org/doi/10.1016/j.datak.2023.102199
Roman AMnich MHryszko J(2023)Empirical Verification of TQED - A New Test Design Heuristic TechniqueTesting Software and Systems10.1007/978-3-031-43240-8_6(80-96)Online publication date: 19-Sep-2023
https://doi.org/10.1007/978-3-031-43240-8_6
Su YHan ZXing ZXia XXu XZhu LLu Q(2022)Constructing a System Knowledge Graph of User Tasks and Failures from Bug Reports to Support Soap Opera TestingProceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering10.1145/3551349.3556967(1-13)Online publication date: 10-Oct-2022
https://dl.acm.org/doi/10.1145/3551349.3556967
Ferreira Costa IRonaldo Bezerra Oliveira S(2022)Development of a teaching plan to support learning activities of exploratory test design and execution2022 IEEE Frontiers in Education Conference (FIE)10.1109/FIE56618.2022.9962486(1-9)Online publication date: 8-Oct-2022
https://doi.org/10.1109/FIE56618.2022.9962486
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten