A Systematic Review on Cloud Testing

Abstract

A systematic literature review is presented that surveyed the topic of cloud testing over the period 2012--2017. Cloud testing can refer either to testing cloud-based systems (testing of the cloud) or to leveraging the cloud for testing purposes (testing in the cloud): both approaches (and their combination into testing of the cloud in the cloud) have drawn research interest. An extensive paper search was conducted by both automated query of popular digital libraries and snowballing, which resulted in the final selection of 147 primary studies. Along the survey, a framework has been incrementally derived that classifies cloud testing research among six main areas and their topics. The article includes a detailed analysis of the selected primary studies to identify trends and gaps, as well as an extensive report of the state-of-the-art as it emerges by answering the identified Research Questions. We find that cloud testing is an active research field, although not all topics have received enough attention and conclude by presenting the most relevant open research challenges for each area of the classification framework.

References

1. Sameera Abar, Pierre Lemarinier, Georgios K. Theodoropoulos, and Gregory MP OHare. 2014. Automated dynamic resource provisioning and monitoring in virtualized large-scale datacenter. In Proceedings of the 28th IEEE International Conference on Advanced Information Networking and Applications (AINA’14). IEEE, 961--970.
2. Veronika Abramova, Jorge Bernardino, and Pedro Furtado. 2014. Testing cloud benchmark scalability with Cassandra. In Proceedings of the IEEE World Congress on Services (SERVICES’14). IEEE, 434--441.
3. Lorenzo Affetti, Giacomo Bresciani, and Sam Guinea. 2015. aDock: A cloud infrastructure experimentation environment based on Open Stack and Docker. In Proceedings of the 8th IEEE International Conference on Cloud Computing (CLOUD’15). IEEE, 203--210.
4. Amro Al-Said Ahmad, Pearl Brereton, and Peter Andras. 2017. A systematic mapping study of empirical studies on software cloud testing methods. In Proceedings of the IEEE International Conference on Software Quality, Reliability and Security. IEEE, 555--562.
5. Michel Albonico, Amine Benelallam, Jean-Marie Mottu, and Gerson Sunyé. 2016. A DSL-based approach for elasticity testing of cloud systems. In Proceedings of the International Workshop on Domain-Specific Modeling (DSM’16). 8--14.
6. Michel Albonico, Stefano Di Alesio, Jean-Marie Mottu, Sagar Sen, and Gerson Sunyé. 2017. Generating test sequences to assess the performance of elastic cloud-based systems. In Proceedings of the International Conference on Cloud Computing. IEEE, 383--390.
7. Michel Albonico, Jean-Marie Mottu, and Gerson Sunyé. 2016. Controlling the elasticity of web applications on cloud computing. In Proceedings of the 31st ACM Symposium on Applied Computing. ACM, 816--819.
8. Michel Albonico, Jean-Marie Mottu, and Gerson Sunyé. 2016. Monitoring-based testing of elastic cloud computing applications. In Proceedings of the International Conference on Performance Engineering. ACM, 3--6.
9. Michel Albonico, Jean-Marie Mottu, Gerson Sunyé, and Frederico Alvares. 2017. Making cloud-based systems elasticity testing reproducible. In Proceedings of the 7th International Conference on Cloud Computing and Services Science.
10. Nassima Aleb and Samir Kechid. 2012. Path coverage testing in the cloud. In Proceedings of the International Conference on Communications and Information Technology (ICCIT’12). IEEE, 118--123.
11. Amira Ali and Nagwa Badr. 2016. Performance testing as a service for web applications. In Proceedings of the 7th IEEE International Conference on Intelligent Computing and Information Systems (ICICIS’15). 356--361.
12. Kyoungho An, Takayuki Kuroda, Aniroddha Gokhale, Sumant Tambe, and Andrea Sorbini. 2014. Model-driven generative framework for automated OMG DDS performance testing in the cloud. ACM Sigplan Not. 49, 3 (2014), 179--182.
13. Marco Anisetti, Claudio Ardagna, Ernesto Damiani, and Filippo Gaudenzi. 2017. A semi-automatic and trustworthy scheme for continuous cloud service certification. IEEE Trans. Serv. Comput. (2017).
14. Michael Armbrust, Armando Fox, Rean Griffith, Anthony D. Joseph, Randy Katz, Andy Konwinski, Gunho Lee, David Patterson, Ariel Rabkin, Ion Stoica, and Matei Zaharia. 2010. A view of cloud computing. Commun. ACM 53, 4 (2010), 50--58.
15. Sunitha Badanahatti and Yelisetty Satya Sree Rama Murthy. 2016. Optimal test case prioritization in cloud based regression testing with aid of KFCM. International Journal of Intelligent Engineering and Systems 10, 2 (2016), 96--106.
16. Xiaoying Bai, Muyang Li, Bin Chen, Wei-Tek Tsai, and Jerry Gao. 2011. Cloud testing tools. In Proceedings of the 6th International Symposium on Service Oriented System Engineering. IEEE, 1--12.
17. Xiaoying Bai, Muyang Li, Xiaofei Huang, Wei-Tek Tsai, and Jerry Gao. 2013. Vee@Cloud: The virtual test lab on the cloud. In Proceedings of the 8th International Workshop on Automation of Software Test (AST’13). 15--18.
18. Miguel Angel Barcelona, Laura García-Borgoñón, and Gonzalo López-Nicolás. 2017. Practical experiences in the usage of MIDAS in the logistics domain. Int. J. Softw. Tools Technol. Trans. 19, 3 (2017), 325--339.
19. Feras A. Batarseh, Avelino J. Gonzalez, and Rainer Knauf. 2013. Context-assisted test cases reduction for cloud validation. In Proceedings of the International and Interdisciplinary Conference on Modeling and Using Context. Springer, 288--301.
20. Jonathan Becedas. 2014. The geo-cloud experiment: Global earth observation system computed in cloud. In Proceedings of the 8th International Conference on Innovative Mobile and Internet Services in Ubiquitous Computing (IMIS’14). IEEE, 9--15.
21. Elhadj Benkhelifa and Thomas Welsh. 2013. Security testing in the cloud by means of ethical worm. In Proceedings of the IEEE Globecom Workshops (GC Wkshps’13). 500--505.
22. Antonia Bertolino. 2007. Software testing research: Achievements, challenges, dreams. In Proceedings of the International Workshop on the Future of Software Engineering, (FOSE @ ICSE’07). IEEE, 85--103.
23. Antonia Bertolino, Antonello Calabró, Guglielmo De Angelis, Micael Gallego, Boni García, and Francisco Gortázar. 2018. When the testing gets tough, the tough get ElasTest. In Proceedings of the 40th International Conference on Software Engineering (ICSE’18). ACM, New York, NY, 17--20.
24. Pearl Brereton, Barbara A. Kitchenham, David Budgen, Mark Turner, and Mohamed Khalil. 2007. Lessons from applying the systematic literature review process within the software engineering domain. J. Syst. Softw. 80, 4 (2007), 571--583.
25. Achim D. Brucker, Lukas Brügger, and Burkhart Wolff. 2015. Formal firewall conformance testing: An application of test and proof techniques. Softw. Test. Verif. Reliab. 25, 1 (2015), 34--71.
26. Mark Buenen and Govind Muthukrishnan. 2017. World Quality Report: 2017--2018. Technical Report. Sogeti -- Part of Capgemini. https://www.sogeti.com/explore/reports/world-quality-report-2017-2018/.
27. Inderveer Chana and Priyanka Chawla. 2013. Testing perspectives for cloud-based applications. In Software Engineering Frameworks for the Cloud Computing Paradigm. Springer, 145--164.
28. Muhammad Aufeef Chauhan, Muhammad Ali Babar, and Christian W. Probst. 2016. A process framework for designing software reference architectures for providing tools as a service. In Proceedings of the International Conference on Product-Focused Software Process Improvement. Springer, 111--126.
29. Priyanka Chawla, Inderveer Chana, and Ajay Rana. 2016. Cloud-based automatic test data generation framework. J. Comput. System Sci. 82, 5 (2016), 712--738.
30. Dazhao Cheng, Changjun Jiang, and Xiaobo Zhou. 2014. Heterogeneity-aware workload placement and migration in distributed sustainable datacenters. In Proceedings of the Parallel and Distributed Processing Symposium. IEEE, 307--316.
31. Lin Cheng, Jialiang Chang, Zijiang Yang, and Chao Wang. 2016. GUICat: GUI testing as a service. In Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering. ACM, 858--863.
32. Chuan-Yen Chiang, Che-Pin Chang, Hung-Yu Chen, Yen-Lin Chen, Shyan-Ming Yuan, and Che Wang. 2016. ATP: A browser-based distributed testing service platform. In Proceedings of the International Computer Symposium (ICS’16). IEEE, 192--197.
33. Domenico Cotroneo, Flavio Frattini, Roberto Pietrantuono, and Stefano Russo. 2015. State-based robustness testing of IaaS cloud platforms. In Proceedings of the 5th International Workshop on Cloud Data and Platforms. ACM, 3.
34. Matheus Cunha, Nabor Mendonca, and Americo Sampaio. 2013. A declarative environment for automatic performance evaluation in IaaS clouds. In Proceedings of the International Conference on Cloud Computing. IEEE, 285--292.
35. Matheus Cunha, N. C. Mendonça, and Américo Sampaio. 2017. Cloud Crawler: A declarative performance evaluation environment for infrastructure-as-a-service clouds. Concur. Comput.: Pract. Exper. 29, 1 (2017).
36. Wenbin William Dai, Laurynas Riliskis, Valeriy Vyatkin, Evgeny Osipov, and Jerker Delsing. 2014. A configurable cloud-based testing infrastructure for interoperable distributed automation systems. In Proceedings of the Industrial Electronics Society Conference (IECON’14). IEEE, 2492--2498.
37. Alberto De Francesco, Claudia Di Napoli, Maurizio Giordano, Giuseppe Ottaviano, Raffaele Perego, and Nicola Tonellotto. 2014. A SOA testing platform on the cloud: The MIDAS experience. In Proceedings of the International Conference on Intelligent Networking and Collaborative Systems (INCoS’14). IEEE, 659--664.
38. Gustavo Sávio De Oliveira and Alexandre Duarte. 2014. A framework for automated software testing on the cloud. In Proceedings of the Conference on Parallel and Distributed Computing, Applications and Technologies (PDCAT’14). 344--349.
39. Ricardo Ramos de Oliveira, Rafael Messias Martins, and Adenilso da Silva Simao. 2017. Impact of the vendor lock-in problem on testing as a service (TaaS). In Proceedings of the International Conference on Cloud Engineering (IC2E’17). IEEE, 190--196.
40. Sergio Di Martino, Filomena Ferrucci, Valerio Maggio, and Federica Sarro. 2012. Towards migrating genetic algorithms for test data generation to the cloud. In Software Testing in the Cloud: Perspectives on an Emerging Discipline. IGI Global, 113--135.
41. Carlos H. G. Ferreira, Luiz H. Nunes, Lourenço A. Pereira, Luis H. V. Nakamura, Julio C. Estrella, and Stephan Reiff-Marganiec. 2016. PEESOS-Cloud: A workload-aware architecture for performance evaluation in service-oriented systems. In Proceedings of the IEEE World Congress on Services (SERVICES’16). IEEE, 118--125.
42. Enno Folkerts, Alexander Alexandrov, Kai Sachs, Alexandru Iosup, Volker Markl, and Cafer Tosun. 2012. Benchmarking in the cloud: What it should, can, and cannot be. In Proceedings of the Technology Conference on Performance Evaluation and Benchmarking. Springer, 173--188.
43. Geoffrey C. Fox, Supun Kamburugamuve, and Ryan D. Hartman. 2012. Architecture and measured characteristics of a cloud based internet of things. In Proceedings of the International Conference on Collaboration Technologies and Systems (CTS’12). IEEE, 6--12.
44. Alessio Gambi, Alessandra Gorla, and Andreas Zeller. 2017. O! Snap: Cost-efficient testing in the cloud. In Proceedings of the IEEE International Conference on Software Testing, Verification and Validation (ICST’17). IEEE, 454--459.
45. Alessio Gambi, Waldemar Hummer, and Schahram Dustdar. 2013. Automated testing of cloud-based elastic systems with AUToCLES. In Proceedings of the International Conference on Automated Software Engineering (ASE’13). 714--717.
46. Alessio Gambi, Waldemar Hummer, Hong-Linh Truong, and Schahram Dustdar. 2013. Testing elastic computing systems. IEEE Internet Comput. 17, 6 (2013), 76--82.
47. Jerry Gao, Xiaoying Bai, and Wei-Tek Tsai. 2011. Cloud testing-issues, challenges, needs and practice. Softw. Eng.: Int. J. 1, 1 (2011), 9--23.
48. Jerry Gao, Xiaoying Bai, Wei-Tek Tsai, and Tadahiro Uehara. 2013. Testing as a service (TaaS) on clouds. In Proceedings of the 7th IEEE International Symposium on Service Oriented System Engineering (SOSE’13). IEEE, 212--223.
49. Jerry Gao, Xiaoying Bai, Wei-Tek Tsai, and Tadahiro Uehara. 2014. Mobile application testing: A tutorial. Computer 47, 2 (2014), 46--55.
50. Jerry Gao, K. Manjula, P. Roopa, E. Sumalatha, Xiaoying Bai, Wei-Tek Tsai, and Tadahiro Uehara. 2012. A cloud-based TaaS infrastructure with tools for SaaS validation, performance and scalability evaluation. In Proceedings of the 4th IEEE International Conference on Cloud Computing Technology and Science (CloudCom’12). IEEE, 464--471.
51. Jerry Gao, Wei-Tek Tsai, Ray Paul, Xiaoying Bai, and Tadahiro Uehara. 2014. Mobile testing-as-a-service (MTaaS)—Infrastructures, issues, solutions and needs. In Proceedings of the International Symposium on High-Assurance Systems Engineering. 158--167.
52. Qiang Gao, Wei Wang, Guoquan Wu, Xuan Li, Jun Wei, and Hua Zhong. 2013. Migrating load testing to the cloud: A case study. In Proceedings of the 7th IEEE 7th International Symposium on Service Oriented System Engineering (SOSE’13). IEEE, 429--434.
53. Vahid Garousi and Michael Felderer. 2017. Worlds apart: Industrial and academic focus areas in software testing. IEEE Softw. 34, 5 (2017), 38--45.
54. Alim Ul Gias, Asif Imran, Rayhanur Rahman, and Kazi Sakib. 2013. IVRIDIO: Design of a software testing framework to provide Test-first Performance as a service. In Proceedings of the International Conference on Innovative Computing Technology, (INTECH’13). 520--525.
55. Marcelo Gonçalves, Matheus Cunha, Nabor C. Mendonça, and Américo Sampaio. 2015. Performance inference: A novel approach for planning the capacity of IaaS cloud applications. In Proceedings of the 8th IEEE International Conference on Cloud Computing (CLOUD’15). IEEE, 813--820.
56. Pablo Graubner, Lars Baumgärtner, Patrick Heckmann, Marcel Müller, and Bernd Freisleben. 2015. Dynalize: Dynamic analysis of mobile apps in a platform-as-a-service cloud. In Proceedings of the International Conference on Cloud Computing. IEEE, 925--932.
57. Meimei He, Guoquan Wu, Hongyin Tang, Wei Chen, Jun Wei, Hua Zhong, and Tao Huang. 2016. X-check: A novel cross-browser testing service based on record/replay. In Proceedings of the International Conference on Web Services (ICWS’16). IEEE, 123--130.
58. Steffen Herbold, Alberto De Francesco, Jens Grabowski, Patrick Harms, Lom M. Hillah, Fabrice Kordon, Ariele-Paolo Maesano, Libero Maesano, Claudia Di Napoli, Fabio De Rosa et al. 2015. The MIDAS cloud platform for testing SOA applications. In Proceedings of the International Conference on Software Testing, Verification and Validation (ICST’15). IEEE, 1--8.
59. Steffen Herbold, Patrick Harms, and Jens Grabowski. 2017. Combining usage-based and model-based testing for service-oriented architectures in the industrial practice. Int. J. Softw. Tools Technol. Trans. 19, 3 (2017), 309--324.
60. Lom Messan Hillah, Ariele-Paolo Maesano, Fabio De Rosa, Fabrice Kordon, Pierre-Henri Wuillemin, Riccardo Fontanelli, Sergio Di Bona, Davide Guerri, and Libero Maesano. 2017. Automation and intelligent scheduling of distributed system functional testing. Int. J. Softw. Tools Technol. Trans. 19, 3 (2017), 281--308.
61. Jhen-Jia Hu. 2014. The verification and validation of a large-scale system: Equipment TaaS as an example. In Proceedings of the International Symposium on Computer, Consumer and Control (IS3C’14). IEEE, 13--18.
62. Yazhou Hu, Bo Deng, Yu Yang, and Dongxia Wang. 2016. Elasticity evaluation of IaaS cloud based on mixed workloads. In Proceedings of the 15th International Symposium on Parallel and Distributed Computing (ISPDC’16). IEEE, 157--164.
63. Jun-fei Huang. 2014. AppACTS: Mobile app automated compatibility testing service. In Proceedings of the 2nd IEEE International Conference on Mobile Cloud Computing, Services, and Engineering (MobileCloud’14). IEEE, 85--90.
64. Jun-fei Huang and Yun-zhan Gong. 2012. Remote mobile test system: A mobile phone cloud for application testing. In Proceedings of the 4th IEEE International Conference on Cloud Computing Technology and Science (CloudCom’12). IEEE, 1--4.
65. Qiang Huang, Lin Ye, Xinran Liu, and Xiaojiang Du. 2013. Auditing CPU performance in public cloud. In Proceedings of the 9th IEEE World Congress on Services (SERVICES’13). IEEE, 286--289.
66. Vincent Shi-Ming Huang, Robert Huang, and Ming Chiang. 2013. A DDoS mitigation system with multi-stage detection and text-based turing testing in cloud computing. In Proceedings of the 27th International Conference on Advanced Information Networking and Applications Workshops (WAINA’13). IEEE, 655--662.
67. Alastair C. Hume, Yahya Al-Hazmi, Bartosz Belter, Konrad Campowsky, Luis M. Carril, Gino Carrozzo, Vegard Engen, David García-Pérez, Jordi Jofre Ponsatí, Roland Kűbert et al. 2012. BonFIRE: A multi-cloud test facility for internet of services experimentation. In Proceedings of the International Conference on Testbeds and Research Infrastructures. Springer, 81--96.
68. Hind Husni and Ahmad A. Saifan. 2017. Cloud testing: Steps, tools, challenges. In Proceedings of the New Trends in Information Technology (NTIT'17). The University of Jordan, Amman, Jordan, Paper 16.
69. Gwan-Hwan Hwang, Chi Wu-Lee, Yuan-Hsin Tung, Chih-Ju Chuang, and Syz-Feng Wu. 2014. Implementing TaaS-based stress testing by MapReduce computing model. In Proceedings of the IEEE International Conference on Software Engineering and Service Sciences (ICSESS’14). 137--140.
70. Kai Hwang, Xiaoying Bai, Yue Shi, Muyang Li, Wen-Guang Chen, and Yongwei Wu. 2016. Cloud performance modeling with benchmark evaluation of elastic scaling strategies. IEEE Trans. Parallel Distrib. Syst. 27, 1 (2016), 130--143.
71. Koray Inçki, Ismail Ari, and Hasan Sözer. 2012. A survey of software testing in the cloud. In Proceedings of the 6th International Conference on Software Security and Reliability (SERE’12). IEEE, 18--23.
72. Ganesh Neelakanta Iyer, Jayakhanna Pasimuthu, and Ramesh Loganathan. 2013. PCTF: An integrated, extensible cloud test framework for testing cloud platforms and applications. In Proceedings of the 13th International Conference on Quality Software (QSIC’13). IEEE, 135--138.
73. Pooyan Jamshidi, Aakash Ahmad, and Claus Pahl. 2013. Cloud migration research: A systematic review. IEEE Trans. Cloud Comput. 1, 2 (2013), 142--157.
74. Vajjiram Janani and K. Krishnamoorthy. 2015. Evaluation of cloud based performance testing for online shopping websites. Indian J. Sci. Technol. 8, 35 (2015).
75. Deepal Jayasinghe, Galen Swint, Simon Malkowski, Jack Li, Qingyang Wang, Junhee Park, and Calton Pu. 2012. Expertus: A generator approach to automate performance testing in IaaS clouds. In Proceedings of the 5th IEEE International Conference on Cloud Computing (CLOUD’12). IEEE, 115--122.
76. Changjiang Jia, Yan Cai, Yuen Tak Yu, and T. H. Tse. 2016. 5W+1H pattern: A perspective of systematic mapping studies and a case study on cloud software testing. J. Syst. Softw. 116 (2016), 206--219.
77. Changjiang Jia, Chunbai Yang, and Wing Kwong Chan. 2015. Architecturing dynamic data race detection as a Cloud-based Service. In Proceedings of the IEEE International Conference on Web Services (ICWS’15). IEEE, 345--352.
78. A. Vanitha Katherine and K. Alagarsamy. 2012. Software testing in cloud platform: A survey. Int. J. Comput. Appl. 46, 6 (May 2012), 21--25.
79. Manveen Kaur. 2016. Testing in the cloud: New challenges. In Proceedings of the International Conference on Computing, Communication and Automation (ICCCA’16). IEEE, 742--746.
80. Konstantinos Kavoussanakis, Alastair C. Hume, Josep Martrat, Carmelo Ragusa, Michael Gienger, Konrad Campowsky, Gregory van Seghbroeck, Constantino Vázquez, Celia Velayos, Frederic Gittler, Philip Inglesant, Giuseppe Carella, Vegard Engen, Michal Giertych, Giada Landi, and David Margery. 2013. BonFIRE: The clouds and services testbed. In Proceedings of the International Conference on Cloud Computing Technology and Science, CloudCom, Vol. 2. 321--326.
81. Mariam Kiran, Andreas Friesen, Anthony J. H. Simons, and Wolfgang K. R. Schwach. 2014. Model-based Testing in Cloud Brokerage Scenarios. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), Vol. 8377 LNCS. 192--208.
82. Mariam Kiran and Anthony J. H. Simons. 2014. Model-based testing for composite web services in cloud brokerage scenarios. In Proceedings of the European Conference on Service-Oriented and Cloud Computing. Springer, 190--205.
83. Barbara Kitchenham. 2004. Procedures for Performing Systematic Reviews. Technical Report TR/SE-0401, Department of Computer Science, Keele University. Keele, UK, Keele University 33, 2004 (2004), 1--26.
84. Chorng-Shiuh Koong, Chih-Hsiong Shih, Chang-Chung Wu, and Pao-Ann Hsiung. 2013. The architecture of parallelized cloud-based automatic testing system. In Proceedings of the 7th International Conference on Complex, Intelligent, and Software Intensive Systems (CISIS’13). IEEE, 467--470.
85. Nikolai Kosmatov, Nicky Williams, Bernard Botella, and Muriel Roger. 2013. Structural unit testing as a service with PathCrawler-online.com. In Proceedings of the 7th IEEE International Symposium on Service Oriented System Engineering (SOSE’13). IEEE, 435--440.
86. Jong Yih Kuo, Chien-Hung Liu, and Wei Ting Yu. 2015. The study of cloud-based testing platform for Android. In Proceedings of the 3rd IEEE International Conference on Mobile Services (MS’15). 197--201.
87. Etiene Lamas, Luis A. V. Dias, and Adilson Marques Da Cunha. 2012. Software architectural drivers for cloud testing. In Proceedings of the 4th International Conference on Advances in System Testing and Validation Lifecycle (VALID’12). 114--120.
88. Junwon Lee, Jaeik Cho, Jungtaek Seo, Taeshik Shon, and Dongho Won. 2013. A novel approach to analyzing for detecting malicious network activity using a cloud computing testbed. Mobile Netw. Appl. 18, 1 (2013), 122--128.
89. Raluca Lefticaru and Anthony J. H. Simons. 2014. X-Machine based testing for cloud services. In Proceedings of the European Conference on Service-Oriented and Cloud Computing. Springer, 175--189.
90. Chien-Hung Liu and Shu-Ling Chen. 2016. Evaluation of cloud testing strategies based on task decomposition and allocation for improving test efficiency. In Proceedings of the IEEE International Conference on Applied System Innovation (ICASI’16).
91. Chien-Hung Liu, Shu-Ling Chen, and Woei-Kae Chen. 2015. Improving resource utilization of a cloud-based testing platform for Android applications. In Proceedings of the IEEE International Conference on Mobile Services (MS’15). IEEE, 202--208.
92. Chien-Hung Liu, Shu-Ling Chen, and Woie-Kae Chen. 2017. Cost-benefit evaluation on parallel execution for improving test efficiency over cloud. In Proceedings of the International Conference on Applied System Innovation (ICASI’17). IEEE, 199--202.
93. Chien-Hung Liu, Woie-Kae Chen, and Shu-Ling Chen. 2016. A concurrent approach for improving the efficiency of Android CTS testing. In Proceedings of the IEEE International Computer Symposium (ICS’16). IEEE, 611--615.
94. Huai Liu, Maria Spichkova, Heinz W. Schmidt, Andreas Ulrich, Horst Sauer, and Jan Wieghardt. 2015. Efficient testing based on logical architecture. In Proceedings of the 24th Australasian Software Engineering Conference (ASWEC’15). ACM, 49--53.
95. Qi Liu, Marcio A. Silva, Michael R. Hines, and Dilma Da Silva. 2012. Hardware-in-the-loop simulation for automated benchmarking of cloud infrastructures. In Proceedings of the IEEE Winter Simulation Conference (WSC’12). IEEE, 1--12.
96. Win-Tsung Lo, Xiao-Long Liu, Ruey-Kai Sheu, Shyan-Ming Yuan, and Chun-Yu Chang. 2015. An architecture for cloud service testing and real time management. In Proceedings of the 39th IEEE Computer Software and Applications Conference (COMPSAC’15), Vol. 3. IEEE, 598--603.
97. Komal Mahajan and Deepak Dahiya. 2014. A cloud based deployment framework for load balancing policies. In Proceedings of the 7th International Conference on Contemporary Computing (IC3’14). IEEE, 565--570.
98. Riyadh Mahmood, Naeem Esfahani, Thabet Kacem, Nariman Mirzaei, Sam Malek, and Angelos Stavrou. 2012. A whitebox approach for automated security testing of Android applications on the cloud. In Proceedings of the 7th International Workshop on Automation of Software Test (AST’12). IEEE, 22--28.
99. Riyadh Mahmood, Nariman Mirzaei, and Sam Malek. 2014. Evodroid: Segmented evolutionary testing of Android apps. In Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering. ACM, 599--609.
100. A. Malini, N. Venkatesh, K. Sundarakantham, and S. Mercyshalinie. 2014. Mobile application testing on smart devices using MTAAS framework in cloud. In Proceedings of the International Conference on Computing and Communication Technologies (ICCCT’14).
101. Joe Mambretti, Jim Chen, and Fei Yeh. 2015. Next generation clouds, the chameleon cloud testbed, and software defined networking (SDN). In Proceedings of the International Conference on Cloud Computing Research and Innovation (ICCCRI’15). IEEE, 73--79.
102. Krishna Markande and Sridhar J. Murthy. 2013. Leveraging potential of cloud for software performance testing. In Cloud Computing. Springer, 293--322.
103. Nicolas Michael, Nitin Ramannavar, Yixiao Shen, Sheetal Patil, and Jan-Lung Sung. 2017. Cloudperf: A performance test framework for distributed and dynamic multi-tenant environments. In Proceedings of the 2017 ACM/SPEC International Conference on Performance Engineering (ICPE’17). 189--200.
104. Chandru Mirchandani. 2014. Cloud computing as a debug tool. In Procedia Comput. Sci., Vol. 36. 359--366.
105. Eman M. Mohamed, Sherif El-Etriby, and Hatem S. Abdul-kader. 2012. Randomness testing of modern encryption techniques in cloud environment. In Proceedings of the 8th IEEE International Conference on Informatics and Systems (INFOS’12). IEEE, CC--1.
106. Bashir Mohammed and Mariam Kiran. 2015. Analysis of cloud test beds using opensource solutions. In Proceedings of the 3rd IEEE International Conference on Future Internet of Things and Cloud (FiCloud’15). IEEE, 195--203.
107. Shraddha Mungekar and Dhanashree Toradmalle. 2015. W TaaS: An architecture of website analysis in a cloud environment. In Proceedings of the 1st IEEE International Conference on Next Generation Computing Technologies (NGCT’15). IEEE, 21--24.
108. Subramanian Nachiyappan and Selwyn Justus. 2015. Cloud testing tools and its challenges: A comparative study. Procedia Comput. Sci. 50 (2015), 482--489.
109. Crescencio Rodrigues Neto and Vinícius Cardoso Garcia. 2013. Cloud testing framework. In Proceedings of the 17th International Conference on Evaluation and Assessment in Software Engineering. ACM, 252--255.
110. Tomáš Oberle and Csaba Szabó. 2015. An architectural prototype for testware as a service. In Proceedings of the 13th IEEE International Symposium on Applied Machine Intelligence and Informatics (SAMI’15). IEEE, 15--19.
111. George Pallis. 2010. Cloud computing: The new frontier of internet computing. IEEE Internet Comput. 14, 5 (Sept. 2010), 70--73.
112. Dessislava Petrova-Antonova, Sylvia Ilieva, and Denitsa Manova. 2016. Automated web service composition testing as a service. In Proceedings of the International Conference on Model-Driven Engineering and Software Development. Springer, 114--131.
113. Dessislava Petrova-Antonova, Sylvia Ilieva, and Denitsa Manova. 2016. TASSA: A testing as a service framework for web service compositions. In Proceedings of the International Workshop on DomAin Specific Model-Based AppRoaches to vErificaTion and validaTiOn (AMARETTO’16). 33--42.
114. C. Mano Prathibhan, A. Malini, N. Venkatesh, and K. Sundarakantham. 2014. An automated testing framework for testing Android mobile applications in the cloud. In Proceedings of the International Conference on Advanced Communication Control and Computing Technologies (ICACCCT’14). IEEE, 1216--1219.
115. Priyanka, Inderveer Chana, and Ajay Rana. 2012. Empirical evaluation of cloud-based testing techniques: A systematic review. SIGSOFT Softw. Eng. Notes 37, Article 3 (May 2012), 9 pages.
116. Mazedur Rahman, Zehua Chen, and Jerry Gao. 2015. A service framework for parallel test execution on a developer’s local development workstation. In Proceedings of the 9th IEEE International Symposium on Service-Oriented System Engineering (SOSE’15), Vol. 30. 153--160.
117. Kaliappa Ravindran, Arun Adiththan, and Michael Iannelli. 2014. SLA evaluation with on-the-fly measurements of distributed service implementation over clouds. In Proceedings of the 6th International Workshop on Principles of Engineering Service-Oriented and Cloud Systems. ACM, 1--10.
118. Kaliappa Ravindran and Michael Iannelli. 2014. SLA evaluation in cloud-based data-centric distributed services. In Proceedings of the 23rd International Conference on Computer Communication and Networks (ICCCN’14). IEEE, 1--8.
119. Leah Riungu-Kalliosaari, Ossi Taipale, Kari Smolander, and Ita Richardson. 2016. Adoption and use of cloud-based testing in practice. Softw. Qual. J. 24, 2 (2016), 337--364.
120. Leah Riungu-Kalliosaari, Ossi Taipale, Kari Smolander, and Ita Richardson. 2016. Adoption and use of cloud-based testing in practice. Softw. Qual. J. 24, 2 (2016), 337--364.
121. Isabel K. Villanes Rojas, Silvia Meireles, and Arilo Claudio Dias-Neto. 2016. Cloud-based mobile app testing framework: Architecture, implementation and execution. In Proceedings of the 1st Brazilian Symposium on Systematic and Automated Software Testing. ACM, 10.
122. Georgia Sakellari and George Loukas. 2013. A survey of mathematical models, simulation approaches and testbeds used for research in cloud computing. Simul. Model. Pract. Theor. 39 (2013), 92--103.
123. Clint Seales, Trung Do, Eugene Belyi, and Suman Kumar. 2015. PHINet: A plug-n-play content-centric testbed framework for health-internet of things. In Proceedings of the IEEE International Conference on Mobile Services (MS’15). IEEE, 368--375.
124. Itai Segall and Rachel Tzoref-Brill. 2015. Feedback-driven combinatorial test design and execution. In Proceedings of the 8th ACM International Systems and Storage Conference. ACM, 12.
125. Akash Shrivastva, Shubham Gupta, and Rinki Tiwari. 2014. Cloud based testing techniques (CTT). Int. J. Comput. Appl. 104, 5 (2014).
126. Tamanna Siddiqui and Riaz Ahmad. 2015. Cloud testing—A systematic review. Int. Res. J. Eng. Technol. 2, 3 (2015), 397--406.
127. Marcio Silva, Michael R. Hines, Diego Gallo, Qi Liu, Kyung Dong Ryu, and Dilma Da Silva. 2013. Cloudbench: Experiment automation for cloud environments. In Proceedings of the IEEE International Conference on Cloud Engineering (IC2E’13). IEEE, 302--311.
128. Stelios Sotiriadis, Andrus Lehmets, Euripides G. M. Petrakis, and Nik Bessis. 2017. Unit and integration testing of modular cloud services. In Proceedings of the 31st IEEE International Conference on Advanced Information Networking and Applications (AINA’17). IEEE, 1116--1123.
129. Oleksii Starov and Sergiy Vilkomir. 2013. Integrated TaaS platform for mobile development: Architecture solutions. In Proceedings of the 8th International Workshop on Automation of Software Test (AST’13). 1--7.
130. Dylan Steinmetz, Brian W. Perrault, Ross Nordeen, Jacob Wilson, and Xinli Wang. 2012. Cloud computing performance benchmarking and virtual machine launch time. In Proceedings of the 13th Conference on Information Technology Education. ACM, 89--90.
131. Muhammad Dhiauddin Mohamed Suffian, Fairul Rizal Fahrurazi, and Suhaimi Ibrahim. 2014. The design and execution of performance testing strategy for cloud-based system. Int. J. Softw. Eng. Technol. 1, 2 (2014).
132. Chuanqi Tao and Jerry Gao. 2016. Cloud-based mobile testing as a service. Int. J. Softw. Eng. Knowl. Eng. 26, 1 (2016), 147--152.
133. Chuanqi Tao and Jerry Gao. 2017. On building a cloud-based mobile testing infrastructure service system. J. Syst. Softw. 124 (2017), 39--55.
134. Chuanqi Tao, Jerry Gao, and Bixin Li. 2016. Cloud-based infrastructure for mobile testing as a service. In Proceedings of the 3rd International Conference on Advanced Cloud and Big Data (CBD’15). 133--140.
135. Dan Tao, Zhaowen Lin, and Cheng Lu. 2015. Cloud platform based automated security testing system for mobile internet. Tsinghua Sci. Technol. 20, 6 (2015), 537--544.
136. Adrien Thiery, Thomas Cerqueus, Christina Thorpe, Gerson Sunyé, and John Murphy. 2014. A DSL for deployment and testing in the cloud. In Proceedings of the 7th IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW’14). IEEE, 376--382.
137. Hong-Linh Truong and Luca Berardinelli. 2017. Testing uncertainty of cyber-physical systems in IoT cloud infrastructures: Combining model-driven engineering and elastic execution. In Proceedings of the 1st ACM SIGSOFT International Workshop on Testing Embedded and Cyber-Physical Systems. ACM, 5--8.
138. Wei-Tek Tsai, Qingyang Li, Charles J. Colbourn, and Xiaoying Bai. 2013. Adaptive fault detection for testing tenant applications in multi-tenancy SaaS systems. In Proceedings of the IEEE International Conference on Cloud Engineering (IC2E’13). IEEE, 183--192.
139. Wei-Tek Tsai, Jie Luo, Guanqiu Qi, and Wenjun Wu. 2014. Concurrent test algebra execution with combinatorial testing. In Proceedings of the 8th IEEE International Symposium on Service Oriented System Engineering (SOSE’14). IEEE, 35--46.
140. Wei-Tek Tsai and Guanqiu Qi. 2015. Integrated adaptive reasoning testing framework with automated fault detection. In Proceedings of the IEEE Symposium on Service-Oriented System Engineering (SOSE’15). IEEE, 169--178.
141. Wei-Tek Tsai and Guanqiu Qi. 2016. Integrated fault detection and test algebra for combinatorial testing in TaaS (Testing-as-a-Service). Simul. Model. Pract. Theor. 68 (2016), 108--124.
142. Wei-Tek Tsai and Guanqiu Qi. 2017. Adaptive fault detection in multi-tenancy SaaS systems. In Combinatorial Testing in Cloud Computing. Springer, 25--36.
143. Wei-Tek Tsai and Guanqiu Qi. 2017. Adaptive reasoning algorithm with automated test cases generation and test algebra in SaaS system. In Combinatorial Testing in Cloud Computing. Springer, 83--99.
144. Wei-Tek Tsai and Guanqiu Qi. 2017. Integrated TaaS with fault detection and test algebra. In Combinatorial Testing in Cloud Computing. Springer, 115--128.
145. Wei-Tek Tsai, Guanqiu Qi, and Kai Hu. 2015. Autonomous decentralized combinatorial testing. In Proceedings of the 12th IEEE International Symposium on Autonomous Decentralized Systems (ISADS’15). IEEE, 40--47.
146. Wei Tek Tsai, Guanqiu Qi, Lian Yu, and Jerry Gao. 2014. TaaS (testing-as-a-service) design for combinatorial testing. In Proceedings of the 8th International Conference on Software Security and Reliability (SERE’14). IEEE, 127--136.
147. Yuan-Hsin Tung, Chen-Chiu Lin, and Hwai-Ling Shan. 2014. Test as a service: A framework for web security TaaS service in cloud environment. In Proceedings of the 8th IEEE International Symposium on Service Oriented System Engineering (SOSE’14). 212--217.
148. Yuan-Hsin Tung, Shian-Shyong Tseng, and Yung-Yu Kuo. 2015. A testing-based approach to SLA evaluation on cloud environment. In Proceedings of the 17th Asia-Pacific Network Operations and Management Symposium (APNOMS’15). IEEE, 495--498.
149. Andrew Turner, Andrew Fox, John Payne, and Hyong S. Kim. 2013. C-mart: Benchmarking the cloud. IEEE Trans. Parallel Distrib. Syst. 24, 6 (2013), 1256--1266.
150. Rob van der Meulen and Christy Pettey. 2017. Press release: Gartner forecasts worldwide public cloud services revenue to reach $260 billion in 2017. Retrieved from: https://www.gartner.com/newsroom/id/3815165.
151. Martti Vasar, Satish Narayana Srirama, and Marlon Dumas. 2012. Framework for monitoring and testing web application scalability on the cloud. In Proceedings of the WICSA/ECSA 2012 Companion Volume. ACM, 53--60.
152. Sergiy Vilkomir. 2012. Cloud testing: A state-of-the-art review. Inform. Sec. 28, 2 (2012), 213--222.
153. Isabel Karina Villanes, Erick Alexandre Bezerra Costa, and Arilo Claudio Dias-Neto. 2015. Automated mobile testing as a service (AM-TaaS). In Proceedings of the IEEE World Congress on Services (SERVICES’15). IEEE, 79--86.
154. Junyi Wang, Xiaoying Bai, Linyi Li, Zhicheng Ji, and Haoran Ma. 2017. A model-based framework for cloud API testing. In Proceedings of the 41st IEEE Computer Software and Applications Conference (COMPSAC’17), Vol. 2. IEEE, 60--65.
155. Claes Wohlin. 2014. Guidelines for snowballing in systematic literature studies and a replication in software engineering. In Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering. ACM, 38.
156. Wenjun Wu, Wei-Tek Tsai, Chao Jin, Guanqiu Qi, and Jie Luo. 2014. Test-algebra execution in a cloud environment. In Proceedings of the 8th IEEE International Symposium on Service Oriented System Engineering (SOSE’14). IEEE, 59--69.
157. Miguel G. Xavier, Kassiano J. Matteussi, Gabriel R. França, Wagner P. Pereira, and Cesar A. F. De Rose. 2017. Mobile application testing on clouds: Challenges, opportunities and architectural elements. In Proceedings of the 25th Euromicro International Conference on Parallel, Distributed and Network-based Processing (PDP’17). IEEE, 181--185.
158. Xiaolin Xu, Hai Jin, Song Wu, Lixiang Tang, and Yihong Wang. 2014. URMG: Enhanced CBMG-based method for automatically testing web applications in the cloud. TSINGHUA Sci. Technol. 19, 1 (2014), 65--75.
159. Minzhi Yan, Hailong Sun, Xu Wang, and Xudong Liu. 2012. Building a TaaS platform for web service load testing. In Proceedings of the IEEE International Conference on Cluster Computing (CLUSTER’12). IEEE, 576--579.
160. Minzhi Yan, Hailong Sun, Xu Wang, and Xudong Liu. 2012. WS-TaaS: A testing as a service platform for web service load testing. In Proceedings of the 18th IEEE International Conference on Parallel and Distributed Systems (ICPADS’12). IEEE, 456--463.
161. Wei Yan and Nirwan Ansari. 2012. Anti-virus in-the-cloud service: Are we ready for the security evolution? Sec. Commun. Netw. 5, 6 (2012), 572--582.
162. Lei Yin, Jin Zeng, Fangwang Liu, and Bo Li. 2013. CTPV: A cloud testing platform based on virtualization. In Proceedings of the 7th IEEE International Symposium on Service Oriented System Engineering (SOSE’13). IEEE, 425--428.
163. Philipp Zech, Felderer Michael, and Ruth Breu. 2012. Towards a model-based security testing approach of cloud computing environments. In Proceedings of the 6th IEEE International Conference on Software Security and Reliability Companion (SERE-C’12). 47--56.
164. Samer Zein, Norsaremah Salleh, and John Grundy. 2016. A systematic mapping study of mobile application testing techniques. J. Syst. Softw. 117 (2016), 334--356.
165. Linghao Zhang, Xiaoxing Ma, Jian Lu, Tao Xie, Nikolai Tillmann, and Peli De Halleux. 2012. Environmental modeling for automated cloud application testing. IEEE Softw. 29, 2 (2012), 30--35.
166. Shenbin Zhang and Bingfeng Pi. 2015. Mobile functional test on TaaS environment. In Proceedings of the IEEE Symposium on Service-Oriented System Engineering (SOSE’15). IEEE, 315--320.
167. Yuchao Zhang, Bin Hong, Ming Zhang, Bo Deng, and Wangqun Lin. 2013. eCAD: Cloud anomalies detection from an evolutionary view. In Proceedings of the International Conference on Cloud Computing and Big Data (CloudCom-Asia’13). IEEE, 328--334.
168. Yang Zheng, Lizhi Cai, Shidong Huang, and Zhihong Wang. 2014. VM scheduling strategies based on artificial intelligence in cloud testing. In Proceedings of the 15th IEEE/ACIS International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing (SNPD’14).
169. Junji Zhi, Sahil Suneja, and Eyal De Lara. 2014. The case for system testing with swift hierarchical vm fork. In Proceedings of the 6th USENIX Workshop on Hot Topics in Cloud Computing. 1, S2 (2014), S3.
170. Junzan Zhou, Shanping Li, Zhen Zhang, and Zhen Ye. 2013. Position paper: Cloud-based performance testing: Issues and challenges. In Proceedings of the International Workshop on Hot Topics in Cloud Services. ACM, 55--62.

Index Terms

• Computer systems organization

  • Architectures

    • Distributed architectures

      • Cloud computing

• Networks

  • Network services

    • Cloud computing

• Software and its engineering

  • Software creation and management

    • Software verification and validation

      • Software defect analysis

        • Software testing and debugging

  • Software organization and properties

    • Software system structures

      • Distributed systems organizing principles

        • Cloud computing