Towards Quality Assurance of Software Product Lines with Adversarial Configurations

ABSTRACT

Software product line (SPL) engineers put a lot of effort to ensure that, through the setting of a large number of possible configuration options, products are acceptable and well-tailored to customers' needs. Unfortunately, options and their mutual interactions create a huge configuration space which is intractable to exhaustively explore. Instead of testing all products, machine learning is increasingly employed to approximate the set of acceptable products out of a small training sample of configurations. Machine learning (ML) techniques can refine a software product line through learned constraints and a priori prevent non-acceptable products to be derived. In this paper, we use adversarial ML techniques to generate adversarial configurations fooling ML classifiers and pinpoint incorrect classifications of products (videos) derived from an industrial video generator. Our attacks yield (up to) a 100% misclassification rate and a drop in accuracy of 5%. We discuss the implications these results have on SPL quality assurance.

References

1. Marco Barreno, Blaine Nelson, Russell Sears, Anthony D Joseph, and J Doug Tygar. 2006. Can machine learning be secure?. In Proceedings of the 2006 ACM Symposium on Information, computer and communications security. ACM, New York, NY, USA, 16--25.
2. Don S. Batory. 2005. Feature Models, Grammars, and Propositional Formulas. In SPLC'05 (LNCS), Vol. 3714. Springer, Berlin, Germany, 7--20.
3. Richard Bellman. 1957. Dynamic Programming (1 ed.). Princeton University Press, Princeton, NJ, USA.
4. David Benavides, Sergio Segura, and Antonio Ruiz-Cortes. 2010. Automated Analysis of Feature Models 20 years Later: a Literature Review. Information Systems 35, 6 (2010), 615--636.
5. Thorsten Berger, Ralf Rublack, Divya Nair, Joanne M. Atlee, Martin Becker, Krzysztof Czarnecki, andAndrzej Wasowski. 2013. A Survey of Variability Modeling in Industrial Practice. In Proceedings of the Seventh International Workshop on Variability Modelling of Software-intensive Systems (VaMoS '13). ACM, New York, NY, USA, Article 7, 8 pages.
6. Battista Biggio, Igino Corona, Davide Maiorca, Blaine Nelson, Nedim Srndic, Pavel Laskov, Giorgio Giacinto, and Fabio Roli. 2013. Evasion attacks against machine learning at test time. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases. Springer Berlin, Berlin, Heidelberg, 387402.
7. B. Biggio, L. Didaci, G. Fumera, and F. Roli. 2013. Poisoning attacks to compromise face templates. In 2013 International Conference on Biometrics (ICB). IEEE, New York, USA, 1--7.
8. Battista Biggio, Giorgio Fumera, and Fabio Roli. 2014. Pattern recognition systems under attack: Design issues and research challenges. International Journal of Pattern Recognition and Artificial Intelligence 28, 07 (2014), 1460002.
9. Battista Biggio, Giorgio Fumera, and Fabio Roli. 2014. Security evaluation of pattern classifiers under attack. IEEE transactions on knowledge and data engineering 26,4 (2014),984--996.
10. BattistaBiggio, Blaine Nelson, andPavelLaskov. 2012. PoisoningAttacks Against Support Vector Machines. In Proceedings of the 29th International Coference on International Conference on Machine Learning (ICML'12). Omnipress, USA, 1467--1474. http://dl.acm.org/citation.cfm?id=3042573.3042761
11. Battista Biggio and Fabio Roli. 2018. Wild patterns: Ten years after the rise of adversarial machine learning. Pattern Recognition 84 (2018), 317--331.
12. Eric Bodden, Tarsis Toledo, Marcio Ribeiro, Claus Brabrand, Paulo Borba, and Mira Mezini. 2013. SPLLIFT: statically analyzing software product lines in minutes instead of years. In ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI '13, Seattle, WA, USA, June 16--19, 2013. ACM, New York, USA, 355--364.
13. Quentin Boucher, Andreas Classen, Paul Faber, and Patrick Heymans. 2010. Introducing TVL, a Text-based Feature Modelling. In Fourth International Workshop on Variability Modelling of Software-Intensive Systems, Linz, Austria, January 27--29, 2010. Proceedings (ICB-Research Report), David Benavides, Don S. Batory, and Paul Griinbacher (Eds.), Vol. 37. Universitat Duisburg-Essen, Essen, Germany, 159--162. http://www.vamos-workshop.net/proceedings/VaMoS_2010_Proceedings.pdf
14. Tom Brown, Dandelion Mane, Aurko Roy, Martin Abadi, and Justin Gilmer. 2017. Adversarial Patch. https://arxiv.org/pdf/1712.09665.pdf (2017).
15. Michael Buhrmester, Tracy Kwang, and Samuel D Gosling. 2011. Amazon's Mechanical Turk: A new source of inexpensive, yet high-quality, data? Perspectives on psychological science 6, 1 (2011), 3--5.
16. Nitesh V Chawla, Kevin W Bowyer, Lawrence O Hall, and W Philip Kegelmeyer. 2002. SMOTE: synthetic minority over-sampling technique. Journal of artificial intelligence research 16 (2002), 321--357.
17. Andreas Classen, QuentinBoucher, and PatrickHeymans. 2011. AText-basedAp-proach to Feature Modelling: Syntax and Semantics of TVL. Science of Computer Programming, Special Issue on Software Evolution, Adaptability and Variability 76, 12 (2011), 1130--1143.
18. Paul Clements and Linda M. Northrop. 2001. Software Product Lines : Practices and Patterns. Addison-Wesley Professional, Boston, USA.
19. Jean-Marc Davril, Patrick Heymans, Guillaume Becan, and Mathieu Acher. 2015. OnBreakingTheCurseofDimensionalityinReverseEngineeringFeatureModels. In 17th International Configuration Workshop (17th International Configuration Workshop), Vol. 17th International Configuration Workshop. Vienna, Austria. https://hal.inria.fr/hal-01243571
20. Ambra Demontis, Marco Melis, Maura Pintor, Matthew Jagielski, Battista Biggio, Alina Oprea, Cristina Nita-Rotaru, and Fabio Roli. 2018. On the Intriguing Connections of Regularization, Input Gradients and Transferability of Evasion and Poisoning Attacks. CoRR abs/1809.02861 (2018). arXiv:1809.02861 http://arxiv.org/abs/1809.02861
21. Ambra Demontis, Marco Melis, Maura Pintor, MatthewJagielski, Battista Biggio, Alina Oprea, Cristina Nita-Rotaru, and Fabio Roli. 2019. Why Do Adversarial Attacks Transfer? Explaining Transferability of Evasion and Poisoning Attacks. In 28th USENIX Security Symposium (USENIX Security 19). USENIX Association, Santa Clara, CA. https://www.usenix.org/conference/usenixsecurity19/presentation/demontis
22. Guneet S Dhillon, Kamyar Azizzadenesheli, Zachary C Lipton, Jeremy Bernstein, Jean Kossaifi, Aran Khanna, and Anima Anandkumar. 2018. Stochastic activation pruning for robust adversarial defense. arXiv preprint arXiv:1803.01442 (2018).
23. Richard W. Dosselman and Xue Dong Yang. 2012. No-Reference Noise and Blur Detection via the Fourier Transform. Technical Report. University of Regina, CANADA.
24. Gamaleldin F Elsayed, Shreya Shankar, Brian Cheung, Nicolas Papernot, Alex Kurakin, Ian Goodfellow, and Jascha Sohl-Dickstein. 2018. Adversarial Examples that Fool both Human and Computer Vision. arXiv preprint arXiv:1802.08195 (2018).
25. Ivan Evtimov, Kevin Eykholt, Earlence Fernandes, Tadayoshi Kohno, Bo Li, Atul Prakash, Amir Rahmati, and Dawn Song. 2017. Robust Physical-World Attacks on Deep Learning Models. arXiv preprint arXiv:1707.08945 1 (2017).
26. Jose Angel Galindo Duarte, Mauricio Alferez, Mathieu Acher, Benoit Baudry, and David Benavides. 2014. A Variability-Based Testing Approach for Synthesizing Video Sequences. In ISSTA '14: International Symposium on Software Testing and Analysis. San Jose, California, United States. https://hal.inria.fr/hal-01003148
27. Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. In Advances in neural information processing systems. 2672--2680.
28. Chuan Guo, Mayank Rana, Moustapha Cisse, and Laurens van der Maaten. 2017. Countering adversarial images using input transformations. arXiv preprint arXiv:1711.00117 (2017).
29. Jianmei Guo, Krzysztof Czarnecki, Sven Apel, Norbert Siegmund, and Andrzej Wasowski. 2013. Variability-aware performance prediction:Astatisticallearning approach. In ASE.
30. Axel Halin, Alexandre Nuttinck, Mathieu Acher, Xavier Devroey, Gilles Perrouin, and Benoit Baudry. 2018. Test them all, is it worth it? Assessing configuration sampling on the JHipster Web development stack. Empirical Software Engineering (July 2018). https://doi.org/10.07980 Empirical Software Engineering journal.
31. Axel Halin, Alexandre Nuttinck, Mathieu Acher, Xavier Devroey, Gilles Perrouin, and Benoit Baudry. 2019. Test them all, is it worth it? Assessing configuration sampling on the JHipster Web development stack. Empirical Software Engineering 24,2 (2019), 674--717.
32. Roberto Ierusalimschy. 2006. Programmingin Lua, SecondEdition. Lua.Org.
33. Cem Kaner, James Bach, and Bret Pettichord. 2001. Lessons Learned in Software Testing. John Wiley & Sons, Inc., New York, NY, USA.
34. Alexander Knuppel, Thomas Thiim, Stephan Mennicke, Jens Meinicke, and Ina Schaefer. 2018. Is There a Mismatch between Real-World Feature Models and Product-Line Research?. In Software Engineering und Software Management 2018, Fachtagung des GI-Fachbereichs Softwaretechnik, SE 2018, 5.-9. Marz 2018, Ulm, Germany. (LNI), Matthias Tichy, Eric Bodden, Marco Kuhrmann, Stefan Wagner, and Jan-Philipp Steghofer (Eds.), Vol. P-279. Gesellschaft fur Informatik, 53--54. https://dl.gi.de/20.500.12116/16312
35. Alexey Kurakin, Ian Goodfellow, and Samy Bengio. 2016. Adversarial examples in the physical world. arXivpreprintarXiv:1607.02533 (2016).
36. Axel Legay and Gilles Perrouin. 2017. On Quantitative Requirements for Product Lines. In Proceedings of the Eleventh International Workshop on Variability Modelling of Software-intensive Systems (VAMOS '17). ACM, New York, NY, USA, 2--4.
37. Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and AdrianVladu.2017. Towardsdeeplearningmodels resistanttoadversarialattacks. arXiv preprint arXiv:1706.06083 (2017).
38. Flavio Medeiros, Christian Kastner, Marcio Ribeiro, Rohit Gheyi, and Sven Apel. 2016. A Comparison of 10 Sampling Algorithms for Configurable Systems. In Proceedings of the 38th International Conference on Software Engineering (ICSE 16). ACM, New York, NY, USA, 643--654.
39. Sarah Nadi, Thorsten Berger, Christian Kastner, and Krzysztof Czarnecki. 2014. Mining configuration constraints: static analyses and empirical results. In 36th International Conference on Software Engineering, ICSE '14, Hyderabad, India -May 31 - June 07, 2014. 140--151.
40. Vivek Nair, Tim Menzies, Norbert Siegmund, and Sven Apel. 2017. Using bad learners to find good configurations. In Proceedings of the 2017 11th Joint Meeting on Foundations ofSoftware Engineering, ESEC/FSE 2017, Paderborn, Germany, September4--8,2017, Eric Bodden, Wilhelm Schafer, Arie van Deursen, andAndrea Zisman (Eds.). ACM, 257--267.
41. Blaine Nelson, Marco Barreno, Fuching Jack Chi, Anthony D Joseph, Benjamin IP Rubinstein, Udam Saini, Charles A Sutton, J Doug Tygar, and Kai Xia. 2008. Exploiting Machine Learning to Subvert Your Spam Filter. LEET 8 (2008), 1--9.
42. Jeho Oh, Don S. Batory, Margaret Myers, and Norbert Siegmund. 2017. Finding near-optimal configurations in product lines by random sampling. In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering, ESEC/FSE 2017, Paderborn, Germany, September 4--8, 2017. 61--71.
43. N. Papernot, P. McDaniel, S. Jha, M. Fredrikson, Z. B. Celik, and A. Swami. 2016. The Limitations of Deep Learning in Adversarial Settings. In 2016 IEEE European Symposium on Security and Privacy (EuroS P). 372--387.
44. Kexin Pei, Yinzhi Cao, Junfeng Yang, and Suman Jana. 2017. DeepXplore: Automated Whitebox Testing of Deep Learning Systems. In Proceedings of the 26th Symposium on Operating Systems Principles (SOSP '17). ACM, New York, NY, USA, 1--18.
45. Juliana Alves Pereira, Hugo Martin, Mathieu Acher, Jean-Marc JÃl'zÃl'quel, Goetz Botterweck, and Anthony Ventresque. 2019. Learning Software Configuration Spaces: A Systematic Literature Review. arXiv:arXiv:1906.03018
46. Quentin Plazar, Mathieu Acher, Gilles Perrouin, Xavier Devroey, and Maxime Cordy. 2019. Uniform Sampling of SAT Solutions for Configurable Systems: Are We There Yet?. In 12th IEEE Conference on Software Testing, Validation and Verification, ICST 2019, Xi'an, China, April 22--27, 2019. 240--251.
47. Klaus Pohl, Günter Böckle, and Frank J. van der Linden. 2005. Software Product Line Engineering: Foundations, Principles and Techniques. Springer-Verlag.
48. A. Sarkar, Jianmei Guo, N. Siegmund, S. Apel, and K. Czarnecki. 2015. Cost-Efficient Sampling for Performance Prediction of Configurable Systems (T). In ASE'15.
49. Pierre-Yves Schobbens, Patrick Heymans, Jean-Christophe Trigaux, and Yves Bontemps. 2007. Generic semantics of feature diagrams. Comput. Netw. 51, 2 (2007), 456--479.
50. Mahmood Sharif, Sruti Bhagavatula, Lujo Bauer, and Michael K Reiter. 2016. Accessorize to a crime: Real and stealthy attacks on state-of-the-art face recognition. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security. ACM, 1528--1540.
51. Norbert Siegmund, Alexander Grebhahn, Christian Kästner, and Sven Apel. {n.d.}. Performance-Influence Models for Highly Configurable Systems. In ESEC/FSE'15.
52. Norbert Siegmund, Marko RosenmüLler, Christian KäStner, Paolo G. Giarrusso, Sven Apel, and Sergiy S. Kolesnikov. 2013. Scalable Prediction of Non-functional Properties in Software Product Lines: Footprint and Memory Consumption. Inf. Softw. Technol. (2013).
53. Norbert Siegmund, Stefan Sobernig, and Sven Apel. 2017. Attributed Variability Models: Outside the Comfort Zone. In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering (ESEC/FSE 2017). ACM, New York, NY, USA, 268--278.
54. Daniel Strüber, Julia Rubin, Thorsten Arendt, Marsha Chechik, Gabriele Taentzer, and Jennifer Plöger. 2018. Variability-based model transformation: formal foundation and application. Formal Asp. Comput. 30, 1 (2018), 133--162.
55. Paul Temple, Mathieu Acher, Jean-Marc Jézéquel, and Olivier Barais. 2017. Learning Contextual-Variability Models. IEEE Software 34, 6 (2017), 64--70.
56. Paul Temple, José Angel Galindo Duarte, Mathieu Acher, and Jean-Marc Jézéquel. 2016. Using Machine Learning to Infer Constraints for Product Lines. In Software Product Line Conference (SPLC). Beijing, China.
57. Maurice H. ter Beek, Alessandro Fantechi, Stefania Gnesi, and Franco Mazzanti. 2016. Modelling and analysing variability in product families: Model checking of modal transition systems with variability constraints. J. Log. Algebr. Meth. Program. 85, 2 (2016), 287--315.
58. Maurice H. ter Beek, Alessandro Fantechi, Stefania Gnesi, and Laura Semini. 2016. Variability-Based Design of Services for Smart Transportation Systems. In Leveraging Applications of Formal Methods, Verification and Validation: Discussion, Dissemination, Applications - 7th International Symposium, ISoLA 2016, Imperial, Corfu, Greece, October 10--14, 2016, Proceedings, Part II. 465--481.
59. Maurice H. ter Beek and Axel Legay. 2019. Quantitative Variability Modeling and Analysis. In Proceedings of the 13th International Workshop on Variability Modelling of Software-Intensive Systems (VAMOS '19). ACM, New York, NY, USA, Article 13, 2 pages.
60. Thomas Thüm, Sven Apel, Christian Kästner, Ina Schaefer, and Gunter Saake. 2014. A Classification and Survey of Analysis Strategies for Software Product Lines. Comput. Surveys (2014).
61. Mahsa Varshosaz, Mustafa Al-Hajjaji, Thomas Thüm, Tobias Runge, Mohammad Reza Mousavi, and Ina Schaefer. 2018. A classification of product sampling for software product lines. In Proceeedings of the 22nd International Systems and Software Product Line Conference - Volume 1, SPLC 2018, Gothenburg, Sweden, September 10--14, 2018. 1--13.
62. Mengshi Zhang, Yuqun Zhang, Lingming Zhang, Cong Liu, and Sarfraz Khurshid. 2018. DeepRoad: GAN-based Metamorphic Testing and Input Validation Framework for Autonomous Driving Systems. In Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering (ASE 2018). ACM, New York, NY, USA, 132--142.

Index Terms

• Software and its engineering

  • Software creation and management

    • Software development techniques

      • Reusability

        • Software product lines