Strong agile metrics: mining log data to determine predictive power of software metrics for continuous delivery teams

ABSTRACT

ING Bank, a large Netherlands-based internationally operating bank, implemented a fully automated continuous delivery pipe-line for its software engineering activities in more than 300 teams, that perform more than 2500 deployments to production each month on more than 750 different applications. Our objective is to examine how strong metrics for agile (Scrum) DevOps teams can be set in an iterative fashion. We perform an exploratory case study that focuses on the classification based on predictive power of software metrics, in which we analyze log data derived from two initial sources within this pipeline. We analyzed a subset of 16 metrics from 59 squads. We identified two lagging metrics and assessed four leading metrics to be strong.

References

1. E. Bouwers, J. Visser en A. van Deursen, „Getting What You Measure,” Communications of the ACM, vol. 55, nr. 7, pp. 54-59, 2012.
2. J. Humble en D. Farley, Continuous Delivery, reliable software releases through build, test and deployment automation, Addison-Wesley, 2010.
3. H. Huijgens, R. van Solingen en A. van Deursen, „How to build a good practice software project portfolio?,” in ACM Companion Proceedings of the 36th International Conference on Software Engineering (ICSE), 2014.
4. J. Münch, F. Fagerholm, P. Johnson, J. Pirttilahti, J. Torkkel en J. Jäarvinen, „Creating minimum viable products in industry-academia collaborations,” in Lean Enterprise Software and Systems, Springer Berlin Heidelberg, 2013, pp. 137-151.
5. „State of DevOps Report,” Puppet, 2016.
6. H. Huijgens, R. Lamping, D. Stevens, H. Rothengatter en G. Gousios, „Strong Agile Metrics - Technical Report TUD-SERG-2017-010,” Delft University of Technology, 2017.
7. D. George en M. Mallery, SPSS for Windows Step by Step: A Simple Guide and Reference, 17.0 update (10a ed.) red., Boston: Pearson., 2010.
8. H. Huijgens, A. van Deursen en R. van Solingen, „The Effects of Perceived Value and Stakeholder Satisfaction on Software Project Impact,” Information and Software Technology, 2017.
9. W. Hopkins, A new view of statistics, Internet Society for Sport Science, 2000.
10. T. Hall, A. Rainer en N. Baddoo, „Implementing Software Process Improvement: An Empirical Study,” Software Process Improvement and Practice, vol. 7, pp. 3-15, 2002.
11. T. Dyba, „An Empirical Investigation of the Key Factors for Success in Software Process Improvement,” IEEE Transactions on Software Engineering, vol. 31, nr. 5, pp. 410-424, 2005.
12. T. Chow en D.-B. Cao, „A survey study of critical success factors in agile software projects,” The Journal of Systems and Software, vol. 81, pp. 961-971, 2008.
13. S. C. Misra, V. Kumar en U. Kumar, „Identifying some important success factors in adopting agile software development practices,” The Journal of Systems and Software, vol. 82, pp. 1869-1890, 2009.
14. J. Sutherland, A. Viktorov, J. Blount en N. Puntikov, „Distributed Scrum: Agile project management with outsourced development teams,” in IEEE 40th Annual Hawaii International Conference on System Sciences (HICSS), 2007.
15. M. Beedle, M. Devos, Y. Sharon, K. Schwaber en J. Sutherland, „SCRUM: An extension Pattern Language for Hyperproductive Software Development,” in Pattern Languages of Program Design, Addison-Wesley, 2000, pp. 637-651.
16. J. Sutherland, N. Harrison en J. Riddle, „Teams that Finish Early Accelerate Faster: A Pattern Language for High Performing Scrum Teams,” in 47th Hawaii International Conference on System Science, 2014.
17. K. El Emam en A. Günes Koru, „A replicated survey of IT software project failures,” IEEE software, vol. 25, nr. 5, pp. 84-90, 2008.
18. M. Bhardwaj en A. Rana, „Key Software Metrics and its Impact on each other for Software Development Projects,” ACM SIGSOFT Software Engineering Notes, vol. 41, nr. 1, pp. 1-4, 2016.
19. D. Rubinstein, „Standish group report: There’s less development chaos today,” Software Development Times, vol. 1, 2007.
20. R. Sonnekus en L. Labuschagne, „Establishing the Relationship between IT Project Management Maturity and IT Project Success in a South African Context,” Proc. 2004,” PMSA Global Knowledge Conf., Project Management South Africa, pp. 183-192, 2004.
21. H. Barki, S. Rivard en J. Talbot, „Toward an assessment of software development risk,” Journal of Management Information Systems, vol. 10, pp. 203-223, 1993.
22. J. Jiang en G. Klein, „Software development risks to project effectiveness,” Journal of Systems and Software, vol. 52, nr. 1, pp. 3-10, 2000.
23. R. Schmidt, K. Lyytinen, P. Cule en M. Keil, „Identifying software project risks: An international Delphi study,” Journal of management information systems, vol. 17, nr. 4, pp. 5-36, 2001.
24. M. V. Mäntylä, M. Jørgensen, P. Ralph en H. Erdogmus, „Guest editorial for special section on success and failure in software engineering,” Empirical Software Engineering, vol. April, pp. 1-17, 2017.
25. R. Premrai, M. Shepperd, B. Kitchenham en P. Forselius, „An Empirical Analysis of Software Productivity Over Time,” in IEEE International Symposium Software Metrics, Como, Italy, 2005.

Index Terms

• General and reference

  • Cross-computing tools and techniques

    • Metrics