What is the foundation of evidence of human factors decisions in language design? an empirical study on programming language workshops

ABSTRACT

In recent years, the programming language design community has engaged in rigorous debate on the role of empirical evidence in the design of general purpose programming languages. Some scholars contend that the language community has failed to embrace a form of evidence that is non-controversial in other disciplines (e.g., medicine, biology, psychology, sociology, physics, chemistry), while others argue that a science of language design is unrealistic. While the discussion will likely persist for some time, we begin here a systematic evaluation of the use of empirical evidence with human users, documenting, paper-by-paper, the evidence provided for human factors decisions, beginning with 359 papers from the workshops PPIG, Plateau, and ESP. This preliminary work provides the following contributions: an analysis of the 1) overall quantity and quality of empirical evidence used in the workshops, and of the 2) overall significant challenges to reliably coding academic papers. We hope that, once complete, this long-term research project will serve as a practical catalog designers can use when evaluating the impact of a language feature on human users.

References

1. R. P. Buse, C. Sadowski, and W. Weimer. Benefits and barriers of user evaluation in software engineering research. SIGPLAN Not., 46(10):643–656, Oct. 2011.
2. S. Clark. Evaluating a new programming language. In The 13th Annual Workshop of Psychology of Programmers Interest Group (PPIG), Poole, United Kingdom, 2001.
3. W. Dann, D. Cosgrove, D. Slater, D. Culyba, and S. Cooper. Mediated transfer: Alice 3 to java. In Proceedings of the 43rd ACM technical symposium on Computer Science Education, SIGCSE ’12, pages 141–146, New York, NY, USA, 2012. ACM.
4. R. J. Enbody, W. F. Punch, and M. McCullen. Python cs1 as preparation for c++ cs2. In Proceedings of the 40th ACM technical symposium on Computer science education, SIGCSE ’09, pages 116–120, New York, NY, USA, 2009. ACM.
5. S. Endrikat and S. Hanenberg. Is aspect-oriented programming a rewarding investment into future code changes? a socio-technical study on development and maintenance time. In Proceedings of the 2011 IEEE 19th International Conference on Program Comprehension, ICPC ’11, pages 51–60, Washington, DC, USA, 2011. IEEE Computer Society.
6. T. R. G. Green and M. Petre. Usability analysis of visual programming environments: A cognitive dimensions framework. Journal of Visual Languages and Computing, 7(2):131–174, 1996.
7. S. Hanenberg. Faith, hope, and love: an essay on software science’s neglect of human factors. In Proceedings of the ACM International Conference on Object Oriented Programming Systems Languages and Applications, OOPSLA ’10, pages 933–946, New York, NY, 2010. ACM.
8. L. Hubert. Kappa revisited. Psychological Bulletin, 84(2):289–297, 1977.
9. C. D. Hundhausen, S. F. Farley, and J. L. Brown. Can direct manipulation lower the barriers to computer programming and promote transfer of training?: An experimental study. ACM Transactions on Computer Human Interaction, 16(3):1–40, 2009.
10. T. J. Kaptchuk. Intentional ignorance: A history of blind assessment and placebo controls in medicine. Bulletin of the History of Medicine, 72(3):389–433, 1998.
11. C. Kelleher and R. Pausch. Lowering the barriers to programming: A taxonomy of programming environments and languages for novice programmers. ACM Computing Surveys, 37(2):83–137, 2005.
12. A. Ko, T. LaToza, and M. Burnett. A practical guide to controlled experiments of software engineering tools with human participants. Empirical Software Engineering, pages 1–32, 2013.
13. A. J. Ko, R. Abraham, L. Beckwith, A. Blackwell, M. Burnett, M. Erwig, C. Scaffidi, J. Lawrance, H. Lieberman, B. Myers, M. B. Rosson, G. Rothermel, M. Shaw, and S. Wiedenbeck. The state of the art in end-user software engineering. ACM Comput. Surv., 43(3):21:1–21:44, Apr. 2011.
14. A. J. Ko and B. A. Myers. Finding causes of program output with the java whyline. In CHI ’09: Proceedings of the 27th International Conference on Human Factors in Computing Systems, pages 1569–1578, New York, NY, USA, 2009. ACM.
15. S. Markstrum. Staking claims: a history of programming language design claims and evidence: A positional work in progress. In Evaluation and Usability of Programming Languages and Tools, PLATEAU ’10, pages 7:1–7:5, New York, NY, USA, 2010. ACM.
16. A. v. Mayrhauser and A. M. Vans. Comprehension processes during large scale maintenance. In ICSE ’94: Proceedings of the 16th International Conference on Software Engineering, pages 39–48, Los Alamitos, CA, 1994. IEEE Computer Society Press.
17. U. D. of Education. Evaluation of evidence-based practices in online learning: A meta-analysis and review of online learning studies, 2010.
18. U. D. of Education Institute of Education Sciences. What Works Clearinghouse Procedures and Standards Handbook. U.S. Department of Education, 2.1 edition, 2010.
19. N. Pennington. Comprehension strategies in programming. In G. M. Olson, S. Sheppard, E. Soloway, and B. Shneiderman, editors, Empirical Studies of Programmers: Second Workshop, pages 100–113, Westport, CT, 1987. Greenwood Publishing Group Inc.
20. N. Pennington. Stimulus structures and mental representations in expert comprehension of computer programs. Cognitive Psychology, 19:295–341, 1987.
21. L. Prechelt. An empirical comparison of seven programming languages. Computer, 33(10):23 –29, oct. 2000.
22. M. Resnick, J. Maloney, A. Monroy-Hernández, N. Rusk, E. Eastmond, K. Brennan, A. Millner, E. Rosenbaum, J. Silver, B. Silverman, and Y. Kafai. Scratch: Programming for all. Communications of the ACM, 52(11):60–67, 2009.
23. J. Rimmer, J. Good, E. Harris, and M. Balaam. Class participation and shyness: affect and learning to program. In The 23rd Annual Workshop of Psychology of Programmers Interest Group (PPIG), Heslington, York, 2011.
24. A. Stefik and S. Siebert. An empirical investigation into programming language syntax. Trans. Comput. Educ., 13(4):19:1–19:40, Nov. 2013.
25. W. Tichy. Should computer scientists experiment more? Computer, 31(5):32–40, 1998.
26. W. P. Vogt. Quantitative Research Methods for Professionals in Education and Other Fields. Allyn and Bacon, Columbus, OH, 1st edition, 2006.

Index Terms

• Human-centered computing

  • Human computer interaction (HCI)

• Software and its engineering

  • Software notations and tools

    • General programming languages