An Empirical Investigation into Programming Language Syntax

Abstract

Recent studies in the literature have shown that syntax remains a significant barrier to novice computer science students in the field. While this syntax barrier is known to exist, whether and how it varies across programming languages has not been carefully investigated. For this article, we conducted four empirical studies on programming language syntax as part of a larger analysis into the, so called, programming language wars. We first present two surveys conducted with students on the intuitiveness of syntax, which we used to garner formative clues on what words and symbols might be easy for novices to understand. We followed up with two studies on the accuracy rates of novices using a total of six programming languages: Ruby, Java, Perl, Python, Randomo, and Quorum. Randomo was designed by randomly choosing some keywords from the ASCII table (a metaphorical placebo). To our surprise, we found that languages using a more traditional C-style syntax (both Perl and Java) did not afford accuracy rates significantly higher than a language with randomly generated keywords, but that languages which deviate (Quorum, Python, and Ruby) did. These results, including the specifics of syntax that are particularly problematic for novices, may help teachers of introductory programming courses in choosing appropriate first languages and in helping students to overcome the challenges they face with syntax.

References

1. Beaubouef, T. and Mason, J. 2005. Why the high attrition rate for computer science students: Some thoughts and observations. SIGCSE Bull. 37, 2, 103--106.
2. Begel, A. and Graham, S. L. 2004. Spoken language support for software development. In Proceedings of the IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC’04). 271--272.
3. Bigham, J. P., Aller, M. B., Brudvik, J. T., Leung, J. O., Yazzolino, L. A., and Ladner, R. E. 2008. Inspiring blind high school students to pursue computer science with instant messaging chatbots. In Proceedings of the 39th SIGCSE Technical Symposium on Computer Science Education (SIGCSE’08). 449--453.
4. Binkley, D., Davis, M., Lawrie, D., and Morrell, C. 2009. To CamelCase or under score. In Proceedings of the IEEE 17th International Conference on Program Comprehension (ICPC’09). 158--167.
5. Bonar, J. and Soloway, E. 1983. Uncovering principles of novice programming. In Proceedings of the 10th ACM SIGACT-SIGPLAN Symposium on Principles of Programming Languages (POPL’83). 10--13.
6. Borning, A. and O’Shea, T. 1987. Deltatalk: An empirically and aesthetically motivated simplification of the Smalltalk-80 language. In Proceedings of the European Conference on Object-Oriented Programming (ECOOP’87). 1--10.
7. Brown, M. H. and Hershberger, J. 1991. Color and sound in algorithm animation. In Proceedings of the IEEE Workshop on Visual Languages (VL’91). 10--17.
8. Cleary, B., Exton, C., Buckley, J., and English, M. 2009. An empirical analysis of information retrieval based concept location techniques in software comprehension. Empir. Softw. Eng. 14, 93--130.
9. Comstock, C., Jiang, Z., and Naudé, P. 2007. Strategic software development: Productivity comparisons of general development programs. World Acad. Sci. Eng. Tech. 34, 25--30.
10. Cooper, S. 2010. The design of Alice. Trans. Comput. Educ. 10, 4, 15:1--15:16.
11. Dann, W., Cosgrove, D., Slater, D., Culyba, D., and Cooper, S. 2012. Mediated transfer: Alice 3 to Java. In Proceedings of the 43rd ACM Technical Symposium on Computer Science Education (SIGCSE ’12). 141--146.
12. Deißenböck, F. and Pizka, M. 2005. Concise and consistent naming. In Proceedings of the 13th International Workshop on Program Comprehension (IWPC’05). 97--106.
13. Delorey, D. P., Knutson, C. D., and Chun, S. 2007. Do programming languages affect productivity? A case study using data from open source projects. In Proceedings of the 1st International Workshop on Emerging Trends in FLOSS Research and Development (FLOSS’07).
14. Delorey, D. P., Knutson, C. D., and Davies, M. 2009. Mining programming language vocabularies from source code. In Proceedings of the Psychology of Programming Interest Group Conference (PPIG 2009).
15. Denny, P., Luxton-Reilly, A., and Tempero, E. 2012. All syntax errors are not equal. In Proceedings of the 17th ACM Annual Conference on Innovation and Technology in Computer Science Education (ITiCSE’12). 75--80.
16. Denny, P., Luxton-Reilly, A., Tempero, E., and Hendrickx, J. 2011. Understanding the syntax barrier for novices. In Proceedings of the 16th Annual Joint Conference on Innovation and Technology in Computer Science Education (ITiCSE’11). 208--212.
17. Dolado, J., Harman, M., Otero, M., and Hu, L. 2003. An empirical investigation of the influence of a type of side effects on program comprehension. Softw. Eng. 29, 7, 665--670.
18. Enbody, R. J. and Punch, W. F. 2010. Performance of Python CS1 students in mid-level non-Python CS courses. In Proceedings of the 41st ACM Technical Symposium on Computer Science Education (SIGCSE’10). 520--523.
19. Enbody, R. J., Punch, W. F., and McCullen, M. 2009. Python CS1 as preparation for C++ CS2. In Proceedings of the 40th ACM Technical Symposium on Computer Science Education (SIGCSE’09). 116--120.
20. Endrikat, S. and Hanenberg, S. 2011. Is aspect-oriented programming a rewarding investment into future code changes? A socio-technical study on development and maintenance time. In Proceedings of the IEEE 19th International Conference on Program Comprehension (ICPC’11). 51--60.
21. Feigenspan, J., Kästner, C., Liebig, J., Apel, S., and Hanenberg, S. 2012. Measuring programming experience. In Proceedings of the IEEE 20th International Conference on Program Comprehension (ICPC’12).
22. Findler, R. B., Clements, J., Flanagan, C., Flatt, M., Krishnamurthi, S., Steckler, P., and Felleisen, M. 2002. Drscheme: A programming environment for scheme. J. Funct. Program. 12, 2, 159--182.
23. Garlick, R. and Cankaya, E. C. 2010. Using Alice in CS1: A quantitative experiment. In Proceedings of the 15th Annual Conference on Innovation and Technology in Computer Science Education (ITiCSE’10). 165--168.
24. Green, T. R. G. and Petre, M. 1996. Usability analysis of visual programming environments: A cognitive dimensions framework. J. Vis. Lang. Comput. 7, 2, 131--174.
25. Hanenberg, S. 2010a. An experiment about static and dynamic type systems: Doubts about the positive impact of static type systems on development time. In Proceedings of the ACM International Conference on Object Oriented Programming Systems Languages and Applications (OOPSLA’10). 22--35.
26. Hanenberg, S. 2010b. Faith, hope, and love: An essay on software science’s neglect of human factors. In Proceedings of the ACM International Conference on Object Oriented Programming Systems Languages and Applications (OOPSLA’10). 933--946.
27. Hanenberg, S., Kleinschmager, S., and Josupeit-Walter, M. 2009. Does aspect-oriented programming increase the development speed for crosscutting code? An empirical study. In Proceedings of the 3rd International Symposium on Empirical Software Engineering and Measurement (ESEM’09). 156--167.
28. Holmboe, C. 2005. The linguistics of object-oriented design: Implications for teaching. In Proceedings of the 10th Annual SIGCSE Conference on Innovation and Technology in Computer Science Education (ITiCSE’05). 188--192.
29. Holt, R. C. and Cordy, J. R. 1988. The Turing programming language. Comm. ACM 31, 1410--1423.
30. Holt, R. C., Wortman, D. B., Barnard, D. T., and Cordy, J. R. 1977. SP/k: A system for teaching computer programming. Comm. ACM 20, 301--309.
31. Høst, E. W. 2007. Understanding programmer language. In Companion to the 22nd ACM SIGPLAN Conference on Object-Oriented Programming Systems and Applications (OOPSLA ’07). 943--944.
32. Høst, E. W. and Østvold, B. M. 2007. The programmer’s lexicon, volume I: The verbs. In Proceedings of the 7th IEEE International Working Conference on Source Code Analysis and Manipulation (SCAM’07). 193--202.
33. Hubert, L. 1977. Kappa revisited. Psychol. Bull. 84, 2, 289--297.
34. Hundhausen, C. D., Farley, S. F., and Brown, J. L. 2009. Can direct manipulation lower the barriers to computer programming and promote transfer of training?: An experimental study. ACM Trans. Comp. Hum. Interac. 16, 3, 1--40.
35. Kaptchuk, T. J. 1998. Intentional ignorance: A history of blind assessment and placebo controls in medicine. Bull. Hist. Med. 72, 3, 389--433.
36. Kelleher, C. and Pausch, R. 2005. Lowering the barriers to programming: A taxonomy of programming environments and languages for novice programmers. ACM Comput. Surv. 37, 2, 83--137.
37. Kiczales, G. 1996. Aspect-oriented programming. ACM Comput. Surv. 28, 4es, 154.
38. Kleinschmager, S., Hanenberg, S., Robbes, R., Tanter, E., and Stefik, A. 2012. Do static type systems improve the maintainability of software systems? An empirical study. In Proceedings of the IEEE 20th International Conference on Program Comprehension (ICPC’12).
39. Kline, P. 2002. An Easy Guide to Factor Analysis. New York, NY: Routledge.
40. Ko, A. J. and Myers, B. A. 2009. Finding causes of program output with the Java whyline. In Proceedings of the 27th International Conference on Human Factors in Computing Systems (CHI’09). 1569--1578.
41. Ko, A. J., Abraham, R., Beckwith, L., Blackwell, A., Burnett, M., Erwig, M., Scaffidi, C., Lawrance, J., Lieberman, H., Myers, B., Rosson, M. B., Rothermel, G., Shaw, M., and Wiedenbeck, S. 2011. The state of the art in end-user software engineering. ACM Comput. Surv. 43, 3, 21:1--21:44.
42. Lewis, B. and Ducassé, M. 2003. Using events to debug Java programs backwards in time. Companion of the 18th Annual ACM SIGPLAN Conference on Object-Oriented Programming, Systems, Languages, and Applications (OOPSLA’03). 96--97.
43. Lister, R., Adams, E. S., Fitzgerald, S., Fone, W., John, H., Lindholm, M., Mccartney, R., Mostroem, J. E., Sander, K., Seppaelae, Simon, B., and Thomas, L. 2004. A multi-national study of reading and tracing skills in novice programmers. In Working Group Reports on Innovation and Technology in Computer Science Education (ITiCSE’04). 119--150.
44. Lopes, C. V., Dourish, P., Lorenz, D. H., and Lieberherr, K. 2003. Beyond AOP: Toward naturalistic programming. In Companion of the 18th Annual ACM SIGPLAN Conference on Object-Oriented Programming, Systems, Languages, and Applications (OOPSLA’03). 198--207.
45. Lukas, G. 1972. Uses of the LOGO programming language in undergraduate instruction. In Proceedings of the ACM Annual Conference - Volume 2 (ACM 72). 1130--1136.
46. Maloney, J., Resnick, M., Rusk, N., Silverman, B., and Eastmond, E. 2010. The scratch programming language and environment. Trans. Comput. Educ. 10, 4, 16:1--16:15.
47. Markstrum, S. 2010. Staking claims: A history of programming language design claims and evidence: A positional work in progress. In Proceedings of the Conference on Evaluation and Usability of Programming Languages and Tools (PLATEAU’10). 7:1--7:5.
48. Mayer, C., Hanenberg, S., Robbes, R., Tanter, E., and Stefik, A. 2012. An empirical study of the influence of static type systems on the usability of undocumented software. In Proceedings of the Annual ACM SIGPLAN Conference on Object-Oriented Programming, Systems, Languages, and Applications (OOPSLA’12).
49. Mayrhauser, A. V. and Vans, A. M. 1997. Program understanding behavior during debugging of large scale software. In Proceedings of the 7th Workshop on Empirical Studies of Programmers (ESP’97). 157--179.
50. McIver, L. K. 2001. Syntactic and semantic issues in introductory programming education. Ph.D. thesis, Monash University.
51. Meyerovich, L. A. and Rabkin, A. S. 2012. Socio-PLT: Principles for programming language adoption. In Proceedings of the ACM International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software (Onward!’12). 39--54.
52. Myers, B. A., Pane, J. F., and Ko, A. 2004. Natural programming languages and environments. Comm. ACM 47, 9, 47--52.
53. Myers, B. A., Ko, A. J., Park, S. Y., Stylos, J., Latoza, T. D., and Beaton, J. 2008. More natural end-user software engineering. In Proceedings of the 4th International Workshop on End-User Software Engineering (WEUSE’08). 30--34.
54. Pane, J. F., Myers, B. A., and Ratanamahatana, C. A. 2001. Studying the language and structure in non-programmers’ solutions to programming problems. Int. J. Hum.-Comp. Stud. 54, 2, 237--264.
55. Pausch, R. 2008. Alice: A dying man’s passion. In Proceedings of the 39th SIGCSE Technical Symposium on Computer Science Education (SIGCSE’08). 1--1.
56. Pennington, N. 1987a. Comprehension strategies in programming. In Empirical Studies of Programmers: Second Workshop, G. M. Olson, S. Sheppard, E. Soloway, and B. Shneiderman Eds., Greenwood Publishing Group Inc., 100--113.
57. Pennington, N. 1987b. Stimulus structures and mental representations in expert comprehension of computer programs. Cogn. Psych. 19, 295--341.
58. Pinker, S. 1991. Rules of language. Sci. 253, 530--535.
59. Pothier, G., Tanter, E., and Piquer, J. 2007. Scalable omniscient debugging. In Proceedings of the 22nd Annual ACM SIGPLAN Conference on Object-Oriented Programming Systems and Applications (OOPSLA’07). 535--552.
60. Ramalingam, V. and Wiedenbeck, S. 1997. An empirical study of novice program comprehension in the imperative and object-oriented styles. In Proceedings of the 7th Workshop on Empirical Studies of Programmers (ESP’97). 124--139.
61. Resnick, M., Maloney, J., Monroy-Hernandez, A., Rusk, N., Eastmond, E., Brennan, K., Millner, A., Rosenbaum, E., Silver, J., Silverman, B., and Kafai, Y. 2009. Scratch: Programming for all. Comm. ACM 52, 11, 60--67.
62. Rossbach, C. J., Hofmann, O. S., and Witchel, E. 2010. Is transactional programming actually easier? SIGPLAN Not. 45, 5, 47--56.
63. Sanchez, J. and Aguayo, F. 2005. Blind learners programming through audio. In Extended Abstracts on Human Factors in Computing Systems (CHI’05). 1769--1772.
64. Schulte, C. and Magenheim, J. 2005. Novices’ expectations and prior knowledge of software development: Results of a study with high school students. In Proceedings of the 1st International Workshop on Computing Education Research (ICER’05). 143--153.
65. Science House of Commons and Technology Committee. 2010. Evidence check 2: Homeopathy, fourth report of session 2009-10. http://www.publications.parliament.uk/pa/cm200910/cmselect/cmsctech/45/45.pdf.
66. Smith, A. C., Cook, J. S., Francioni, J. M., Hossain, A., Anwar, M., and Rahman, M. F. 2004. Nonvisual tool for navigating hierarchical structures. In Proceedings of the10th International ACM SIGACCESS Conference on Computers and Accessibility (SIGACCESS’04). 133--139.
67. Soloway, E., Bonar, J., and Ehrlich, K. 1983. Cognitive strategies and looping constructs: An empirical study. Comm. ACM 26, 11, 853--860.
68. Stefik, A. 2008. On the design of program execution environments for non-sighted computer programmers. Ph.D. thesis, Washington State University.
69. Stefik, A., Alexander, R., Patterson, R., and Brown, J. 2007. WAD: A feasibility study using the wicked audio debugger. In Proceedings of the 15th IEEE International Conference on Program Comprehension (ICPC’07).
70. Stefik, A. and Gellenbeck, E. 2009. Using spoken text to aid debugging: An empirical study. In Proceedings of the 17th IEEE International Conference on Program Comprehension (ICPC’09). 110--119.
71. Stefik, A. and Gellenbeck, E. 2011. Empirical studies on programming language stimuli. Softw. Qual. J. 19, 1, 65--99.
72. Stefik, A., Hundhausen, C., and Patterson, R. 2011a. An empirical investigation into the design of auditory cues to enhance computer program comprehension. Int. J. Hum.-Comp. Stud. 69, 12, 820--838.
73. Stefik, A., Hundhausen, C., and Smith, D. 2011b. On the design of an educational infrastructure for the blind and visually impaired in computer science. In Proceedings of the 42nd ACM Technical Symposium on Computer Science Education (CSE’11).
74. Stefik, A., Siebert, S., Stefik, M., and Slattery, K. 2011c. An empirical comparison of the accuracy rates of novices using the quorum, perl, and randomo programming languages. In Proceedings of the 3rd ACM SIGPLAN Workshop on Evaluation and Usability of Programming Languages and Tools (PLATEAU’11). 3--8.
75. Steimann, F. 2006. The paradoxical success of aspect-oriented programming. In Proceedings of the 21st Annual ACMSIGPLAN Conference on Object-Oriented Programming Systems, Languages, and Applications (OOPSLA’06). 481--497.
76. Stylos, J. and Clarke, S. 2007. Usability implications of requiring parameters in objects’ constructors. In Proceedings of the 29th International Conference on Computer Software Education (ICSE’07). 529--539.
77. Stylos, J. and Myers, B. A. 2008. The implications of method placement on API learnability. In Proceedings of the 16th ACM SIGSOFT International Symposium on Foundations of Software Engineering (SIGSOFT’08/FSE-16). 105--112.
78. Taylor, J. 1990. Analyzing novices analyzing prolog: What stories do novices tell themselves about prolog? Instruct. Sci. 19, 283--309.
79. Teitelbaum, T. and Reps, T. 1981. The Cornell program synthesizer: A syntax-directed programming environment. Comm. ACM 24, 9, 563--573.
80. Tew, A. E. and Guzdial, M. 2011. The FCS1: A language independent assessment of CS1 knowledge. In Proceedings of the 42nd ACM Technical Symposium on Computer Science Education (SIGCSE’11). 111--116.
81. Vickers, P. and Alty, J. L. 2002. When bugs sing. Interact. Comp. 14, 6, 793--819.
82. Vogt, W. P. 2006. Quantitative Research Methods for Professionals in Education and Other Fields, 1st Ed. Allyn and Bacon, Columbus, OH.
83. Whitney, P. 1998. The Psychology of Language. Houghton Mifflin Company, Boston, MA.

Index Terms

• Computing methodologies

  • Artificial intelligence

    • Philosophical/theoretical foundations of artificial intelligence

      • Cognitive science

• Human-centered computing

  • Human computer interaction (HCI)

    • Empirical studies in HCI

• Software and its engineering

  • Software notations and tools

    • General programming languages