A Survey on the Usage of Eye-Tracking in Computer Programming

Abstract

Traditional quantitative research methods of data collection in programming, such as questionnaires and interviews, are the most common approaches for researchers in this field. However, in recent years, eye-tracking has been on the rise as a new method of collecting evidence of visual attention and the cognitive process of programmers. Eye-tracking has been used by researchers in the field of programming to analyze and understand a variety of tasks such as comprehension and debugging. In this article, we will focus on reporting how experiments that used eye-trackers in programming research are conducted, and the information that can be collected from these experiments. In this mapping study, we identify and report on 63 studies, published between 1990 and June 2017, collected and gathered via manual search on digital libraries and databases related to computer science and computer engineering. Among the five main areas of research interest are program comprehension and debugging, which received an increased interest in recent years, non-code comprehension, collaborative programming, and requirements traceability research, which had the fewest number of publications due to possible limitations of the eye-tracking technology in this type of experiments. We find that most of the participants in these studies were students and faculty members from institutions of higher learning, and while they performed programming tasks on a range of programming languages and programming representations, we find Java language and Unified Modeling Language (UML) representation to be the most used materials. We also report on a range of eye-trackers and attention tracking tools that have been utilized, and find Tobii eye-trackers to be the most used devices by researchers.

References

1. Nasir Ali, Zohreh Sharafi, Yann-Gael Gueheneuc, and Giuliano Antoniol. 2015. An empirical study on the importance of source code entities for requirements traceability. Empirical Software Engineering 20, 2 (2015), 442--478.
2. Lorin W. Anderson, David R. Krathwohl, and Benjamin Samuel Bloom. 2001. A Taxonomy for Learning, Teaching, and Assessing: A Revision of Bloom’s Taxonomy of Educational Objectives. Allyn 8 Bacon.
3. Magdalena Andrzejewska, Anna Stoli’ńska, Władysław Błasiak, Paweł Pkȩczkowski, Roman Rosiek, Boźena Roźek, Mirosława Sajka, and Dariusz Wcisło. 2016. Eye-tracking verification of the strategy used to analyse algorithms expressed in a flowchart and pseudocode. Interactive Learning Environments 24, 8 (2016), 1981--1995.
4. Stuart Charters and Barbara Kitchenham. 2007. Guidelines for Performing Systematic Literature Reviews in Software Engineering. Technical Report. Keele University and Durham University Joint Report.
5. Roman Bednarik, Teresa Busjahn, and Carsten Schulte (Eds.). 2014. Eye Movements in Programming Education: Analyzing the Experts Gaze. Technical Report. University of Eastern Finland, Joensuu, Finland.
6. Roman Bednarik, Niko Myller, Erkki Sutinen, and Markku Tukiainen. 2005. Applying eye-movememt tracking to program visualization. In Proceedings of the 2005 IEEE Symposium on Visual Languages and Human-Centric Computing. IEEE, 302--304.
7. Roman Bednarik, Niko Myller, Erkki Sutinen, and Markku Tukiainen. 2005. Effects of experience on gaze behaviour during program animation. In Proceedings of the 17th Annual Psychology of Programming Interest Group Workshop. 49--61.
8. Roman Bednarik, Niko Myller, Erkki Sutinen, and Markku Tukiainen. 2006. Analyzing individual differences in program comprehension. Technology Instruction Cognition and Learning 3, 3/4 (2006), 205.
9. Roman Bednarik, Niko Myller, Erkki Sutinen, and Markku Tukiainen. 2006. Program visualization: Comparing eye-tracking patterns with comprehension summaries and performance. In Proceedings of the 18th Annual Psychology of Programming Workshop. 66--82.
10. Roman Bednarik and Markku Tukiainen. 2004. Visual attention and representation switching in Java program debugging: A study using eye movement tracking. In Proceedings of the 16th Annual Workshop of the Psychology of Programming Interest Group. 159--169.
11. Roman Bednarik and Markku Tukiainen. 2004. Visual attention tracking during program debugging. In Proceedings of the 3rd Nordic Conference on Human-Computer Interaction. ACM, 331--334.
12. Roman Bednarik and Markku Tukiainen. 2005. Effects of display blurring on the behavior of novices and experts during program debugging. In CHI’05 Extended Abstracts on Human Factors in Computing Systems. ACM, 1204--1207.
13. Roman Bednarik and Markku Tukiainen. 2006. An eye-tracking methodology for characterizing program comprehension processes. In Proceedings of the 2006 Symposium on Eye Tracking Research and Applications. ACM, 125--132.
14. Roman Bednarik and Markku Tukiainen. 2007. Analysing and interpreting quantitative eye-tracking data in studies of programming: Phases of debugging with multiple representations. In Proceedings of the 19th Annual Workshop of the Psychology of Programming Interest Group (PPIG’07). 158--172.
15. Roman Bednarik and Markku Tukiainen. 2007. Validating the restricted focus viewer: A study using eye-movement tracking. Behavior Research Methods 39, 2 (2007), 274--282.
16. Roman Bednarik and Markku Tukiainen. 2008. Temporal eye-tracking data: Evolution of debugging strategies with multiple representations. In Proceedings of the 2008 Symposium on Eye Tracking Research and Applications. ACM, 99--102.
17. Mordechai Ben-Ari, Roman Bednarik, Ronit Ben-Bassat Levy, Gil Ebel, Andrés Moreno, Niko Myller, and Erkki Sutinen. 2011. A decade of research and development on program animation: The Jeliot experience. Journal of Visual Languages and Computing 22, 5 (2011), 375--384.
18. Alan F. Blackwell, Anthony R. Jansen, and Kim Marriott. 2000. Restricted focus viewer: A tool for tracking visual attention. Theory and Application of Diagrams. Springer, 162--177.
19. Crescencio Bravo, Rafael Duque, and Jess Gallardo. 2013. A groupware system to support collaborative programming: Design and experiences. Journal of Systems and Software 86, 7 (2013), 1759--1771.
20. Teresa Busjahn, Roman Bednarik, Andrew Begel, Martha Crosby, James H. Paterson, Carsten Schulte, Bonita Sharif, and Sascha Tamm. 2015. Eye movements in code reading: Relaxing the linear order. In Proceedings of the 2015 IEEE 23rd International Conference on Program Comprehension. IEEE Press, 255--265.
21. Teresa Busjahn, Roman Bednarik, and Carsten Schulte. 2014. What influences dwell time during source code reading?: Analysis of element type and frequency as factors. In Proceedings of the Symposium on Eye Tracking Research and Applications. ACM, 335--338.
22. Teresa Busjahn, Carsten Schulte, and Andreas Busjahn. 2011. Analysis of code reading to gain more insight in program comprehension. In Proceedings of the 11th Koli Calling International Conference on Computing Education Research. ACM, 1--9.
23. Teresa Busjahn, Carsten Schulte, Bonita Sharif, Andrew Begel, Michael Hansen, Roman Bednarik, Paul Orlov, Petri Ihantola, Galina Shchekotova, Maria Antropova, and others. 2014. Eye tracking in computing education. In Proceedings of the 10th Annual Conference on International Computing Education Research. ACM, 3--10.
24. Teresa Busjahn, Carsten Schulte, Sascha Tamm, and Roman Bednarik (Eds.). 2015. Eye Movements in Programming Education II: Analyzing the Novice’s Gaze. Technical Report. Freie Universität Berlin, Department of Mathematics and Computer Science, Berlin, Germany. 1--41 pages.
25. Zoya Bylinskii, Michelle A. Borkin, Nam Wook Kim, Hanspeter Pfister, and Aude Oliva. 2015. Eye fixation metrics for large scale evaluation and comparison of information visualizations. In Workshop on Eye Tracking and Visualization. Springer, 235--255.
26. Martha E. Crosby and Jan Stelovsky. 1990. How do we read algorithms? A case study. Computer 23, 1 (1990), 25--35.
27. Sarah D’Angelo and Andrew Begel. 2017. Improving communication between pair programmers using shared gaze awareness. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. ACM, 6245--6290.
28. Benoit De Smet, Lorent Lempereur, Zohreh Sharafi, Yann-Gaël Guéhéneuc, Giuliano Antoniol, and Naji Habra. 2014. Taupe: Visualizing and analyzing eye-tracking data. Science of Computer Programming 79 (2014), 260--278.
29. Fadi P. Deek and James A. McHugh. 1998. A survey and critical analysis of tools for learning programming. Computer Science Education 8, 2 (1998), 130--178.
30. Tore Dyba, Torgeir Dingsoyr, and Geir K. Hanssen. 2007. Applying systematic reviews to diverse study types: An experience report. In Proceedings of the 1st International Symposium on Empirical Software Engineering and Measurement (ESEM’07). IEEE, 225--234.
31. Joseph H. Goldberg and Xerxes P. Kotval. 1999. Computer interface evaluation using eye movements: Methods and constructs. International Journal of Industrial Ergonomics 24, 6 (1999), 631--645.
32. Anabela Gomes and António José Mendes. 2007. An environment to improve programming education. In Proceedings of the 2007 International Conference on Computer Systems and Technologies. ACM, 88.
33. Anabela Gomes and António José Mendes. 2007. Learning to program—Difficulties and solutions. In Proceedings of the International Conference on Engineering Education (ICEE), Vol. 2007.
34. Yann-Gael Guéhéneuc, Huzefa Kagdi, and Jonathan I. Maletic. 2009. Working session: Using eye-tracking to understand program comprehension. In Proceedings of the IEEE 17th International Conference on Program Comprehension (ICPC’09). IEEE, 278--279.
35. Victor Henning and Jan Reichelt. 2008. Mendeley—A last.fm for research?. In Proceedings of the IEEE 4th International Conference on eScience (eScience’0). IEEE, 327--328.
36. Ahmad Jbara and Dror G. Feitelson. 2015. How programmers read regular code: A controlled experiment using eye tracking. In Proceedings of the 2015 IEEE 23rd International Conference on Program Comprehension. IEEE, 244--254.
37. Tomoko Kashima, Shimpei Matsumoto, and Shuichi Yamagishi. 2014. Proposal of a method to measure difficulty level of programming code with eye-tracking. In Human-Computer Interaction. Advanced Interaction Modalities and Techniques. Springer, 264--272.
38. Barbara Kitchenham and Pearl Brereton. 2013. A systematic review of systematic review process research in software engineering. Information and Software Technology 55, 12 (2013), 2049--2075.
39. Barbara A. Kitchenham, Shari Lawrence Pfleeger, Lesley M. Pickard, Peter W. Jones, David C. Hoaglin, Khaled El Emam, and Jarrett Rosenberg. 2002. Preliminary guidelines for empirical research in software engineering. IEEE Transactions on Software Engineering 28, 8 (2002), 721--734.
40. Tomasz Kocejko, Jacek Ruminski, Adam Bujnowski, and Jerzy Wtorek. 2016. The evaluation of eGlasses eye tracking module as an extension for Scratch. In Proceedings of the 2016 9th International Conference on Human System Interactions (HSI). IEEE, 465--471.
41. Martin Konopka. 2015. Combining eye tracking with navigation paths for identification of cross-language code dependencies. In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering. ACM, 1057--1059.
42. Meng-Lung Lai, Meng-Jung Tsai, Fang-Ying Yang, Chung-Yuan Hsu, Tzu-Chien Liu, Silvia Wen-Yu Lee, Min-Hsien Lee, Guo-Li Chiou, Jyh-Chong Liang, and Chin-Chung Tsai. 2013. A review of using eye-tracking technology in exploring learning from 2000 to 2012. Educational Research Review 10 (2013), 90--115.
43. Iain Milne and Glenn Rowe. 2002. Difficulties in learning and teaching programming views of students and tutors. Education and Information Technologies 7, 1 (2002), 55--66.
44. David Montano, Jairo Aponte, and Andrian Marcus. 2009. Sv3D meets eclipse. In Proceedings of the 5th IEEE International Workshop on Visualizing Software for Understanding and Analysis (VISSOFT’09). IEEE, 51--54.
45. IT Chan Mow. 2008. Issues and difficulties in teaching novice computer programming. In Innovative Techniques in Instruction Technology, e-learning, e-assessment, and Education. Springer, 199--204.
46. Gail C. Murphy, Mik Kersten, and Leah Findlater. 2006. How are Java software developers using the Elipse IDE? IEEE Software 23, 4 (2006), 76--83.
47. Seppo Nevalainen and Jorma Sajaniemi. 2004. Comparison of three eye tracking devices in psychology of programming research. In Proceedings of the 16th Annual Psychology of Programming Interest Group Workshop. 151--158.
48. Christopher Palmer and Bonita Sharif. 2016. Towards automating fixation correction for source code. In Proceedings of the 9th Biennial ACM Symposium on Eye Tracking Research and Applications. ACM, 65--68.
49. Oren Patashnik. 1988. BibTeXing. documentation for general BibTeX users. Electronic Document Accompanying BibTeX Distribution.
50. Patrick Peachock, Nicholas Iovino, and Bonita Sharif. 2017. Investigating Eye Movements in Natural Language and C++ Source Code—A Replication Experiment. Springer International Publishing, Cham, 206--218. DOI:http://dx.doi.org/10.1007/978-3-319-58628-1_17
51. Kai Petersen, Robert Feldt, Shahid Mujtaba, and Michael Mattsson. 2008. Systematic mapping studies in software engineering. In Proceedings of the 12th International Conference on Evaluation and Assessment in Software Engineering (EASE’08), Vol. 8. 68--77.
52. Kai Petersen, Sairam Vakkalanka, and Ludwik Kuzniarz. 2015. Guidelines for conducting systematic mapping studies in software engineering: An update. Information and Software Technology 64 (2015), 1--18.
53. Razvan Petrusel and Jan Mendling. 2013. Eye-tracking the factors of process model comprehension tasks. In Advanced Information Systems Engineering. Springer, 224--239.
54. Sami Pietinen, Roman Bednarik, Tatiana Glotova, Vesa Tenhunen, and Markku Tukiainen. 2008. A method to study visual attention aspects of collaboration: Eye-tracking pair programmers simultaneously. In Proceedings of the 2008 Symposium on Eye Tracking Research and Applications (ETRA’08). ACM, New York, 39--42. DOI:http://dx.doi.org/10.1145/1344471.1344480
55. Sami Pietinen, Roman Bednarik, and Markku Tukiainen. 2010. Shared visual attention in collaborative programming: A descriptive analysis. In Proceedings of the 2010 ICSE Workshop on Cooperative and Human Aspects of Software Engineering (CHASE’10). ACM, New York, 21--24. DOI:http://dx.doi.org/10.1145/1833310.1833314
56. Gerardo Cepeda Porras and Yann-Gaël Guéhéneuc. 2010. An empirical study on the efficiency of different design pattern representations in UML class diagrams. Empirical Software Engineering 15, 5 (2010), 493--522.
57. Keith Rayner. 1998. Eye movements in reading and information processing: 20 years of research. Psychological Bulletin 124, 3 (1998), 372.
58. Keith Rayner. 2009. Eye movements and attention in reading, scene perception, and visual search. The Quarterly Journal of Experimental Psychology 62, 8 (2009), 1457--1506.
59. V. Renumol, S. Jayaprakash, and D. Janakiram. 2009. Classification of cognitive difficulties of students to learn computer programming. Indian Institute of Technology, India.
60. Mitchel Resnick, John Maloney, Andrés Monroy-Hernández, Natalie Rusk, Evelyn Eastmond, Karen Brennan, Amon Millner, Eric Rosenbaum, Jay Silver, Brian Silverman, and others. 2009. Scratch: Programming for all. Communications of the ACM 52, 11 (2009), 60--67.
61. Paige Rodeghero and Collin McMillan. 2015. An empirical study on the patterns of eye movement during summarization tasks. In Proceedings of the 2015 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM’15). 1--10. DOI:http://dx.doi.org/10.1109/ESEM.2015.7321188
62. Paige Rodeghero, Collin McMillan, Paul W. McBurney, Nigel Bosch, and Sidney D’Mello. 2014. Improving automated source code summarization via an eye-tracking study of programmers. In Proceedings of the 36th International Conference on Software Engineering (ICSE’14). ACM, New York, 390--401. DOI:http://dx.doi.org/10.1145/2568225.2568247
63. Pablo Romero, Benedict Boulay, Richard Cox, and Rudi Lutz. 2003. Java debugging strategies in multi-representational environments. In Proceedings of the 15th Annual Psychology of Programming Interest Group Workshop. 421--435.
64. Md. Sami Uddin, Varun Gaur, Carl Gutwin, and Chanchal K. Roy. 2015. On the comprehension of code clone visualizations: A controlled study using eye tracking. In Proceedings of the 2015 IEEE 15th International Working Conference on Source Code Analysis and Manipulation (SCAM’15). 161--170. DOI:http://dx.doi.org/10.1109/SCAM.2015.7335412
65. Zohreh Sharafi. 2011. A systematic analysis of software architecture visualization techniques. In Proceedings of the 2011 IEEE 19th International Conference on Program Comprehension (ICPC’11). 254--257. DOI:http://dx.doi.org/10.1109/ICPC.2011.40
66. Zohreh Sharafi, A. Marchetto, A. Susi, G. Antoniol, and Yann-Gaël Guéhéneuc. 2013. An empirical study on the efficiency of graphical vs. textual representations in requirements comprehension. In Proceedings of the 2013 IEEE 21st International Conference on Program Comprehension (ICPC’13). 33--42. DOI:http://dx.doi.org/10.1109/ICPC.2013.6613831
67. Zohreh Sharafi, Zéphyrin Soh, and Yann-Gaël Guéhéneuc. 2015. A systematic literature review on the usage of eye-tracking in software engineering. Information and Software Technology 67 (2015), 79--107.
68. Zohreh Sharafi, Zéphyrin Soh, Yann-Gaël Guéhéneuc, and G. Antoniol. 2012. Women and men—Different but equal: On the impact of identifier style on source code reading. In Proceedings of the 2012 IEEE 20th International Conference on Program Comprehension (ICPC’12). 27--36. DOI:http://dx.doi.org/10.1109/ICPC.2012.6240505
69. Bonita Sharif, Michael Falcone, and Jonathan I. Maletic. 2012. An eye-tracking study on the role of scan time in finding source code defects. In Proceedings of the Symposium on Eye Tracking Research and Applications (ETRA’12). ACM, New York, 381--384. DOI:http://dx.doi.org/10.1145/2168556.2168642
70. Bonita Sharif, G. Jetty, J. Aponte, and E. Parra. 2013. An empirical study assessing the effect of seeit 3D on comprehension. In Proceedings of the 2013 1st IEEE Working Conference on Software Visualization (VISSOFT’13). 1--10. DOI:http://dx.doi.org/10.1109/VISSOFT.2013.6650519
71. Bonita Sharif and Huzefa Kagdi. 2011. On the use of eye tracking in software traceability. In Proceedings of the 6th International Workshop on Traceability in Emerging Forms of Software Engineering (TEFSE’11). ACM, New York, 67--70. DOI:http://dx.doi.org/10.1145/1987856.1987872
72. Bonita Sharif and Jonathan I. Maletic. 2010. An eye tracking study on camelcase and under_score identifier styles. In Proceedings of the 2010 IEEE 18th International Conference on Program Comprehension (ICPC’10). 196--205. DOI:http://dx.doi.org/10.1109/ICPC.2010.41
73. Bonita Sharif and Jonathan I. Maletic. 2010. An eye tracking study on the effects of layout in understanding the role of design patterns. In Proceedings of the 2010 IEEE International Conference on Software Maintenance (ICSM’10). 1--10. DOI:http://dx.doi.org/10.1109/ICSM.2010.5609582
74. Kshitij Sharma, Patrick Jermann, Marc-Antoine Nüssli, and Pierre Dillenbourg. 2013. Understanding collaborative program comprehension: Interlacing gaze and dialogues. In Proceedings of Computer Supported Collaborative Learning (CSCL’13).
75. Judy Sheard, S. Simon, Margaret Hamilton, and Jan Lönnberg. 2009. Analysis of research into the teaching and learning of programming. In Proceedings of the 5th International Workshop on Computing Education Research Workshop. ACM, 93--104.
76. Dag I. K. Sjøberg, Jo Erskine Hannay, Ove Hansen, Vigdis By Kampenes, Amela Karahasanovic, N.-K. Liborg, and Anette C. Rekdal. 2005. A survey of controlled experiments in software engineering. IEEE Transactions on Software Engineering 31, 9 (2005), 733--753.
77. Zéphyrin Soh. 2011. Context and vision: Studying two factors impacting program comprehension. In Proceedings of the 2011 IEEE 19th International Conference on Program Comprehension (ICPC’11). 258--261. DOI:http://dx.doi.org/10.1109/ICPC.2011.37
78. Zéphyrin Soh, Zohreh Sharafi, Bertrand Van den Plas, Gerardo Cepeda Porras, Yann-Gaël Guéhéneuc, and Giuliano Antoniol. 2012. Professional status and expertise for UML class diagram comprehension: An empirical study. In Proceedings of the 2012 IEEE 20th International Conference on Program Comprehension (ICPC’12). 163--172. DOI:http://dx.doi.org/10.1109/ICPC.2012.6240484
79. Koji Torii, Ken-ichi Matsumoto, Kumiyo Nakakoji, Yoshihiro Takada, Shingo Takada, and Kazuyuki Shima. 1999. Ginger2: An environment for computer-aided empirical software engineering. IEEE Transactions on Software Engineering 25, 4 (July 1999), 474--492. DOI:http://dx.doi.org/10.1109/32.799942
80. Rachel Turner, Michael Falcone, Bonita Sharif, and Alina Lazar. 2014. An eye-tracking study assessing the comprehension of C++ and python source code. In Proceedings of the Symposium on Eye Tracking Research and Applications (ETRA’14). ACM, New York, 231--234. DOI:http://dx.doi.org/10.1145/2578153.2578218
81. Hidetake Uwano, Akito Monden, and Ken-ichi Matsumoto. 2008. DRESREM 2: An analysis system for multi-document software review using reviewers’ eye movements. In Proceedings of the 3rd International Conference on Software Engineering Advances (ICSEA’08). 177--183. DOI:http://dx.doi.org/10.1109/ICSEA.2008.49
82. Hidetake Uwano, Masahide Nakamura, Akito Monden, and Ken-ichi Matsumoto. 2007. Exploiting eye movements for evaluating reviewer’s performance in software review. IEICE Transactions on Fundamentals of Electronics, Communications and Computer Sciences 90, 10 (2007), 2290--2300.
83. Johannes Zagermann, Ulrike Pfeil, and Harald Reiterer. 2016. Measuring cognitive load using eye tracking technology in visual computing. In Proceedings of the 6th Workshop on Beyond Time and Errors on Novel Evaluation Methods for Visualization (BELIV’16). 78--85.

Index Terms

• Human-centered computing

  • Human computer interaction (HCI)

    • Empirical studies in HCI

  • Visualization

    • Empirical studies in visualization

    • Visualization design and evaluation methods