Getting users to pay attention to anti-phishing education: evaluation of retention and transfer

ABSTRACT

Educational materials designed to teach users not to fall for phishing attacks are widely available but are often ignored by users. In this paper, we extend an embedded training methodology using learning science principles in which phishing education is made part of a primary task for users. The goal is to motivate users to pay attention to the training materials. In embedded training, users are sent simulated phishing attacks and trained after they fall for the attacks. Prior studies tested users immediately after training and demonstrated that embedded training improved users' ability to identify phishing emails and websites. In the present study, we tested users to determine how well they retained knowledge gained through embedded training and how well they transferred this knowledge to identify other types of phishing emails. We also compared the effectiveness of the same training materials delivered via embedded training and delivered as regular email messages. In our experiments, we found that: (a) users learn more effectively when the training materials are presented after users fall for the attack (embedded) than when the same training materials are sent by email (non-embedded); (b) users retain and transfer more knowledge after embedded training than after non-embedded training; and (c) users with higher Cognitive Reflection Test (CRT) scores are more likely than users with lower CRT scores to click on the links in the phishing emails from companies with which they have no account.

References

1. Anandpara, V., Dingman, A., Jakobsson, M., Liu, D., and Roinestad, H. Phishing IQ tests measure fear, not ability. Usable Security (USEC'07)(2007). http://usablesecurity.org/papers/anandpara.pdf.
2. Anderson, J. R. Rules of the Mind. Lawrence Erlbaum Associates, Inc., 1993.
3. Anderson, J. R., and Simon, H. A. Situated learning and education. Educational Researcher 25 (1996), 5--11.
4. Anton, A. I., Earp, E. A. J. B., Bolchini, D., He, Q., Jensen, C., and Stufflebeam, W. The Lack of Clarity in Financial Privacy Policies and the Need for Standardization. IEEE Security and Privacy 2(2) (2004), pp. 36--45. Retrieved Dec 20, 2004, http://www.theprivacyplace.org/papers/glb_secPriv_tr.pdf.
5. Clark, R. C. and E. M. Richard. 2002. E-Learning and the science of instruction: proven guidelines for consumers and designers of multimedia learning. Pfeiffer, San Francisco, USA.
6. Dhamija, R., Tygar, J. D., and Hearst, M. 2006. Why phishing works. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Montréal, Québec, Canada, April 22 - 27, 2006). R. Grinter, T. Rodden, P. Aoki, E. Cutrell, R. Jeffries, and G. Olson, Eds. CHI '06. ACM Press, New York, NY, 581--590. DOI=http://doi.acm.org/10.1145/1124772.1124861.
7. Downs, J. S., Holbrook, M. B., and Cranor, L. F. 2006. Decision strategies and susceptibility to phishing. In Proceedings of the Second Symposium on Usable Privacy and Security (Pittsburgh, Pennsylvania, July 12 - 14, 2006). SOUPS '06, vol. 149. ACM Press, New York, NY, 79--90. DOI=http://doi.acm.org/10.1145/1143120.1143131.
8. eBay Toolbar. Retrieved December 30, 2006. http://pages.ebay.com/ebay_toolbar/
9. Fette, I., N. Sadeh and A. Tomasic. Learning to Detect Phishing Emails. June 2006. ISRI Technical report, CMU-ISRI-06-112 (To be presented at WWW 2007).htt p://reports-archive.adm.cs.cmu.edu/anon/isri2006/CMU-ISRI-06-112.pdf.
10. Frederick, S. Cognitive reflection and decision making. Journal of Economic Perspectives 19, 4 (2005), 25--42.
11. Keinan, G. Decision making under stress: scanning of alternatives under controllable and uncontrollable threats. Journal of personality and social psychology 52, 3 (1987), 639--644.
12. Kirkley, J. R., and et al. Problem-based embedded training: An instructional methodology for embedded training using mixed and virtual reality technologies. In Interservice/Industry Training, Simulation, and Education Conference (I/ITSEC) (2003). http://www.iforces.org/downloads/problem-based.pdf.
13. Klein, G. Sources of power: How people make decisions? The MIT Press Cambridge, Massachusetts The MIT Press, Cambridge, Massachusetts, London, England, February 1999.
14. Kumaraguru, P., Sheng, S., Acquisti, A., Cranor, L. F., and Hong, J. Teaching johnny not to fall for phish. Tech. rep., Cranegie Mellon University, 2007. http://www.cylab.cmu.edu/files/cmucylab07003.pdf.
15. Kumaraguru, P., Y. Rhee, A. Acquisti, L. Cranor, J. Hong, and E. Nunge. In Proceedings of CHI 2007. Protecting People from Phishing: The Design and Evaluation of an Embedded Training Email System.
16. Mayer, R. E. Multimedia Learning. 2001. New York Cambridge University Press.
17. Mayer, R. E., and Anderson, R. B. The instructive animation: Helping students build connections between words and pictures in multimedia learning. Journal of Educational Psychology 84, 4 (December 1992), 444--452.
18. Merrienboer, J. V., de croock, M., and Jelsma, O. The transfer paradox: Effects of contextual interference on retention andtransfer performance of a complex cognitive skill. Perceptual and motor skills 84 (1997), 784--786.
19. Moreno, R., Mayer, R. E., Spires, H. A., and Lester, J. C. The case for social agency in computer-based teaching: Do students learn more deeply when they interact with animated pedagogical agents? Cognition and Instruction 19, 2 (2001), 177--213.
20. Robila, S. A., J. James and W. Ragucci. 2006. Don't be a phish: steps in user education. ITICSE '06: Proceedings of the 11th annual SIGCSE conference on Innovation and technology in computer science education. pp 237--241. New York, NY, USA.
21. Rubin, D. C., and Wenzel, A. E. One hundred years of forgetting: A quantitative description of retention. Psychological Review 103, 4 (1996), 734--760.
22. Schmidt, R. A., and Bjork, R. A. New conceptualizations of practice: Common principles in three paradigms suggest new concepts for training. Psychological Science 3, 4 (July 1992), 207--217.
23. Sheng, S., B. Magnien, P. Kumaraguru, A. Acquisti, L. Cranor, J. Hong, and E. Nunge. Anti-Phishing Phil: The Design and Evaluation of a Game That Teaches People Not to Fall for Phish. To appear in Symposium on Usable Privacy and Security 2007.
24. SpamAssasin. Retrieved September 10, 2006. http://spamassassin.apache.org/
25. SpoofGuard. Retrieved September 10, 2006, http://crypto.stanford.edu/SpoofGuard/
26. SpoofStick. Retrieved September 10, 2006. http://www.spoofstick.com/
27. SquirrelMail. Retrieved September 10, 2006. http://www.squirrelmail.org/
28. Tversky, A., and Kahneman, D. Judgment under Uncertainty: Heuristics and Biases. Science 185, 4157 (1974), 1124--1131.
29. Tversky, A., and Shafir, E. The disjunction effect in choice under uncertainty. American Psychological Society 3, 5 (September 1992), 305--309.
30. Whitten, A and J. D. Tygar. 1999. Why Johnny Can't Encrypt: A Usability Evaluation of PGP 5.0. Proceedings of the 8th USENIX Security Symposium. http://www.cs.berkeley.edu/~tygar/papers/Why_Johnny_Cant_Encrypt/USENIX.pdf.
31. Whitten, W. B., and Bjork, R. A. Learning from tests: Effects of spacing. Journal of Verbal Learning and Verbal Behavior 16, 4 (August 1977), 465--478.

Index Terms

• Human-centered computing

  • Human computer interaction (HCI)

• Security and privacy

  • Systems security

    • Operating systems security

• Social and professional topics

  • Computing / technology policy

    • Computer crime