skip to main content
10.1145/3055305.3055310acmotherconferencesArticle/Chapter ViewAbstractPublication PageshotsosConference Proceedingsconference-collections
research-article

Use of Phishing Training to Improve Security Warning Compliance: Evidence from a Field Experiment

Published: 04 April 2017 Publication History

Abstract

The current approach to protect users from phishing attacks is to display a warning when the webpage is considered suspicious. We hypothesize that users are capable of making correct informed decisions when the warning also conveys the reasons why it is displayed. We chose to use traffic rankings of domains, which can be easily described to users, as a warning trigger and evaluated the effect of the phishing warning message and phishing training. The evaluation was conducted in a field experiment. We found that knowledge gained from the training enhances the effectiveness of phishing warnings, as the number of participants being phished was reduced. However, the knowledge by itself was not sufficient to provide phishing protection. We suggest that integrating training in the warning interface, involving traffic ranking in phishing detection, and explaining why warnings are generated will improve current phishing defense.

References

[1]
2014a. Global Phishing Survey 1H2014: Trends and Domain Name Use. (2014). https://docs.apwg.org/reports/APWG_Global_Phishing_Report_1H_2014.pdf.
[2]
2014b. Phishing Activity Trends Report. (2014). https://docs.apwg.org/reports/apwg_trends_report_q2_2014.pdf.
[3]
2015. Email authentication. (2015). https://support.google.com/mail/answer/180707?hl=en.
[4]
2016. Chrome Privacy White Paper. (2016). https://www.google.com/chrome/browser/privacy/whitepaper.html.
[5]
Devdatta Akhawe and Adrienne Porter Felt. 2013. Alice in Warningland: A Large-Scale Field Study of Browser Security Warning Effectiveness. In Usenix Security. 257--272.
[6]
John R Anderson. 1983. The architecture of cognition. Psychology Press.
[7]
Lujo Bauer, Cristian Bravo-Lillo, LF Cranor, and Elli Fragkaki. 2013. Warning design guidelines. Pittsburgh, PA: Carnegie Mellon University (2013).
[8]
L. E. Bourne and A. F. Healy. 2012. Training and Its Cognitive Underpinnings. In Training cognition: Optimizing Efficiency, Durability, and Generalizability, A. F. Healy and L. E. Bourne (Eds.). Psychology Press.
[9]
Cristian Bravo-Lillo, Lorrie Cranor, Julie Downs, Saranga Komanduri, Stuart Schechter, and Manya Sleeper. 2012. Operating system framed in case of mistaken identity: measuring the success of web-based spoofing attacks on OS password-entry dialogs. In Proceedings of the 2012 ACM conference on computer and communications security. ACM, 365--377.
[10]
Neil Chou, Robert Ledesma, Yuka Teraguchi, John C Mitchell, and others. 2004. Client-Side Defense Against Web-Based Identity Theft. In NDSS.
[11]
Jason W Clark and Damon McCoy. 2013. There Are No Free iPads: An Analysis of Survey Scams as a Business. In LEET.
[12]
Fergus IM Craik and Janine M Jennings. 1992. Human memory. Lawrence Erlbaum Associates, Inc, Chapter Handbook of aging and cognition.
[13]
Lorrie Faith Cranor. 2008. A Framework for Reasoning About the Human in the Loop. UPSEC 8 (2008), 1--15.
[14]
Rachna Dhamija and J Doug Tygar. 2005. The battle against phishing: Dynamic security skins. In Proceedings of the 2005 symposium on Usable privacy and security. ACM, 77--88.
[15]
Julie S Downs, Barbagallo Donato, and Acquisti Alessandro. 2015. Predictors of risky decisions: Improving judgment and decision making based on evidence from phishing attacks. In Neuroeconomics, judgment, and decision making, Evan A Wilhelms and Valerie F Reyna (Eds.). Psychology Press, 239--253.
[16]
Julie S Downs, Mandy B Holbrook, and Lorrie Faith Cranor. 2006. Decision strategies and susceptibility to phishing. In Proceedings of the second symposium on Usable privacy and security. ACM, 79--90.
[17]
W Keith Edwards, Erika Shehan Poole, and Jennifer Stoll. 2008. Security automation considered harmful?. In Proceedings of the 2007 Workshop on New Security Paradigms. ACM, 33--42.
[18]
Serge Egelman, Lorrie Faith Cranor, and Jason Hong. 2008. You've been warned: an empirical study of the effectiveness of web browser phishing warnings. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 1065--1074.
[19]
Adrienne Porter Felt, Alex Ainslie, Robert W Reeder, Sunny Consolvo, Somas Thyagaraja, Alan Bettes, Helen Harris, and Jeff Grimes. 2015. Improving SSL Warnings: Comprehension and Adherence. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. ACM, 2893--2902.
[20]
J Paul Frantz. 1994. Effect of location and procedural explicitness on user processing of and compliance with product warnings. Human Factors: The Journal of the Human Factors and Ergonomics Society 36, 3 (1994), 532--546.
[21]
Stefan Görling. 2006. The myth of user education. In Virus Bulletin Conference, Vol. 11. 13--16.
[22]
Amir Herzberg and Ahmad Gbara. 2004. Trustbar: Protecting (even naive) web users from spoofing and phishing attacks. Technical Report. ryptology ePrint Archive, Report 2004/155. http://eprint.iacr.org/2004/155.
[23]
Robert R Hoffman. 2014. The psychology of expertise: Cognitive research and empirical AI. Psychology Press.
[24]
Ponnurangam Kumaraguru, Justin Cranshaw, Alessandro Acquisti, Lorrie Cranor, Jason Hong, Mary Ann Blair, and Theodore Pham. 2009. School of phish: a real-world evaluation of anti-phishing training. In Proceedings of the 5th Symposium on Usable Privacy and Security. ACM, 3--15.
[25]
Kenneth R Laughery and Michael S Wogalter. 2006. Designing effective warnings. Reviews of human factors and ergonomics 2, 1 (2006), 241--271.
[26]
Eric Lin, Saul Greenberg, Eileah Trotter, David Ma, and John Aycock. 2011. Does domain highlighting help people identify phishing sites?. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 2075--2084.
[27]
Christian Ludl, Sean McAllister, Engin Kirda, and Christopher Kruegel. 2007. On the effectiveness of techniques to detect phishing sites. In DIMVA, Vol. 7. Springer, 20--39.
[28]
Justin Ma, Lawrence K Saul, Stefan Savage, and Geoffrey M Voelker. 2009a. Beyond blacklists: learning to detect malicious web sites from suspicious URLs. In Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining. ACM, 1245--1254.
[29]
Justin Ma, Lawrence K Saul, Stefan Savage, and Geoffrey M Voelker. 2009b. Identifying suspicious URLs: an application of large-scale online learning. In Proceedings of the 26th Annual International Conference on Machine Learning. ACM, 681--688.
[30]
Kathryn Parsons, Agata McCormac, Malcolm Pattinson, Marcus Butavicius, and Cate Jerram. 2015. The design of phishing studies: Challenges for researchers. Computers & Security (2015).
[31]
Robert W Proctor and Addie Dutta. 1995. Skill acquisition and human performance. Sage Publications, Inc.
[32]
Steve Sheng, Mandy Holbrook, Ponnurangam Kumaraguru, Lorrie Faith Cranor, and Julie Downs. 2010. Who falls for phish?: a demographic analysis of phishing susceptibility and effectiveness of interventions. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 373--382.
[33]
Steve Sheng, Brad Wardman, Gary Warner, Lorrie Cranor, Jason Hong, and Chengshan Zhang. 2009. An empirical analysis of phishing blacklists. In Sixth Conference on Email and Anti-Spam (CEAS). California, USA.
[34]
Andreas Sotirakopoulos, Kirstie Hawkey, and Konstantin Beznosov. 2011. On the challenges in usable security lab studies: lessons learned from replicating a study on SSL warnings. In Proceedings of the Seventh Symposium on Usable Privacy and Security. 3--15.
[35]
Colin Whittaker, Brian Ryner, and Marria Nazif. 2010. Large-Scale Automatic Classification of Phishing Pages. In NDSS, Vol. 10.
[36]
Christopher D Wickens. 2014. Effort in human factors performance and decision making. Human Factors: The Journal of the Human Factors and Ergonomics Society (2014), 1--8.
[37]
Michael S Wogalter, Dave DeJoy, and Kenneth R Laughery. 2005. Warnings and risk communication. CRC Press.
[38]
Michael S Wogalter, Russell J Sojourner, and John W Brelsford. 1997. Comprehension and retention of safety pictorials. Ergonomics 40, 5 (1997), 531--542.
[39]
Min Wu, Robert C Miller, and Simson L Garfinkel. 2006. Do security toolbars actually prevent phishing attacks?. In Proceedings of the SIGCHI conference on Human Factors in computing systems. ACM, 601--610.
[40]
Guang Xiang, Jason Hong, Carolyn P Rose, and Lorrie Cranor. 2011. Cantina+: A feature-rich machine learning framework for detecting phishing web sites. ACM Transactions on Information and System Security (TISSEC) 14, 2 (2011), 21.
[41]
Yue Zhang, Serge Egelman, Lorrie Cranor, and Jason Hong. 2006. Phinding phish: Evaluating anti-phishing tools. ISOC.

Cited By

View all
  • (2024)Cognition in Social Engineering Empirical Research: A Systematic Literature ReviewACM Transactions on Computer-Human Interaction10.1145/363514931:2(1-55)Online publication date: 29-Jan-2024
  • (2024)Exploring the evidence for email phishing trainingComputers and Security10.1016/j.cose.2023.103695139:COnline publication date: 16-May-2024
  • (2023)Checking, nudging or scoring? evaluating e-mail user security tools76Proceedings of the Nineteenth USENIX Conference on Usable Privacy and Security10.5555/3632186.3632190(57-76)Online publication date: 7-Aug-2023
  • Show More Cited By

Index Terms

  1. Use of Phishing Training to Improve Security Warning Compliance: Evidence from a Field Experiment

      Recommendations

      Comments

      Information & Contributors

      Information

      Published In

      cover image ACM Other conferences
      HoTSoS: Proceedings of the Hot Topics in Science of Security: Symposium and Bootcamp
      April 2017
      99 pages
      ISBN:9781450352741
      DOI:10.1145/3055305
      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

      In-Cooperation

      • National Security Agency: National Security Agency
      • Vanderbilt University: Vanderbilt University
      • University of Maryland: University of Maryland

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      Published: 04 April 2017

      Permissions

      Request permissions for this article.

      Check for updates

      Author Tags

      1. Active Warning
      2. Field Study
      3. Phishing

      Qualifiers

      • Research-article
      • Research
      • Refereed limited

      Conference

      HoTSoS '17
      HoTSoS '17: Symposium and Bootcamp
      April 4 - 5, 2017
      MD, Hanover, USA

      Acceptance Rates

      HoTSoS Paper Acceptance Rate 9 of 17 submissions, 53%;
      Overall Acceptance Rate 34 of 60 submissions, 57%

      Contributors

      Other Metrics

      Bibliometrics & Citations

      Bibliometrics

      Article Metrics

      • Downloads (Last 12 months)102
      • Downloads (Last 6 weeks)8
      Reflects downloads up to 18 Jan 2025

      Other Metrics

      Citations

      Cited By

      View all
      • (2024)Cognition in Social Engineering Empirical Research: A Systematic Literature ReviewACM Transactions on Computer-Human Interaction10.1145/363514931:2(1-55)Online publication date: 29-Jan-2024
      • (2024)Exploring the evidence for email phishing trainingComputers and Security10.1016/j.cose.2023.103695139:COnline publication date: 16-May-2024
      • (2023)Checking, nudging or scoring? evaluating e-mail user security tools76Proceedings of the Nineteenth USENIX Conference on Usable Privacy and Security10.5555/3632186.3632190(57-76)Online publication date: 7-Aug-2023
      • (2023)"To do this properly, you need more resources"Proceedings of the 32nd USENIX Conference on Security Symposium10.5555/3620237.3620467(4105-4122)Online publication date: 9-Aug-2023
      • (2023)Phishing in Social Media: Investigating Training Techniques on Instagram ShopProceedings of the Human Factors and Ergonomics Society Annual Meeting10.1177/2169506723119258867:1(1850-1855)Online publication date: 7-Nov-2023
      • (2023)SMiShing Attack Vector: Surveying End-User Behavior, Experience, and KnowledgeProceedings of the Human Factors and Ergonomics Society Annual Meeting10.1177/2169506723119219367:1(1911-1915)Online publication date: 18-Oct-2023
      • (2023)A Taxonomy of SETA Methods and Linkage to Delivery PreferencesACM SIGMIS Database: the DATABASE for Advances in Information Systems10.1145/3631341.363134854:4(107-133)Online publication date: 30-Oct-2023
      • (2023)Phishing to improve detectionProceedings of the 2023 European Symposium on Usable Security10.1145/3617072.3617121(334-343)Online publication date: 16-Oct-2023
      • (2023)Influence of URL Formatting on Users' Phishing URL DetectionProceedings of the 2023 European Symposium on Usable Security10.1145/3617072.3617111(318-333)Online publication date: 16-Oct-2023
      • (2023)Let warnings interrupt the interaction and explain: designing and evaluating phishing email warningsExtended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544549.3585802(1-6)Online publication date: 19-Apr-2023
      • Show More Cited By

      View Options

      Login options

      View options

      PDF

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      Media

      Figures

      Other

      Tables

      Share

      Share

      Share this Publication link

      Share on social media