You've been warned: an empirical study of the effectiveness of web browser phishing warnings

ABSTRACT

Many popular web browsers are now including active phishing warnings after previous research has shown that passive warnings are often ignored. In this laboratory study we examine the effectiveness of these warnings and examine if, how, and why they fail users. We simulated a spear phishing attack to expose users to browser warnings. We found that 97% of our sixty participants fell for at least one of the phishing messages that we sent them. However, we also found that when presented with the active warnings, 79% of participants heeded them, which was not the case for the passive warning that we tested---where only one participant heeded the warnings. Using a model from the warning sciences we analyzed how users perceive warning messages and offer suggestions for creating more effective warning messages within the phishing context.

REFERENCES

• 1: Amer, T. S., and Maris, J. B. Signal words and signal icons in application control and information technology exception messages -- hazard matching and habituation effects. Tech. Rep. Working Paper Series-06-05, Northern Arizona University, Flagstaff, AZ, October 2006.
• 2: Bank of America. How Bank of America SiteKey Works for Online Banking Security. http://www.bankofamerica.com/privacy/sitekey/, 2007.
• 3: José Carlos Brustoloni , Ricardo Villamarín-Salomón, Improving security decisions with polymorphic and audited dialogs, Proceedings of the 3rd symposium on Usable privacy and security, July 18-20, 2007, Pittsburgh, Pennsylvania [doi>10.1145/1280680.1280691]
• 4: Certification Authority/Browser Forum. Extended validation ssl certificates, Accessed: July 27, 2007. http://cabforum.org/.
• 5: Lorrie Faith Cranor, What do they "indicate?": evaluating security and privacy indicators, interactions, v.13 n.3, May + June 2006 [doi>10.1145/1125864.1125890]
• 6: Rachna Dhamija , J. D. Tygar, The battle against phishing: Dynamic Security Skins, Proceedings of the 2005 symposium on Usable privacy and security, p.77-88, July 06-08, 2005, Pittsburgh, Pennsylvania [doi>10.1145/1073001.1073009]
• 7: Rachna Dhamija , J. D. Tygar , Marti Hearst, Why phishing works, Proceedings of the SIGCHI conference on Human Factors in computing systems, April 22-27, 2006, Montréal, Québec, Canada [doi>10.1145/1124772.1124861]
• 8: Julie S. Downs , Mandy B. Holbrook , Lorrie Faith Cranor, Decision strategies and susceptibility to phishing, Proceedings of the second symposium on Usable privacy and security, July 12-14, 2006, Pittsburgh, Pennsylvania [doi>10.1145/1143120.1143131]
• 9: Dinei Florencio , Cormac Herley, A large-scale study of web password habits, Proceedings of the 16th international conference on World Wide Web, May 08-12, 2007, Banff, Alberta, Canada [doi>10.1145/1242572.1242661]
• 10: B. J. Fogg , Jonathan Marshall , Othman Laraki , Alex Osipovich , Chris Varma , Nicholas Fang , Jyoti Paul , Akshay Rangnekar , John Shon , Preeti Swani , Marissa Treinen, What makes Web sites credible?: a report on a large quantitative study, Proceedings of the SIGCHI conference on Human factors in computing systems, p.61-68, March 2001, Seattle, Washington, United States [doi>10.1145/365024.365037]
• 11: Gartner, Inc. Gartner Says Number of Phishing E-Mails Sent to U.S. Adults Nearly Doubles in Just Two Years. http://www.gartner.com/it/page.jsp?id=498245, November 9 2006.
• 12: Hellier, E., Wright, D. B., Edworthy, J., and Newstead, S. On the stability of the arousal strength of warning signal words. Applied Cognitive Psychology 14 (2000), 577--592.
• 13: Jackson, C., Simon, D., Tan, D., and Barth, A. An evaluation of extended validation and picture-in-picture phishing atttacks. In Proceedings of the 2007 Usable Security (USEC'07) Workshop (February 2007). http://www.usablesecurity.org/papers/jackson.pdf.
• 14: Ponnurangam Kumaraguru , Yong Rhee , Alessandro Acquisti , Lorrie Faith Cranor , Jason Hong , Elizabeth Nunge, Protecting people from phishing: the design and evaluation of an embedded training email system, Proceedings of the SIGCHI conference on Human factors in computing systems, April 28-May 03, 2007, San Jose, California, USA [doi>10.1145/1240624.1240760]
• 15: Moore, T., and Clayton, R. An empirical analysis of the current state of phishing attack and defence. In Proceedings of the 2007 Workshop on The Economics of Information Security (WEIS2007) (May 2007). http://www.cl.cam.ac.uk/ twm29/weis07-phishing.pdf.
• 16: Oberheide, J. Google safe browsing, November 6 2006. http://jon.oberheide.org/blog/2006/11/13/google-safe-browsing/.
• 17: OpenDNS. PhishTank Annual Report. http://www.phishtank.com/, October 2007.
• 18: Refsnes Data. Browser statistics, Accessed: April 4, 2007. http://www.w3schools.com/browsers/browsers_stats.asp.
• 19: Stuart E. Schechter , Rachna Dhamija , Andy Ozment , Ian Fischer, The Emperor's New Security Indicators, Proceedings of the 2007 IEEE Symposium on Security and Privacy, p.51-65, May 20-23, 2007 [doi>10.1109/SP.2007.35]
• 20: Steve Sheng , Bryant Magnien , Ponnurangam Kumaraguru , Alessandro Acquisti , Lorrie Faith Cranor , Jason Hong , Elizabeth Nunge, Anti-Phishing Phil: the design and evaluation of a game that teaches people not to fall for phish, Proceedings of the 3rd symposium on Usable privacy and security, July 18-20, 2007, Pittsburgh, Pennsylvania [doi>10.1145/1280680.1280692]
• 21: Wogalter, M. S. Communication-Human Information Processing (C-HIP) Model. In Handbook of Warnings, M. S. Wogalter, Ed. Lawrence Erlbaum Associates, 2006, pp. 51--61.
• 22: Min Wu , Robert C. Miller, Fighting phishing at the user interface, Massachusetts Institute of Technology, Cambridge, MA, 2006
• 23: Min Wu , Robert C. Miller , Simson L. Garfinkel, Do security toolbars actually prevent phishing attacks?, Proceedings of the SIGCHI conference on Human Factors in computing systems, April 22-27, 2006, Montréal, Québec, Canada [doi>10.1145/1124772.1124863]
• 24: Zishuang (Eileen) Ye , Sean Smith, Trusted Paths for Browsers, Proceedings of the 11th USENIX Security Symposium, p.263-279, August 05-09, 2002
• 25: Ka-Ping Yee , Kragen Sitaker, Passpet: convenient password management and phishing protection, Proceedings of the second symposium on Usable privacy and security, July 12-14, 2006, Pittsburgh, Pennsylvania [doi>10.1145/1143120.1143126]
• 26: Zhang, Y., Egelman, S., Cranor, L. F., and Hong, J. Phinding phish: Evaluating anti-phishing tools. In Proceedings of the 14th Annual Network & Distributed System Security Symposium (NDSS 2007) (28th February - 2nd March, 2007). http://lorrie.cranor.org/pubs/toolbars.html.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.1 MODELS AND PRINCIPLES
      H.1.2 User/Machine Systems

Additional Classification:
  D. Software
  D.4 OPERATING SYSTEMS
      D.4.6 Security and Protection

  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)

General Terms:
Human Factors

Keywords:
mental models, phishing, usable privacy and security, warning messages