Is your web page accessible?: a comparative study of methods for assessing web page accessibility for the blind

ABSTRACT

Web access for users with disabilities is an important goal and challenging problem for web content developers and designers. This paper presents a comparison of different methods for finding accessibility problems affecting users who are blind. Our comparison focuses on techniques that might be of use to Web developers without accessibility experience, a large and important group that represents a major source of inaccessible pages. We compare a laboratory study with blind users to an automated tool, expert review by web designers with and without a screen reader, and remote testing by blind users. Multiple developers, using a screen reader, were most consistently successful at finding most classes of problems, and tended to find about 50% of known problems. Surprisingly, a remote study with blind users was one of the least effective methods. All of the techniques, however, had different, complementary strengths and weaknesses.

REFERENCES

• 1: Bobby Worldwide, http://bobby.watchfire.com/bobby/html/en/index.jsp
• 2: Bureau of the Census, "Survey of Income and Program Participation," 1994--95, http://www.census.go v/hhes/www/disable/dissipp.html
• 3: Joe Clark, Building Accessible Websites, Pearson Education, 2001
• 4: Gilbert Cockton , Darryn Lavery , Alan Woolrych, Inspection-based evaluations, The human-computer interaction handbook: fundamentals, evolving technologies and emerging applications, Lawrence Erlbaum Associates, Inc., Mahwah, NJ, 2002
• 5: Cockton, G. and Woolrych, A., "Understanding inspection methods: Lessons from an assessment of heuristic evaluation," In A. Blandford and J. Vanderdonckt, (Eds.), People & Computers XV, Springer-Verlag, pp. 171--192, 2001.
• 6: Colwell, C. and Petrie, H., "Evaluation of guidelines for designing accessible web content," In C. Bühler & H. Knops (Eds), Assistive technology on the threshold of the new millennium, IOS press, 1999.
• 7: Coyne, K. P. and Nielsen, J., "Beyond ALT Text: Making the web easy to use for users with disabilities," Nielsen, Norman Group, October, 2001, Available at: http://www.nngroup.com/reports/accessibility/
• 8: Coyne, K. P. and Nielsen, J., "How to conduct usability evaluations for accessibility: Methodology guidelines for testing websites and intranets with users who use assistive technology," Nielsen Norman Group, October, 2001, Available at: http://www.nngroup.com/reports/accessibility/testing/
• 9: Diaper, D. and Worman, L., "Two falls out of three in automated accessibility assessment of world wide web sites: A-prompt vs. Bobby," In Johnson, P. and Palanque, P. (Eds.), People and Computers XVII, Springer-Verlag.
• 10: Gray, W.D, and Salzman, M.C., "Damaged merchandise? A review of experiments that compare usability evaluation methods," Human-Computer Interaction, 13(3):203--261, 1998.
• 11: Hartson, H. R., Andre, T. S. and Williges, R. C., "Criteria for evaluating usability evaluation methods," International Journal of Human Computer Interaction, 13(4):373--410, 2001.
• 12: Hertzum, M. and Jacobsen, N. E., "The Evaluator Effect: A Chilling Fact about Usability Evaluation Methods," International Journal of Human Computer Interaction, 13(4):412--443, 2001.
• 13: Ivory, M. and Chevalier, A., "A Study of Automated Web Site Evaluation Tools," Technical Report UW-CSE-02-10-01, University of Washington, Department of Computer Science and Engineering, 2002.
• 14: LIFT, http://www.usablenet.com/
• 15: National Center for Health Statistics, "National health interview survey - disability supplement," 1994 and 1995.
• 16: Jakob Nielsen , Rolf Molich, Heuristic evaluation of user interfaces, Proceedings of the SIGCHI conference on Human factors in computing systems: Empowering people, p.249-256, April 01-05, 1990, Seattle, Washington, United States [doi>10.1145/97243.97281]
• 17: Mike G. Paciello , Michael G. Paciello, Web Accessibility for People with Disabilities, C M P Books, 2000
• 18: Petrie, H. and Colwell, C., "Tool to assist authors in creating accessible web pages," In Proc. of NTEVH 98: Telematics in the education of the visually handicapped, 1998. http://www.snv.jussieu.fr/inova/publi/ntevh/tools.htm
• 19: Murray Rowan , Peter Gregor , David Sloan , Paul Booth, Evaluating web resources for disability access, Proceedings of the fourth international ACM conference on Assistive technologies, p.80-84, November 13-15, 2000, Arlington, Virginia, United States [doi>10.1145/354324.354346]
• 20: Sears, A., "Heuristic Walkthroughs: Finding the problems without the noise," International Journal of Human-Computer Interaction, 9:213--234.
• 21: Shneiderman, B. and Plaisant, C., Designing the user interface, 4th Edition, Pearson Education.
• 22: David Sloan , Peter Gregor , Murray Rowan , Paul Booth, Accessible accessibility, Proceedings on the 2000 conference on Universal Usability, p.96-101, November 16-17, 2000, Arlington, Virginia, United States [doi>10.1145/355460.355480]
• 23: Terry Sullivan , Rebecca Matson, Barriers to use: usability and content accessibility on the Web's most popular sites, Proceedings on the 2000 conference on Universal Usability, p.139-144, November 16-17, 2000, Arlington, Virginia, United States [doi>10.1145/355460.355549]
• 24: Jim Thatcher , Cynthia Waddell, Constructing Accessible Websites, APress L. P., 2003
• 25: Woolrych, A. and Cockton, G., "Assessing Heuristic Evaluation: Mind the quality, not just the percentages," in Proc. of HCI 2000, pp. 35--36, 2000.
• 26: W3C Markup Validation Service, http://validator.w3.org/
• 27: W3C Web Content Accessibility Guidelines 1.0, http://www.w3.org/TR/WAI-WEBCONTENT

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Evaluation/methodology

Additional Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.4 Hypertext/Hypermedia
          Subjects: User issues

  K. Computing Milieux
  K.4 COMPUTERS AND SOCIETY
      K.4.2 Social Issues
          Subjects: Assistive technologies for persons with disabilities

Keywords:
assistive technologies, disability, evaluation, web accessibility