Validity and reliability of web accessibility guidelines

ABSTRACT

Although widely used,Web Content Accessibility Guidelines (WCAG) have not been studied from the viewpoint of their validity and reliability. WCAG 2.0 explicitly claim that they are based on "testable" criteria, but no scientific evidence exists that this is actually the case. Validity (how well all and only the true problems can be identified) and reliability (the extent to which different evaluations of the same page lead to same results) are key factors for quality of accessibility evaluation methods. They need to be well studied and understood for methods, and guidelines, that are expected to have a major impact.

This paper presents an experiment aimed at finding out what is the validity and reliability of different checkpoints taken from WCAG 1.0 and WCAG 2.0. The experiment employed 35 young web developers with some knowledge on web accessibility. Although this is a small-scale experiment, unlikely to provide definite and general answers, results un-equivocally show that with respect to the kind of evaluators chosen in the experiment, checkpoints in general fare very low in terms of reliability, and that from this perspective WCAG 2.0 are not an improvement over WCAG 1.0.

REFERENCES

• 1: G. Brajnik. Comparing accessibility evaluation tools: a method for tool effectiveness. Int. Journal on Universal Access in the Information Society, 3(3-4):252--263, Oct. 2004.
• 2: G. Brajnik. Beyond conformance: the role of accessibility evaluation methods. In S. Hartmann, X. Zhou, and M. Kirchberg, editors, WISE 2008: 9th Int. Conference on Web Information Systems Engineering - 2nd International Workshop on Web Usability and Accessibility IWWUA08, LNCS 5176, pages 63--80, Auckland, New Zealand, Sept. 2008. Springer-Verlag. Keynote speech.
• 3: G. Brajnik. A comparative test of web accessibility evaluation methods. In S. Harper, editor, 10th Int. ACM SIGACCESS Conference on Computers and Accessibility, ASSETS, Halifax, Canada, Oct. 2008. ACM Press.
• 4: G. Brajnik. Towards a sustainable accessibility. In Accessible Design in the Digital World, York, UK, September 22-24 2008. http://www.addw08.orgYork University.
• 5: G. Brajnik, A. Mulas, and C. Pitton. Effects of sampling methods on web accessibility evaluations. In S. Trewin and E. Pontelli, editors, 9th Int. ACM SIGACCESS Conference on Computers and Accessibility, ASSETS, Tempe, AZ, Oct. 2007. ACM Press.
• 6: DRC. Formal investigation report: web accessibility. Disability Rights Commission, www.drc-gb.org/publicationsandreports/report.asp, April 2004. Visited Jan. 2006.
• 7: S. Henry and M. Grossnickle. Just Ask: Accessibility in the User-Centered Design Process. Georgia Tech Research Corporation, Atlanta, Georgia, USA, 2004. On-line book: www.UIAccess.com/AccessUCD.
• 8: M. Hertzum and N. Jacobsen. The evaluator effect: a chilling fact about usability evaluation methods. Int. Journal of Human-Computer Interaction, 1(4):421--443, 2001.
• 9: K. Hornbæk and E. Frøkjær. A study of the evaluator effect in usability testing. Human-Computer Interaction, 23:251--277, 2008.
• 10: Italian Government. Technical assessment and technical accessibility requirements of internet technology-based applications. http://www.pubbliaccesso.it/normative/DM080705-A-en.htm, July 2005.
• 11: B. Kelly, D. Sloan, S. Brown, J. Seale, H. Petrie, P. Lauke, and S. Ball. Accessibility 2.0: people, policies and processes. In W4A '07: Proc. of the 2007 international cross-disciplinary conference on Web accessibility (W4A), pages 138--147, New York, NY, USA, 2007. ACM.
• 12: T. Lang. Comparing website accessibility evaluation methods and learnings from usability evaluation methods. http://www.peakusability.com.au/about-us/pdf/website_accessibility.pdf, Visited May 2008, 2003.
• 13: J. Mankoff, H. Fait, and T. Tran. Is your web page accessible?: a comparative study of methods for assessing web page accessibility for the blind. In CHI 2005: Proc. of the SIGCHI conference on Human factors in computing systems, pages 41--50, New York, NY, USA, 2005. ACM.
• 14: H. Petrie, F. Hamilton, N. King, and P. Pavan. Remote usability evaluations with disabled people. In CHI 2006: Proc. of the SIGCHI conference on Human factors in computing systems, pages 1133--1141, New York, NY, USA, 2006. ACM.
• 15: H. Petrie and O. Kheir. The relationship between accessibility and usability of websites. In Proc. CHI 2007, pages 397--406, San Jose, CA, USA, 2007. ACM.
• 16: G. Sampson-Wild. Testability costs too much. http://www.alistapart.com/articles/testability, June 2007.
• 17: J. Slatin and S. Rush. Maximum Accessibility: Making Your Web Site More Usable for Everyone. Addison-Wesley, 2003.
• 18: J. Smith. Testability in WCAG 2.0. http://webaim.org/blog/wcag-2-testability, June 2007.
• 19: J. Thatcher, M. Burks, C. Heilmann, S. Henry, A. Kirkpatrick, P. Lauke, B. Lawson, B. Regan, R. Rutter, M. Urban, and C. Waddell. Web Accessibility: Web Standards and Regulatory Compliance. FriendsofED, 2006.
• 20: U.S. Dept. of Justice. Section 508 of the Rehabilitation Act. www.access-board.gov/sec508/guide/1194.22.htm, 2001.
• 21: W3C/WAI. Requirements for WCAG 2.0 Checklists and Techniques. http://www.w3.org/TR/2003/WD-wcag2-tech-req-20030207, 2003.
• 22: W3C/WAI. Conformance evaluation of web sites for accessibility. www.w3.org/WAI/eval/conformance.html, 2008. Visited May 2008.
• 23: W3C/WAI. Web content accessibility guidelines (wcag) 2.0. World Wide Web Consortium - Web Accessibility Initiative, www.w3.org/TR/WCAG20, December 2008.
• 24: Y. Yesilada, G. Brajnik, and S. Harper. How Much Does Expertise Matter? A Barrier Walkthrough Study with Experts and Non-Experts. In Proc. of 11th Int. ACM SIGACCESS Conference on Computers and Accessibility - ASSETS 2009, Pittsburgh, PA, Oct. 2009. ACM SIGACCESS.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Evaluation/methodology

Additional Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Input devices and strategies (e.g., mouse, touchscreen); User-centered design

  K. Computing Milieux
  K.4 COMPUTERS AND SOCIETY
      K.4.2 Social Issues
          Subjects: Handicapped persons/special needs**; Assistive technologies for persons with disabilities

General Terms:
Experimentation, Human Factors, Measurement, Standardization

Keywords:
accessibility evaluation evaluation, web accessibility guidelines