Take a look at the new version of this page: [ beta version ]. Tell us what you think.

ABSTRACT

Manual accessibility evaluation plays an important role in validating the accessibility of Web pages. This role has become increasingly critical with the advent of the Web Content Accessibility Guidelines (WCAG) 2.0 and their reliance on user evaluation to validate certain conformance measures. However, the role of expertise, in such evaluations, is unknown and has not previously been studied. This paper sets out to investigate the interplay between expert and non-expert evaluation by conducting a Barrier Walkthrough (BW) study with 19 expert and 51 non-expert judges. The BW method provides an evaluation framework that can be used to manually assess the accessibility of Web pages for different user groups including motor impaired, hearing impaired, low vision, cognitive impaired, etc. We conclude that the level of expertise is an important factor in the quality of accessibility evaluation of Web pages. Expert judges spent significantly less time than non-experts; rated themselves as more productive and confident than non-experts; and ranked and rated pages differently against each type of disability. Finally, both effectiveness and reliability of the expert judges are significantly higher than non-expert judges.

REFERENCES

• 1: G. Brajnik. Barrier walkthrough: Heuristic evaluation guided by accessibility barriers. http://users.dimi.uniud.it/giorgio.brajnik/projects/bw/bw.html.
• 2: G. Brajnik. Web accessibility testing: When the method is the culprit. In In Proc. of ICCHP 2006, 2006.
• 3: B. Caldwell, M. Cooper, L.G. Reid, and G. Vanderheiden. Web Content Accessibility Guidelines 2.0 (WCAG 2.0). W3C, 2008. http://www.w3.org/TR/WCAG20/.
• 4: M.B. Catani and D.W. Biers. Usability evaluation and prototype fidelity: Users and usability professionals. In Proceedings of the Human Factors and Ergonomics Society 42nd Annual Meeting, 1998.
• 5: Julien Cegarra , Jean-Michel Hoc, Cognitive styles as an explanation of experts' individual differences: A case study in computer-assisted troubleshooting diagnosis, International Journal of Human-Computer Studies, v.64 n.2, p.123-136, February 2006 [doi>10.1016/j.ijhcs.2005.06.003]
• 6: G. Cockton and A. Woolrych. Understanding inspection methods: lessons from an assessment of heuristic evaluation. In People&Computers XV, pages 171--192. Springer, 2001.
• 7: DRC. Formal investigation report: web accessibility. Disability Rights Commission, www.drc-gb.org, April 2004. Visited Jan. 2006.
• 8: Wayne D. Gray , Marilyn C. Salzman, Damaged merchandise? a review of experiments that compare usability evaluation methods, Human-Computer Interaction, v.13 n.3, p.203-261, September 1998 [doi>10.1207/s15327051hci1303_2]
• 9: T. S. Andre H. R. Hartson and R. C. Williges. Criteria for evaluating usability evaluation methods. Int. Journal of Human-Computer Interaction, 15(1):145--183, 2003.
• 10: S.L. Henry and M. Grossnickle. Just Ask: Accessibility in the User-Centered Design Process. Georgia Tech Research Corporation, Atlanta, Georgia, USA, 2004.
• 11: Morten Hertzum , Niels Ebbe Jacobsen, The Evaluator Effect during First-Time Use of the Cognitive Walkthrough Technique, Proceedings of HCI International (the 8th International Conference on Human-Computer Interaction) on Human-Computer Interaction: Ergonomics and User Interfaces-Volume I, p.1063-1067, August 22-26, 1999
• 12: Morten Hertzum , Niels Ebbe Jacobsen , Rolf Molich, Usability inspections by groups of specialists: perceived agreement in spite of disparate observations, CHI '02 extended abstracts on Human factors in computing systems, April 20-25, 2002, Minneapolis, Minnesota, USA [doi>10.1145/506443.506534]
• 13: M. Hertzum and N. Ebbe Jacobsen. The evaluator effect: A chilling fact about usability evaluation methods. International Journal of Human Computer Interaction, 13(4), 2001.
• 14: K. Hornbeak and E. Frokjear. A study of the evaluator effect in usability testing. Human-Computer Interaction,, 23(3):251 -- 277, 2008.
• 15: Melody Y. Ivory , Marti A Hearst, The state of the art in automating usability evaluation of user interfaces, ACM Computing Surveys (CSUR), v.33 n.4, p.470-516, December 2001 [doi>10.1145/503112.503114]
• 16: Niels Ebbe Jacobsen , Morten Hertzum , Bonnie E. John, The evaluator effect in usability tests, CHI 98 conference summary on Human factors in computing systems, p.255-256, April 18-23, 1998, Los Angeles, California, United States [doi>10.1145/286498.286737]
• 17: T. Lang. Comparing website accessibility evaluation methods and learnings from usability evaluation methods. http://www.peakusability.com.au, Visited May 2008, 2003.
• 18: Chen Ling , Gavriel Salvendy, Effect of evaluators' cognitive style on heuristic evaluation: Field dependent and field independent evaluators, International Journal of Human-Computer Studies, v.67 n.4, p.382-393, April, 2009 [doi>10.1016/j.ijhcs.2008.11.002]
• 19: Jennifer Mankoff , Holly Fait , Tu Tran, Is your web page accessible?: a comparative study of methods for assessing web page accessibility for the blind, Proceedings of the SIGCHI conference on Human factors in computing systems, April 02-07, 2005, Portland, Oregon, USA [doi>10.1145/1054972.1054979]
• 20: Jakob Nielsen, Finding usability problems through heuristic evaluation, Proceedings of the SIGCHI conference on Human factors in computing systems, p.373-380, May 03-07, 1992, Monterey, California, United States [doi>10.1145/142750.142834]
• 21: Helen Petrie , Omar Kheir, The relationship between accessibility and usability of websites, Proceedings of the SIGCHI conference on Human factors in computing systems, April 28-May 03, 2007, San Jose, California, USA [doi>10.1145/1240624.1240688]
• 22: I. Curson S. Butler E. Kindlund D. Miller R. Molich, N. Bevan and J. Kirakowski. Comparative evaluation of usability tests. In In Proc. of the Usability Professionals Association Conference, 1998.
• 23: A. Sears. Heuristic walkthroughs: Finding the problems without the noise. International Journal of Human-Computer Interaction, 9:213 -- 234, 1997.
• 24: Patrick E. Shrout and Joseph L. Fleiss. Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86(2):420--428, Mar 1979.
• 25: A. Woolrych and G. Cockton. Assessing heuristic evaluation: mind the quality, not just the percentages. In In Proc. of HCI 2000, pages 35--36, 2000.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Evaluation/methodology

Additional Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: User-centered design; Interaction styles (e.g., commands, menus, forms, direct manipulation); Input devices and strategies (e.g., mouse, touchscreen)

  K. Computing Milieux
  K.4 COMPUTERS AND SOCIETY
      K.4.2 Social Issues
          Subjects: Handicapped persons/special needs**; Assistive technologies for persons with disabilities

General Terms:
Experimentation, Human Factors

Keywords:
evaluation, expertise, guideline, web accessibility