How much does expertise matter?

Subscribe (Full Service)


	Search: The ACM Digital Library The Guide

	Feedback

How much does expertise matter?: a barrier walkthrough study with experts and non-experts

Full text

Pdf (552 KB)

Source

ACM SIGACCESS Conference on Computers and Accessibility archive
Proceedings of the 11th international ACM SIGACCESS conference on Computers and accessibility table of contents

Pittsburgh, Pennsylvania, USA

SESSION: Web accessibility II table of contents

Pages 203-210

Year of Publication: 2009

ISBN:978-1-60558-558-1

Authors

Yeliz Yesilada	University of Manchester, Manchester, United Kingdom
Giorgio Brajnik	University of Udine, Udine, Italy
Simon Harper	University of Manchester, Manchester, United Kingdom

Sponsor

SIGACCESS: ACM Special Interest Group on Accessible Computing

Publisher

ACM New York, NY, USA

Bibliometrics

Downloads (6 Weeks): 17, Downloads (12 Months): 17, Citation Count: 0

Additional Information:

abstract references index terms

Tools and Actions:	Request Permissions Review this Article Save this Article to a Binder Display Formats: BibTeX EndNote ACM Ref

DOI Bookmark:	Use this link to bookmark this Article: http://doi.acm.org/10.1145/1639642.1639678 What is a DOI?

ABSTRACT

Manual accessibility evaluation plays an important role in validating the accessibility of Web pages. This role has become increasingly critical with the advent of the Web Content Accessibility Guidelines (WCAG) 2.0 and their reliance on user evaluation to validate certain conformance measures. However, the role of expertise, in such evaluations, is unknown and has not previously been studied. This paper sets out to investigate the interplay between expert and non-expert evaluation by conducting a Barrier Walkthrough (BW) study with 19 expert and 51 non-expert judges. The BW method provides an evaluation framework that can be used to manually assess the accessibility of Web pages for different user groups including motor impaired, hearing impaired, low vision, cognitive impaired, etc. We conclude that the level of expertise is an important factor in the quality of accessibility evaluation of Web pages. Expert judges spent significantly less time than non-experts; rated themselves as more productive and confident than non-experts; and ranked and rated pages differently against each type of disability. Finally, both effectiveness and reliability of the expert judges are significantly higher than non-expert judges.

REFERENCES

Note: OCR errors may be found in this Reference List extracted from the full text article. ACM has opted to expose the complete List rather than only correct and linked references.

	1	G. Brajnik. Barrier walkthrough: Heuristic evaluation guided by accessibility barriers. http://users.dimi.uniud.it/giorgio.brajnik/projects/bw/bw.html.
	2	G. Brajnik. Web accessibility testing: When the method is the culprit. In In Proc. of ICCHP 2006, 2006.
	3	B. Caldwell, M. Cooper, L.G. Reid, and G. Vanderheiden. Web Content Accessibility Guidelines 2.0 (WCAG 2.0). W3C, 2008. http://www.w3.org/TR/WCAG20/.
	4	M.B. Catani and D.W. Biers. Usability evaluation and prototype fidelity: Users and usability professionals. In Proceedings of the Human Factors and Ergonomics Society 42nd Annual Meeting, 1998.
	5	J. Cegarra and J. Hoc. Cognitive styles as an explanation of experts' individual differences: A case study in computer-assisted troubleshooting diagnosis. Int. J. Hum.-Comput. Stud., 64(2):123--136, 2006.
	6	G. Cockton and A. Woolrych. Understanding inspection methods: lessons from an assessment of heuristic evaluation. In People & Computers XV, pages 171--192. Springer, 2001.
	7	DRC. Formal investigation report: web accessibility. Disability Rights Commission, www.drc-gb.org, April 2004. Visited Jan. 2006.
	8	W. Gray and M. Salzman. Damaged merchandise: a review of experiments that compare usability evaluation methods. Human -Computer Interaction, 13(3):203--261, 1998.
	9	T. S. Andre H. R. Hartson and R. C. Williges. Criteria for evaluating usability evaluation methods. Int. Journal of Human-Computer Interaction, 15(1):145--183, 2003.
	10	S.L. Henry and M. Grossnickle. Just Ask: Accessibility in the User-Centered Design Process. Georgia Tech Research Corporation, Atlanta, Georgia, USA, 2004.
	11	M. Hertzum and N. Jacobsen. The evaluator effect during first-time use of the cognitive walkthrough technique. In Proceedings of HCI International on Human-Computer Interaction: Ergonomics and User Interfaces-Volume I, pages 1063--1067, 1999.
	12	M. Hertzum, N. Jacobsen, and R. Molich. Usability inspections by groups of specialists: perceived agreement in spite of disparate observations. In CHI '02 extended abstracts, pages 662--663. ACM, 2002.
	13	M. Hertzum and N. Ebbe Jacobsen. The evaluator effect: A chilling fact about usability evaluation methods. International Journal of Human Computer Interaction, 13(4), 2001.
	14	K. Hornbeak and E. Frokjear. A study of the evaluator effect in usability testing. Human-Computer Interaction,, 23(3):251 -- 277, 2008.
	15	M. Ivory and M. Hearst. The state of the art in automating usability evaluation of user interfaces. ACM Computer Survey, 33(4):470--516, 2001.
	16	N. Jacobsen, M. Hertzum, and B. John. The evaluator effect in usability tests. In CHI '98, pages 255--256. ACM, 1998.
	17	T. Lang. Comparing website accessibility evaluation methods and learnings from usability evaluation methods. http://www.peakusability.com.au, Visited May 2008, 2003.
	18	C. Ling and G. Salvendy. Effect of evaluators' cognitive style on heuristic evaluation: Field dependent and field independent evaluators. Int. J. Hum.-Comput. Stud., 67(4):382--393, 2009.
	19	J. Mankoff, H. Fait, and T. Tran. Is your web page accessible?: a comparative study of methods for assessing web page accessibility for the blind. In CHI'05, pages 41--50. ACM, 2005.
	20	J. Nielsen. Finding usability problems through heuristic evaluation. In CHI '92, pages 373--380. ACM, 1992.
	21	H. Petrie and O. Kheir. The relationship between accessibility and usability of websites. In Proc. CHI 2007, pages 397--406, San Jose, CA, USA, 2007. ACM.
	22	I. Curson S. Butler E. Kindlund D. Miller R. Molich, N. Bevan and J. Kirakowski. Comparative evaluation of usability tests. In In Proc. of the Usability Professionals Association Conference, 1998.
	23	A. Sears. Heuristic walkthroughs: Finding the problems without the noise. International Journal of Human-Computer Interaction, 9:213 -- 234, 1997.
	24	Patrick E. Shrout and Joseph L. Fleiss. Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86(2):420--428, Mar 1979.
	25	A. Woolrych and G. Cockton. Assessing heuristic evaluation: mind the quality, not just the percentages. In In Proc. of HCI 2000, pages 35--36, 2000.