The state of the art in automating usability evaluation of user interfaces

ABSTRACT

Usability evaluation is an increasingly important part of the user interface design process. However, usability evaluation can be expensive in terms of time and human resources, and automation is therefore a promising way to augment existing approaches. This article presents an extensive survey of usability evaluation methods, organized according to a new taxonomy that emphasizes the role of automation. The survey analyzes existing techniques, identifies which aspects of usability evaluation automation are likely to be of use in future research, and suggests new ways to expand existing approaches to better support usability evaluation.

REFERENCES

Note: OCR errors may be found in this Reference List extracted from the full text article. ACM has opted to expose the complete List rather than only correct and linked references.

	1	ABELOW, D. 1993. Automating feedback on software product use. CASE Trends December, 15- 17.
	2	ADDY & ASSOCIATES. 2000. Dr. Watson version 4.0. Available at http://watson.addy.com/.
	3	Christopher Ahlberg , Ben Shneiderman, Visual information seeking: tight coupling of dynamic query filters with starfield displays, Proceedings of the SIGCHI conference on Human factors in computing systems: celebrating interdependence, p.313-317, April 24-28, 1994, Boston, Massachusetts, United States [doi>10.1145/191666.191775]
	4	Ghassan Al-Qaimari , Darren McRostie, KALDI: a computer-aided usability engineering tool for supporting testing and analysis of human-computer interaction, Proceedings of the third international conference on Computer-aided design of user interfaces, p.337-355, October 1999, Louvain-la-Neuve, Belgium
	5	ANDERSON, J. 1993. Rules of the Mind. Hillsdale, NJ: Lawrence Erlbaum Associates, Inc.
	6	ANDERSON, J. R. 1990. The Adaptive Character of Thought. Hillsdale, NJ: Lawrence Erlbaum Associates, Inc.
	7	BACHELDOR, B. 1999. Push for performance. Information Week September 20, 18-20.
	8	BALBO, S. 1995. Automatic evaluation of user interface usability: Dream or reality. In S. Balbo, Ed., Proceedings of the Queensland Computer- Human Interaction Symposium (Queensland, Australia, August). Bond University.
	9	BALBO, S. 1996. EMA: Automatic analysis mechanism for the ergonomic evaluation of user interfaces. Tech. Rep. 96/44 (August), CSIRO Division of Information Technology. Available at http: // www.cmis.csiro.au / sandrine.balbo / Ema/ ema tr/ema-tr.doc.
	10	Philip J. Barnard, Cognitive resources and the learning of human-computer dialogs, Interfacing thought: cognitive aspects of human-computer interaction, MIT Press, Cambridge, MA, 1987
	11	BARNARD,P.J.AND TEASDALE, J. D. 1991. Interacting cognitive subsystems: A systemic approach to cognitive-affective interaction and change. Cognition and Emotion 5, 1-39.
	12	Lynn K. Baumeister , Bonnie E. John , Michael D. Byrne, A comparison of tools for building GOMS models, Proceedings of the SIGCHI conference on Human factors in computing systems, p.502-509, April 01-06, 2000, The Hague, The Netherlands [doi>10.1145/332040.332485]
	13	BEARD,D.V.,SMITH,D.K.,AND DENELSBECK,K.M. 1996. Quick and dirty GOMS: A case study of computed tomography interpretation. Human- Computer Interaction 11, 2, 157-180.
	14	Victoria Bellotti, Implications of current design practice for the use of HCI techniques, Proceedings of the Fourth Conference of the British Computer Society on People and computers IV, p.13-34, October 1988, Univ. of Manchester, United Kingdom
	15	François Bodart , Anne-Marie Hennebert , Jean-Marie Leheureux , Jean Vanderdonckt, Towards a dynamic strategy for computer-aided visual placement, Proceedings of the workshop on Advanced visual interfaces, p.78-87, June 01-04, 1994, Bari, Italy [doi>10.1145/192309.192328]
	16	Jośe A. Borges , Israel Morales , Néstor J. Rodríguez, Guidelines for designing usable World Wide Web pages, Conference companion on Human factors in computing systems: common ground, p.277-278, April 13-18, 1996, Vancouver, British Columbia, Canada [doi>10.1145/257089.257320]
	17	Neil Bowers, Weblint: quality assurance for the World Wide Web, Proceedings of the fifth international World Wide Web conference on Computer networks and ISDN systems, p.1283-1290, May 1996, Paris, France
	18	BRAJNIK, G. 2000. Automatic web usability evaluation: Where is the limit? In Proceedings of the Sixth Conference on Human Factors & the Web (Austin, TX, June). Available at http://www.tri.sbc.com/hfweb/brajnik/hfwebbrajnik. html.
	19	Michael D. Byrne , Bonnie E. John , Neil S. Wehrle , David C. Crow, The tangled Web we wove: a taskonomy of WWW use, Proceedings of the SIGCHI conference on Human factors in computing systems: the CHI is the limit, p.544-551, May 15-20, 1999, Pittsburgh, Pennsylvania, United States [doi>10.1145/302979.303154]
	20	Michael D. Byrne , Scott D. Wood , James D. Foley , David E. Kieras , Piyawadee Noi Sukaviriya, Automating interface evaluation, Proceedings of the SIGCHI conference on Human factors in computing systems: celebrating interdependence, p.232-237, April 24-28, 1994, Boston, Massachusetts, United States [doi>10.1145/191666.191752]
	21	Stuart K. Card , Allen Newell , Thomas P. Moran, The Psychology of Human-Computer Interaction, Lawrence Erlbaum Associates, Inc., Mahwah, NJ, 1983
	22	CENTERLINE. 1999. QC/Replay. Available at http:// www.centerline.com/productline/qcreplay/qcre-play. html.
	23	CHAK, A. 2000. Usability tools: A useful start. Web Techniques (2000), August, 18-20. Available at http://www.webtechniques.com/archives/2000 /08/stratrevu/.
	24	Ed H. Chi , Peter Pirolli , James Pitkow, The scent of a site: a system for analyzing and predicting information scent, usage, and usability of a Web site, Proceedings of the SIGCHI conference on Human factors in computing systems, p.161-168, April 01-06, 2000, The Hague, The Netherlands [doi>10.1145/332040.332423]
	25	CLARK,D.AND DARDAILLER, D. 1999. Accessibility on the web: Evaluation & repair tools to make it possible. In Proceedings of the CSUN Technology and Persons with Disabilities Conference (Los Angeles, CA, March). Available at http://www.dinf.org/csun 99/session0030.html.
	26	COMBER, T. 1995. Building usable web pages: An HCI perspective. In R. Debreceny and A. Ellis, Eds., Proceedings of the First Australian World Wide Web Conference (Ballina, Australia, April), pp. 119-124. Ballina, Australia: Norsearch. Available at http://www. scu.edu.au/sponsored/ausweb/ausweb95/papers/ hypertext/comber/.
	27	COOPER, M. 1999. Universal design of a web site. In Proceedings of the CSUN Technology and Persons with Disabilities Conference (Los Angeles, CA, March). Available at http://www.dinf.org/csun 99/session0030.html.
	28	Joëlle Coutaz, Evaluation Techniques: Exploring the Intersection of HCI and Software Engineering, Proceedings of the Workshop on Software Engineering and Human-Computer Interaction, p.35-48, May 16-17, 1994
	29	John Cugini , Jean Scholtz, VISVIP: 3D Visualization of Paths through Web Sites, Proceedings of the 10th International Workshop on Database & Expert Systems Applications, p.259, September 01-03, 1999
	30	DE HAAN,G.,VAN DER VEER,G.C.,AND VAN VLIET,J.C. 1992. Formal modelling techniques in humancomputerF interaction. In G. C. van der Veer, S. Bagnara, and G. A. M. Kempen, Eds., Cognitive Ergonomics: Contributions from Experimental Psychology, Theoretical Issues, pp. 27-67. Amsterdam, The Netherlands: Elsevier Science Publishers.
	31	Flavio de Souza , Nigel Bevan, The use of guidelines in menu interface design: Evaluation of a draft standard, Proceedings of the IFIP TC13 Third Interational Conference on Human-Computer Interaction, p.435-440, August 27-31, 1990
	32	DETWEILER,M.C.AND OMANSON, R. C. 1996. Ameritech web page user interface standards and design guidelines. Ameritech Corporation, Chicago, IL. Available at http:// www.ameritech.com/corporate/testtown/library/ standard/web guidelines/index.html.
	33	Alan Dix , Janet Finley , Gregory Abowd , Russell Beale, Human-computer interaction (2nd ed.), Prentice-Hall, Inc., Upper Saddle River, NJ, 1998
	34	M. Carl Drott, Using Web server logs to improve site design, Proceedings of the 16th annual international conference on Computer documentation, p.43-50, September 24-26, 1998, Quebec, Quebec, Canada [doi>10.1145/296336.296350]
	35	ETGEN,M.AND CANTOR, J. 1999. What does getting WET (web event-logging tool) mean for web usability. In Proceedings of the Fifth Conference on Human Factors & the Web (Gaithersburg, MD, June). Available at http://www.nist.gov/itl/div894/vvrg/hfweb/ proceedings/etgen-cantor/index.html.
	36	FARADAY, P. 2000. Visually critiquing web pages. In Proceedings of the Sixth Conference on Human Factors & the Web (Austin, TX, June). Available at http://www.tri.sbc.com/hfweb/faraday/ faraday.htm.
	37	Pete Faraday , Alistair Sutcliffe, Providing advice for multimedia designers, Proceedings of the SIGCHI conference on Human factors in computing systems, p.124-131, April 18-23, 1998, Los Angeles, California, United States [doi>10.1145/274644.274663]
	38	Christelle Farenc , Philippe Palanque, A generic framework based on ergonomics rules for computer aided design of user interface, Proceedings of the third international conference on Computer-aided design of user interfaces, p.281-292, October 1999, Louvain-la-Neuve, Belgium
	39	FARENC, C., LIBERATI,V.,AND BARTHET, M.-F. 1999. Automatic ergonomic evaluation: What are the limits. In Proceedings of the Third International Conference on Computer-Aided Design of User Interfaces (Louvain-la-Neuve, Belgium, October). Dordrecht, The Netherlands: Kluwer Academic Publishers.
	40	FOGG, B. J. 1999. What variables affect web credibility? Available at http://www. webcredibility.org/variables files/v3 document. htm.
	41	BJ Fogg , Jonathan Marshall , Alex Osipovich , Chris Varma , Othman Laraki , Nicholas Fang , Jyoti Paul , Akshay Rangnekar , John Shon , Preeti Swani , Marissa Treinen, Elements that affect web credibility: early results from a self-report study, CHI '00 extended abstracts on Human factors in computing systems, April 01-06, 2000, The Hague, The Netherlands [doi>10.1145/633292.633460]
	42	FULLER,R.AND DE GRAAFF, J. J. 1996. Measuring user motivation from server log files. In Proceedings of the Second Conference on Human Factors & the Web (Redmond, WA, October). Available at http://www.microsoft.com/usability/ webconf/ fuller/fuller.htm.
	43	GLENN, F. A., SCHWARTZ,S.M.,AND ROSS, L. V. 1992. Development of a human operator simulator version v (HOS-V): Design and implementation. U.S. Army Research Institute for the Behavioral and Social Sciences, PERI-POX, Alexandria, VA.
	44	GUZDIAL, M., SANTOS, P., BADRE, A., HUDSON,S., AND GRAY, M. 1994. Analyzing and visualizing log files: A computational science of usability. GVU Center TR GIT-GVU-94-8, Georgia Institute of Technology. Available at http:// www.cc.gatech.edu/gvu/reports/1994/abstracts/ 94-08.html.
	45	Monty L. Hammontree , Jeffrey J. Hendrickson , Billy W. Hensley, Integrated data capture and analysis tools for research and testing on graphical user interfaces, Proceedings of the SIGCHI conference on Human factors in computing systems, p.431-432, May 03-07, 1992, Monterey, California, United States [doi>10.1145/142750.142886]
	46	HARPER,B.AND NORMAN, K. 1993. Improving user satisfaction: The questionnaire for user satisfaction interaction version 5.5. In Proceedings of the First Annual Mid-Atlantic Human Factors Conference (Virginia Beach, VA), pp. 224-228.
	47	H. Rex Hartson , José C. Castillo , John Kelso , Wayne C. Neale, Remote evaluation: the network as an extension of the usability laboratory, Proceedings of the SIGCHI conference on Human factors in computing systems: common ground, p.228-235, April 13-18, 1996, Vancouver, British Columbia, Canada [doi>10.1145/238386.238511]
	48	HELFRICH,B.AND LANDAY, J. A. 1999. QUIP: quantitative user interface profiling. Unpublished manuscript. Available at http://home. earthlink.net/ >> bhelfrich/quip/index.html.
	49	Harry Hochheiser , Ben Shneiderman, Using interactive visualizations of WWW log data to characterize access patterns and inform site design, Journal of the American Society for Information Science and Technology, v.52 n.4, p.331-343, Feb. 2001 [doi>10.1002/1532-2890(2000)9999:9999<::AID-ASI1066>3.3.CO;2-P]
	50	HOM, J. 1998. The usability methods toolbox. Available at http://www.best.com/ >> jthom/usability/ usable.htm.
	51	Scott E. Hudson , Bonnie E. John , Keith Knudsen , Michael D. Byrne, A tool for creating predictive performance models from user interface demonstrations, Proceedings of the 12th annual ACM symposium on User interface software and technology, p.93-102, November 07-10, 1999, Asheville, North Carolina, United States [doi>10.1145/320719.322590]
	52	HUMAN FACTORS ENGINEERING. 1999. Usability evaluation methods. Available at http://www. cs.umd.edu/ zzj/UsabilityHome.html.
	53	INTERNATIONAL STANDARDS ORGANIZATION. 1999. Ergonomic requirements for office work with visual display terminals, part 11: Guidance on usability. Available at http://www.iso.ch/iso/ en/catalogueDetailPage.catalogueDetail? CS-Number D 16883&ICS1 D 13&ICS2 D 180&ICS3 D IVORY, M. Y. 2001.
	54	IVORY, M. Y., SINHA,R.R.,AND HEARST,M.A. 2000. Preliminary findings on quantitative measures for distinguishing highly rated information-centric web pages. In Proceedings of the Sixth Conference on Human Factors & the Web (Austin, TX, June). Available at http://www.tri.sbc.com/hfweb/ivory/paper.html.
	55	Melody Y. Ivory , Rashmi R. Sinha , Marti A. Hearst, Empirically validated web page design metrics, Proceedings of the SIGCHI conference on Human factors in computing systems, p.53-60, March 2001, Seattle, Washington, United States [doi>10.1145/365024.365035]
	56	JAIN, R. 1991. Human-Computer Interaction. Wiley-Interscience, New York, NY.
	57	Robin Jeffries , James R. Miller , Cathleen Wharton , Kathy Uyeda, User interface evaluation in the real world: a comparison of four techniques, Proceedings of the SIGCHI conference on Human factors in computing systems: Reaching through technology, p.119-124, April 27-May 02, 1991, New Orleans, Louisiana, United States [doi>10.1145/108844.108862]
	58	JIANG, J., MURPHY, E., AND CARTER, L. 1993. Computer-human interaction models (CHIMES): Revision 3. Tech. Rep. DSTL-94- 008 (May), National Aeronautics and Space Administration.
	59	Bonnie E. John , David E. Kieras, The GOMS family of user interface analysis techniques: comparison and contrast, ACM Transactions on Computer-Human Interaction (TOCHI), v.3 n.4, p.320-351, Dec. 1996 [doi>10.1145/235833.236054]
	60	David J. Kasik , Harry G. George, Toward automatic generation of novice user test scripts, Proceedings of the SIGCHI conference on Human factors in computing systems: common ground, p.244-251, April 13-18, 1996, Vancouver, British Columbia, Canada [doi>10.1145/238386.238519]
	61	KIERAS,D.AND POLSON, P. G. 1985. An approach to the formal analysis of user complexity. International Journal of Man-Machine Studies 22,4, 365-394.
	62	David E. Kieras , Scott D. Wood , Kasem Abotel , Anthony Hornof, GLEAN: a computer-based tool for rapid GOMS model usability evaluation of user interface designs, Proceedings of the 8th annual ACM symposium on User interface and software technology, p.91-100, November 15-17, 1995, Pittsburgh, Pennsylvania, United States [doi>10.1145/215585.215700]
	63	David E. Kieras , Scott D. Wood , David E. Meyer, Predictive engineering models based on the EPIC architecture for a multimodal high-performance human-computer interaction task, ACM Transactions on Computer-Human Interaction (TOCHI), v.4 n.3, p.230-275, Sept. 1997 [doi>10.1145/264645.264658]
	64	Won Chul Kim , James D. Foley, Providing high-level control and expert assistance in the user interface presentation design, Proceedings of the INTERCHI '93 conference on Human factors in computing systems, p.430-437, May 1993, Amsterdam, The Netherlands
	65	KIRAKOWSKI,J.AND CLARIDGE, N. 1998. Human centered measures of success in web site design. In Proceedings of the Fourth Conference on Human Factors & the Web (Basking Ridge, NJ, June). Available at http:// www. research.att. com/conf/hfweb/proceedings/ kirakowski/index.html.
	66	LAIRD,J.E.AND ROSENBLOOM, P. 1996. The evolution of the Soar cognitive architecture. In D. M. Steier and T. M. Mitchell, Eds., Mind Matters: A Tribute to Allen Newell, pp. 1-50. Mahwah, NJ: Lawrence Erlbaum Associates, Inc.
	67	Kevin Larson , Mary Czerwinski, Web page design: implications of memory, structure and scent for information retrieval, Proceedings of the SIGCHI conference on Human factors in computing systems, p.25-32, April 18-23, 1998, Los Angeles, California, United States [doi>10.1145/274644.274649]
	68	Andreas Lecerof , Fabio Paternò, Automatic Support for Usability Evaluation, IEEE Transactions on Software Engineering, v.24 n.10, p.863-888, October 1998 [doi>10.1109/32.729686]
	69	LEE, K. 1997. Motif FAQ. Available at http://wwwbioeng. ucsd.edu/ >> fvetter/misc/Motif-FAQ.txt.
	70	LEVINE, R. 1996. Guide to web style. Sun Microsytems. Available at http://www.sun.com/ styleguide/.
	71	Clayton Lewis , Peter G. Polson , Cathleen Wharton , John Rieman, Testing a walkthrough methodology for theory-based design of walk-up-and-use interfaces, Proceedings of the SIGCHI conference on Human factors in computing systems: Empowering people, p.235-242, April 01-05, 1990, Seattle, Washington, United States [doi>10.1145/97243.97279]
	72	Jonas Löwgren , Tommy Nordqvist, Knowledge-based evaluation as design support for graphical user interfaces, Proceedings of the SIGCHI conference on Human factors in computing systems, p.181-188, May 03-07, 1992, Monterey, California, United States [doi>10.1145/142750.142786]
	73	LYNCH, G., PALMITER,S.,AND TILT, C. 1999. The max model: A standard web site user model. In Proceedings of the Fifth Conference on Human Factors & the Web (Gaithersburg, MD, June). Available at http://www.nist.gov/itl/div894/ vvrg/hfweb/ proceedings/lynch/index.html.
	74	Patrick J. Lynch , Sarah Horton, Web Style Guide: Basic Design Principles for Creating Web Sites, Yale University Press, New Haven, CT, 1999
	75	MACLEOD,M.AND RENGGER, R. 1993. The development of DRUM: A software tool for videoassisted usability evaluation. In Proceedings of the HCI Conference on People and Computers VIII (Loughborough, UK, September), pp. 293- 309. Cambridge University Press, Cambridge, UK.
	76	Rohit Mahajan , Ben Shneiderman, Visual and Textual Consistency Checking Tools for Graphical User Interfaces, IEEE Transactions on Software Engineering, v.23 n.11, p.722-735, November 1997 [doi>10.1109/32.637386]
	77	MAY,J.AND BARNARD, P. J. 1994. Supportive evaluation of interface design. In C. Stary, Ed., Proceedings of the First Interdisciplinary Workshop on Cognitive Modeling and User Interface Design (Vienna, Austria, December).
	78	MERCURY INTERACTIVE. 2000. Winrunner. Available at http://www-svca.mercuryinteractive.com/ products/winrunner/.
	79	MOLICH, R., BEVAN, N., BUTLER, S., CURSON, I., KIND- LUND, E., KIRAKOWSKI,J.,AND MILLER, D. 1998. Comparative evaluation of usability tests. In Proceedings of the UPA Conference (Washington, DC, June), pp. 189-200. Usability Professionals' Association, Chicago, IL.
	80	Rolf Molich , Ann Damgaard Thomsen , Barbara Karyukina , Lars Schmidt , Meghan Ede , Wilma van Oel , Meeta Arcuri, Comparative evaluation of usability tests, CHI '99 extended abstracts on Human factors in computing systems, May 15-20, 1999, Pittsburgh, Pennsylvania [doi>10.1145/632716.632768]
	81	MORAN, T. P. 1981. The command language grammar: Arepresentation for the user interface of interactive computer systems. International Journal of Man-Machine Studies 15, 1, 3-50.
	82	Thomas P. Moran, Getting into a system: External-internal task mapping analysis, Proceedings of the SIGCHI conference on Human Factors in Computing Systems, p.45-49, December 12-15, 1983, Boston, Massachusetts, United States [doi>10.1145/800045.801578]
	83	NETRAKER. 2000. The NetRaker suite. Available at http://www.netraker.com/info/applications/ index.asp.
	84	Jakob Nielsen, Usability Engineering, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1993
	85	Dan Olsen, User Interface Management Systems: Models and Algorithms, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1992
	86	Dan R. Olsen , Bradley W. Halversen, Interface usage measurements in a user interface management system, Proceedings of the 1st annual ACM SIGGRAPH symposium on User Interface Software, p.102-108, October 17-19, 1988, Alberta, Canada [doi>10.1145/62402.62420]
	87	CORPORATE Open Software Foundation, OSF/Motif programmer's guide: revision 1.1 (for OSF/Motif release 1.1), Prentice-Hall, Inc., Upper Saddle River, NJ, 1990
	88	PALANQUE, P., FARENC,C.,AND BASTIDE, R. 1999. Embedding ergonomic rules as generic requirements in the development process of interactive software. In A. Sasse and C. Johnson, Eds., Proceedings of IFIP TC13 Seventh International Conference on Human-Computer Interaction (Edinburgh, Scotland, August). Amsterdam, The Netherlands: IOS Press.
	89	PARUSH, A., NADIR, R., AND SHTUB, A. 1998. Evaluating the layout of graphical user interface screens: Validation of a numerical, computerized model. International Journal of Human Com-puter Interaction 10, 4, 343-360.
	90	PATERN O,F.AND BALLARDIN, G. 1999. Modelaided remote usability evaluation. In A. Sasse and C. Johnson, Eds., Proceedings of the IFIP TC13 Seventh International Conference on Human- Computer Interaction (Edinburgh, Scotland, August), pp. 434-442. Amsterdam, The Netherlands: IOS Press.
	91	Fabio Paternò , Cristiano Mancini , Silvia Meniconi, ConcurTaskTrees: A Diagrammatic Notation for Specifying Task Models, Proceedings of the IFIP TC13 Interantional Conference on Human-Computer Interaction, p.362-369, July 14-18, 1997
	92	PAYNE,S.J.AND GREEN, T. R. G. 1986. Taskaction grammars: A model of the mental representation of task languages. Human-Computer Interaction 2, 93-133.
	93	Virginia A. Peck , Bonnie E. John, Browser-Soar: a computational model of a highly interactive task, Proceedings of the SIGCHI conference on Human factors in computing systems, p.165-172, May 03-07, 1992, Monterey, California, United States [doi>10.1145/142750.142782]
	94	PETRI, C. A. 1973. Concepts of net theory. In Mathematical Foundations of Computer Science: Proceedings of the Symposium and Summer School (High Tatras, Czechoslovakia, September), pp. 137-146. Mathematical Institute of the Slovak Academy of Sciences.
	95	PEW,R.W.AND MAVOR,A.S.,EDS. 1998. Modeling Human and Organizational Behavior: Application to Military Simulations. Washington, DC: National Academy Press. Available at http://books.nap.edu/html/model.
	96	POLK,T.A.AND ROSENBLOOM, P. S. 1994. Taskindependent constraints on a unified theory of cognition. In F. Boller and J. Grafman, Eds., Handbook of Neuropsychology, Volume 9. Amsterdam, The Netherlands: Elsevier Science Publishers.
	97	Julie Ratner , Eric M. Grose , Chris Forsythe, Characterization and assessment of HTML style guides, Conference companion on Human factors in computing systems: common ground, p.115-116, April 13-18, 1996, Vancouver, British Columbia, Canada [doi>10.1145/257089.257193]
	98	RAUTERBERG, M. 1995. From novice to expert decision behaviour: A qualitative modeling approach with Petri nets. In Y. Anzai, K. Ogawa, and H. Mori, Eds., Symbiosis of Human and Artifact: Human and Social Aspects of Human-Computer Interaction, Volume 20B of Advances in Human Factors/Ergonomics (1995), pp. 449-454. Amsterdam, The Netherlands: Elsevier Science Publishers.
	99	RAUTERBERG, M. 1996a. How to measure and to quantify usability of user interfaces. In A. Ozok and G. Salvendy, Eds., Advances in Applied Ergonomics (1996), pp. 429-432. West Lafayette, IN: USA Publishing.
	100	RAUTERBERG, M. 1996b. A Petri net based analyzing and modeling tool kit for logfiles in humancomputer interaction. In Proceedings of Cognitive Systems Engineering in Process Control (Kyoto, Japan, November), pp. 268-275. Kyoto University: Graduate School of Energy Science.
	101	RAUTERBERG,M.AND AEPPILI, R. 1995. Learning in man-machine systems: the measurement of behavioural and cognitive complexity. In Proceedings of the IEEE Conference on Systems, Man and Cybernetics (Vancouver, BC, October), pp. 4685-4690. Institute of Electrical and Electronics Engineers.
	102	REISNER, P. 1984. Formal grammar as a tool for analyzing ease of use: Some fundamental concepts. In J. C. Thomas and M. L. Schneider, Eds., Human Factors in Computer Systems, pp. 53-78. Norwood, NJ: Ablex Publishing Corp.
	103	REITERER, H. 1994. A user interface design assistant approach. In K. Brunnstein and E. Raubold, Eds., Proceedings of the IFIP 13th World Computer Congress, Volume 2 (Hamburg, Germany, August), pp. 180-187. Amsterdam, The Netherlands: Elsevier Science Publishers.
	104	John Rieman , Susan Davies , D. Charles Hair , Mary Esemplare , Peter Polson , Clayton Lewis, An automated cognitive walkthrough, Proceedings of the SIGCHI conference on Human factors in computing systems: Reaching through technology, p.427-428, April 27-May 02, 1991, New Orleans, Louisiana, United States [doi>10.1145/108844.108986]
	105	SCAPIN, D., LEULIER, C., VANDERDONCKT, J., MARIAGE, C., BASTIEN, C., FARENC, C., PALANQUE,P.,AND BASTIDE, R. 2000. A framework for organizing web usability guidelines. In Proceedings of the Sixth Conference on Human Factors & the Web (Austin, TX, June). Available at http:// www.tri.sbc.com/hfweb/scapin/Scapin.html.
	106	SCHOLTZ,J.AND LASKOWSKI, S. 1998. Developing usability tools and techniques for designing and testing web sites. In Proceedings of the Fourth Conference on Human Factors & the Web (Basking Ridge, NJ, June). Available at http://www.research.att.com/conf/hfweb/ proceedings/ scholtz/index.html.
	107	SCHWARTZ, M. 2000. Web site makeover. Computerworld January 31. Available at http:// www.computerworld.com/home/print.nsf/all/ 000126e3e2.
	108	Andrew Sears, AIDE: a step toward metric-based interface development tools, Proceedings of the 8th annual ACM symposium on User interface and software technology, p.101-110, November 15-17, 1995, Pittsburgh, Pennsylvania, United States [doi>10.1145/215585.215704]
	109	SERVICE METRICS. 1999. Service metrics solutions. Available at http://www.servicemetrics.com/solutions/ solutionsmain.asp.
	110	Ben Shneiderman, Designing the user interface (2nd ed.): strategies for effective human-computer interaction, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1992
	111	Antonio C. Siochi , Deborah Hix, A study of computer-supported user interface evaluation using maximal repeating pattern analysis, Proceedings of the SIGCHI conference on Human factors in computing systems: Reaching through technology, p.301-305, April 27-May 02, 1991, New Orleans, Louisiana, United States [doi>10.1145/108844.108926]
	112	SMITH, S. L. 1986. Standards versus guidelines for designing user interface software. Behaviour and Information Technology 5, 1, 47-61.
	113	SMITH,S.L.AND MOSIER, J. N. 1986. Guidelines for designing user interface software. Tech. Rep. ESD-TR-86-278, The MITRE Corporation, Bedford, MA 01730.
	114	STEIN, L. D. 1997. The rating game. Available at http://stein.cshl.org/ lstein/rater/.
	115	STREVELER,D.J.AND WASSERMAN, A. I. 1984. Quantitative measures of the spatial properties of screen designs. In B. Shackel, Ed., Proceedings of the IFIP TC13 First International Conference on Human-Computer Interaction (London, UK, September), pp. 81-89. Amsterdam, The Netherlands: North-Holland.
	116	SULLIVAN, T. 1997. Reading reader reaction: A proposal for inferential analysis of web server log files. In Proceedings of the Third Conference on Human Factors & the Web (Boulder, CO, June). Available at http://www.research.att.com/ conf/hfweb/conferences/denver3.zip.
	117	Michael J. Tauber, ETAG: Extended task action grammar. A language for the description of the user's task language, Proceedings of the IFIP TC13 Third Interational Conference on Human-Computer Interaction, p.163-168, August 27-31, 1990
	118	THENG,Y.L.AND MARSDEN, G. 1998. Authoring tools: Towards continuous usability testing of web documents. In Proceedings of the First International Workshop on Hypermedia Development (Pittsburgh, PA, June). Available at http://www.eng.uts.edu.au/ >> dbl/HypDev/ht98w/ YinLeng/HT98 YinLeng.html.
	119	Harold Thimbleby, Gentler: a tool for systematic web authoring, International Journal of Human-Computer Studies, v.47 n.1, p.139-168, July 1997 [doi>10.1006/ijhc.1997.0131]
	120	TULLIS, T. S. 1983. The formatting of alphanumeric displays: A review and analysis. Human Factors 25, 657-682.
	121	Dana L. Uehling , Karl Wolf, User action graphing effort (UsAGE), Conference companion on Human factors in computing systems, p.290-291, May 07-11, 1995, Denver, Colorado, United States [doi>10.1145/223355.223679]
	122	USABLE NET. 2000. LIFT online. Available at http://www.usablenet.com.
	123	Jean Vanderdonckt , Xavier Gillo, Visual techniques for traditional and multimedia layouts, Proceedings of the workshop on Advanced visual interfaces, p.95-104, June 01-04, 1994, Bari, Italy [doi>10.1145/192309.192334]
	124	WEB ACCESSIBILITY INITIATIVE. 1999. Web content accessibility guidelines 1.0. World Wide Web Consortium, Geneva, Switzerland. Available at http://www.w3.org/TR/WAI-WEBCON-TENT/.
	125	WEB CRITERIA. 1999. Max, and the objective measurement of web sites. Available at http://www.webcriteria.com.
	126	WEBTRENDS CORPORATION. 2000. Webtrends live. Available at http://www.webtrendslive.com/default. htm.
	127	WHITEFIELD, A., WILSON,F.,AND DOWELL, J. 1991. A framework for human factors evaluation. Behaviour and Information Technology 10, 1, 65- 79.
	128	Marian G. Williams, Translation in participatory design: lessons from a workshop, INTERACT '93 and CHI '93 conference companion on Human factors in computing systems, p.55-56, April 24-29, 1993, Amsterdam, The Netherlands [doi>10.1145/259964.260063]
	129	WORLD WIDE WEB CONSORTIUM. 2000. HTML valiation service. Available at http://validator.w3. org/.
	130	R. M. Young , T. R. G. Green , T. Simon, Programmable user models for predictive evaluation of interface designs, Proceedings of the SIGCHI conference on Human factors in computing systems: Wings for the mind, p.15-19, March 1989
	131	ZACHARY, W., MENTEC, J.-C. L., AND RYDER,J. 1996. Interface agents in complex systems. In C. N. Ntuen and E. H. Park, Eds., Human Interaction With Complex Systems: Conceptual Principles and Design Practice. Dordrecht, The Netherlands: Kluwer Academic Publishers.
	132	ZAPHIRIS,P.AND MTEI, L. 1997. Depth vs. breadth in the arrangement of Web links. Available at http://www.otal.umd.edu/SHORE/bs04.
	133	Luke S. Zettlemoyer , Robert St. Amant , Martin S. Dulberg, IBOTS: agent control through the user interface, Proceedings of the 4th international conference on Intelligent user interfaces, p.31-37, January 05-08, 1999, Los Angeles, California, United States [doi>10.1145/291080.291087]